You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(ollama): stop reloading the model on every run - #268
Kit loads an Ollama model with a warmup request to /api/generate. This
request always sent num_gpu and main_gpu: -1 and 0 from the flag
defaults in the CLI, and from setSDKDefaults in the SDK. Kit sends the
chat requests to the OpenAI-compatible /v1 endpoint, without GPU
options.
Since Ollama 0.30.0 (ollama/ollama#16031), main_gpu is a pointer in the
runner options, so main_gpu 0 is no longer the same as no main_gpu.
Ollama loads the model again when the runner options of a request
differ from those of the loaded model (needsReload in server/sched.go).
So the warmup loaded the model again, and the first chat request loaded
it once more.
BuildProviderConfig now sets NumGPU and MainGPU only when num-gpu-layers
or main-gpu is set explicitly: with a flag, a KIT_* variable or the
config file. It already works this way for the sampling parameters.
setSDKDefaults no longer registers defaults for the two keys, because a
viper default makes IsSet return true. Both changes are necessary:
without the defaults, the old code would send num_gpu 0, which puts all
layers on the CPU.
Known limit: with an explicit --main-gpu, Ollama still loads the model
again for the chat requests, because the /v1 endpoint has no field for
runner options. On master, this happens on every run. A main_gpu set as
a model parameter (PARAMETER main_gpu N in a Modelfile) applies to all
requests and causes no reloads.
Testing:
go vet ./... passes. go test -race ./... on master f9e6cc5 gives the
same results with and without the change, plus the new tests. All
packages pass, except TestSetParentLinkErrorKeepsLock in
internal/session: it fails on master too when the tests run as root.
The new tests fail on master. TestOllamaWarmupGPUOptions also fails
if either of the two changes is missing.
Model loads with Ollama 0.40.14 on CPU and a tiny GGUF test model,
counted as llama-server starts in the server log:
Kit run
master
this PR
kit --model ollama/tiny "hello", per run
2
0
TUI start
1
0
TUI, first prompt
1
0
TUI, second prompt
0
0
kit --main-gpu 0 --model ollama/tiny "hello", per run
2
2
Summary by CodeRabbit
Bug Fixes
GPU settings are omitted from Ollama requests when left unset.
Explicit GPU layer counts and GPU selections are included when configured, including a selection of GPU 0.
Kit loads an Ollama model with a warmup request to /api/generate. This
request always sent num_gpu and main_gpu: -1 and 0 from the flag
defaults in the CLI, and from setSDKDefaults in the SDK. Kit sends the
chat requests to the OpenAI-compatible /v1 endpoint, without GPU
options.
Since Ollama 0.30.0, main_gpu 0 is no longer the same as no main_gpu.
Ollama loads the model again when the runner options of a request
differ from those of the loaded model. So the warmup loaded the model
again, and the first chat request loaded it once more. With Ollama
0.40.14, each one-shot run loaded the model twice.
BuildProviderConfig now sets NumGPU and MainGPU only when num-gpu-layers
or main-gpu is set explicitly: with a flag, a KIT_* variable or the
config file. It already works this way for the sampling parameters.
setSDKDefaults no longer registers defaults for the two keys, because a
viper default makes IsSet return true. Both changes are necessary:
without the defaults, the old code would send num_gpu 0, which puts all
layers on the CPU.
With an explicit main-gpu, Ollama still loads the model again for the
chat requests, because the /v1 endpoint has no field for runner options.
TestBuildProviderConfigGPUFlags checks the CLI flags.
TestOllamaWarmupGPUOptions checks the options of the warmup request.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Matthias G. Eckermann <mge@suse.com>
No actionable comments were generated in the recent review.
Walkthrough
GPU settings no longer receive Viper defaults. Provider configuration sets GPU option pointers only when their keys are explicitly configured. Tests check unset options and explicit values in provider configuration and Ollama warmup requests.
Changes: Ollama GPU option handling
Layer / File(s)
Summary
GPU option configuration and validation pkg/kit/config.go, internal/kitsetup/setup.go, internal/kitsetup/setup_test.go, pkg/kit/ollama_warmup_test.go
Viper no longer sets defaults for num-gpu-layers and main-gpu. BuildProviderConfig sets each pointer only when its key is configured, preserving explicit zero values. Tests check unset and configured options, including values in warmup request options.
The change avoids sending unconfigured GPU options while preserving explicitly configured values. No actionable merge-blocking issue was identified; merge after normal checks pass.
Pre-merge checks: 5
Check name
Status
Explanation
Description Check
Check skipped - CodeRabbit’s high-level summary is enabled.
Title check
The title clearly states the primary change: preventing Ollama from reloading the model on every run. The changes implement this by omitting unset GPU options.
Linked Issues check
Issue [#265] reports unnecessary Ollama model reloads during CLI runs. BuildProviderConfig now leaves NumGPU and MainGPU unset unless the user configures them. setSDKDefaults no longer makes t…
Out of Scope Changes check
The production changes and tests remain limited to Ollama GPU option configuration and warmup behavior. The tests support the linked issue by checking option presence and values. No unrelated implemen…
Docstring Coverage
Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Full details: Linked Issues check
Explanation
Issue [#265] reports unnecessary Ollama model reloads during CLI runs. BuildProviderConfig now leaves NumGPU and MainGPU unset unless the user configures them. setSDKDefaults no longer makes these keys appear set. The added tests verify omitted options, explicit zero values, and the warmup request. These changes address the reported reload mismatch. The documented --main-gpu limitation does not prevent this fix for the default configuration.
Full details: Out of Scope Changes check
Explanation
The production changes and tests remain limited to Ollama GPU option configuration and warmup behavior. The tests support the linked issue by checking option presence and values. No unrelated implementation or test changes are shown.
Full details: Docstring Coverage
Explanation
Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 4 functions across 4 files.
Finishing Touches
Generate unit tests (beta)
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Tip
🎯 Customize your risk assessments
Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.
✅ Automatic approvals, with your rules
Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.
CodeRabbit Verify
Run CodeRabbit Verify to test the changes in this pull request.
Recent review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: def3617d-6209-48d9-8726-5e0535906427
Commits
Reviewing files that changed from the base of the PR and between b88851d and 0ff8e80.
Files selected for processing: 1
internal/kitsetup/setup_test.go
Files skipped from review as they are similar to previous changes: 1
internal/kitsetup/setup_test.go
Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.
The lint job (golangci-lint, modernize/newexpr) flagged the int32Ptr
helper in the new test. Since Go 1.26, new accepts an expression, so
the test table now uses new(int32(N)) directly, as other tests in the
repository do.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Matthias G. Eckermann <mge@suse.com>
Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.
formatInt32Ptr was the only function in this change without a doc
comment.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Matthias G. Eckermann <mge@suse.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #265.
Kit loads an Ollama model with a warmup request to /api/generate. This
request always sent num_gpu and main_gpu: -1 and 0 from the flag
defaults in the CLI, and from setSDKDefaults in the SDK. Kit sends the
chat requests to the OpenAI-compatible /v1 endpoint, without GPU
options.
Since Ollama 0.30.0 (ollama/ollama#16031), main_gpu is a pointer in the
runner options, so main_gpu 0 is no longer the same as no main_gpu.
Ollama loads the model again when the runner options of a request
differ from those of the loaded model (needsReload in server/sched.go).
So the warmup loaded the model again, and the first chat request loaded
it once more.
BuildProviderConfig now sets NumGPU and MainGPU only when num-gpu-layers
or main-gpu is set explicitly: with a flag, a KIT_* variable or the
config file. It already works this way for the sampling parameters.
setSDKDefaults no longer registers defaults for the two keys, because a
viper default makes IsSet return true. Both changes are necessary:
without the defaults, the old code would send num_gpu 0, which puts all
layers on the CPU.
Known limit: with an explicit --main-gpu, Ollama still loads the model
again for the chat requests, because the /v1 endpoint has no field for
runner options. On master, this happens on every run. A main_gpu set as
a model parameter (PARAMETER main_gpu N in a Modelfile) applies to all
requests and causes no reloads.
Testing:
same results with and without the change, plus the new tests. All
packages pass, except TestSetParentLinkErrorKeepsLock in
internal/session: it fails on master too when the tests run as root.
if either of the two changes is missing.
Model loads with Ollama 0.40.14 on CPU and a tiny GGUF test model,
counted as llama-server starts in the server log:
kit --model ollama/tiny "hello", per runkit --main-gpu 0 --model ollama/tiny "hello", per runSummary by CodeRabbit