mirror of
https://github.com/ollama/ollama.git
synced 2026-05-06 08:02:14 -05:00
[GH-ISSUE #9782] "Error: llama runner process has terminated: CUDA error: out of memory" while having some idling GPUs #52908
Closed
opened 2026-04-29 01:20:20 -05:00 by GiteaMirror
·
20 comments
No Branch/Tag Specified
main
parth-mlx-decode-checkpoints
dhiltgen/ci
hoyyeva/editor-config-repair
parth-launch-codex-app
hoyyeva/fix-codex-model-metadata-warning
hoyyeva/qwen
hoyyeva/launch-backup-ux
parth/hide-claude-desktop-till-release
hoyyeva/opencode-image-modality
parth-add-claude-code-autoinstall
release_v0.22.0
pdevine/manifest-list
codex/fix-codex-model-metadata-warning
pdevine/addressable-manifest
brucemacd/launch-fetch-reccomended
jmorganca/llama-compat
launch-copilot-cli
hoyyeva/opencode-thinking
release_v0.20.7
parth-auto-save-backup
parth-test
jmorganca/gemma4-audio-replacements
fix-manifest-digest-on-pull
hoyyeva/vscode-improve
brucemacd/install-server-wait
brucemacd/download-before-remove
parth/update-claude-docs
parth-anthropic-reference-images-path
brucemac/start-ap-install
pdevine/mlx-update
pdevine/qwen35_vision
drifkin/api-show-fallback
mintlify/image-generation-1773352582
hoyyeva/server-context-length-local-config
jmorganca/faster-reptition-penalties
jmorganca/convert-nemotron
parth-pi-thinking
pdevine/sampling-penalties
jmorganca/fix-create-quantization-memory
dongchen/resumable_transfer_fix
pdevine/sampling-cache-error
jessegross/mlx-usage
hoyyeva/openclaw-config
hoyyeva/app-html
pdevine/qwen3next
brucemacd/sign-sh-install
brucemacd/tui-update
brucemacd/usage-api
jmorganca/launch-empty
fix-app-dist-embed
mxyng/mlx-compile
mxyng/mlx-quant
mxyng/mlx-glm4.7
mxyng/mlx
brucemacd/simplify-model-picker
jmorganca/qwen3-concurrent
fix-glm-4.7-flash-mla-config
drifkin/qwen3-coder-opening-tag
brucemacd/usage-cli
fix-cuda12-fattn-shmem
ollama-imagegen-docs
parth/fix-multiline-inputs
brucemacd/config-docs
mxyng/model-files
mxyng/simple-execute
fix-imagegen-ollama-models
mxyng/async-upload
jmorganca/lazy-no-dtype-changes
imagegen-auto-detect-create
parth/decrease-concurrent-download-hf
fix-mlx-quantize-init
jmorganca/x-cleanup
usage
imagegen-readme
jmorganca/glm-image
mlx-gpu-cd
jmorganca/imagegen-modelfile
parth/agent-skills
parth/agent-allowlist
parth/signed-in-offline
parth/agents
parth/fix-context-chopping
improve-cloud-flow
parth/add-models-websearch
parth/prompt-renderer-mcp
jmorganca/native-settings
jmorganca/download-stream-hash
jmorganca/client2-rebased
brucemacd/oai-chat-req-multipart
jessegross/multi_chunk_reserve
grace/additional-omit-empty
grace/mistral-3-large
mxyng/tokenizer2
mxyng/tokenizer
jessegross/flash
hoyyeva/windows-nacked-app
mxyng/cleanup-attention
grace/deepseek-parser
hoyyeva/remember-unsent-prompt
parth/add-lfs-pointer-error-conversion
parth/olmo2-test2
hoyyeva/ollama-launchagent-plist
nicole/olmo-model
parth/olmo-test
mxyng/remove-embedded
parth/render-template
jmorganca/intellect-3
parth/remove-prealloc-linter
jmorganca/cmd-eval
nicole/nomic-embed-text-fix
mxyng/lint-2
hoyyeva/add-gemini-3-pro-preview
hoyyeva/load-model-list
mxyng/expand-path
mxyng/environ-2
hoyyeva/deeplink-json-encoding
parth/improve-tool-calling-tests
hoyyeva/conversation
hoyyeva/assistant-edit-response
hoyyeva/thinking
origin/brucemacd/invalid-char-i-err
parth/improve-tool-calling
jmorganca/required-omitempty
grace/qwen3-vl-tests
mxyng/iter-client
parth/docs-readme
nicole/embed-test
pdevine/integration-benchstat
parth/remove-generate-cmd
parth/add-toolcall-id
mxyng/server-tests
jmorganca/glm-4.6
jmorganca/gin-h-compat
drifkin/stable-tool-args
pdevine/qwen3-more-thinking
parth/add-websearch-client
nicole/websearch_local
jmorganca/qwen3-coder-updates
grace/deepseek-v3-migration-tests
mxyng/fix-create
jmorganca/cloud-errors
pdevine/parser-tidy
revert-12233-parth/simplify-entrypoints-runner
parth/enable-so-gpt-oss
brucemacd/qwen3vl
jmorganca/readme-simplify
parth/gpt-oss-structured-outputs
revert-12039-jmorganca/tools-braces
mxyng/embeddings
mxyng/gguf
mxyng/benchmark
mxyng/types-null
parth/move-parsing
mxyng/gemma2
jmorganca/docs
mxyng/16-bit
mxyng/create-stdin
pdevine/authorizedkeys
mxyng/quant
parth/opt-in-error-context-window
brucemacd/cache-models
brucemacd/runner-completion
jmorganca/llama-update-6
brucemacd/benchmark-list
brucemacd/partial-read-caps
parth/deepseek-r1-tools
mxyng/omit-array
parth/tool-prefix-temp
brucemacd/runner-test
jmorganca/qwen25vl
brucemacd/model-forward-test-ext
parth/python-function-parsing
jmorganca/cuda-compression-none
drifkin/num-parallel
drifkin/chat-truncation-fix
jmorganca/sync
parth/python-tools-calling
drifkin/array-head-count
brucemacd/create-no-loop
parth/server-enable-content-stream-with-tools
qwen25omni
mxyng/v3
brucemacd/ropeconfig
jmorganca/silence-tokenizer
parth/sample-so-test
parth/sampling-structured-outputs
brucemacd/doc-go-engine
parth/constrained-sampling-json
jmorganca/mistral-wip
brucemacd/mistral-small-convert
parth/sample-unmarshal-json-for-params
brucemacd/jomorganca/mistral
pdevine/bfloat16
jmorganca/mistral
brucemacd/mistral
pdevine/logging
parth/sample-correctness-fix
parth/sample-fix-sorting
jmorgan/sample-fix-sorting-extras
jmorganca/temp-0-images
brucemacd/parallel-embed-models
brucemacd/shim-grammar
jmorganca/fix-gguf-error
bmizerany/nameswork
jmorganca/faster-releases
bmizerany/validatenames
brucemacd/err-no-vocab
brucemacd/rope-config
brucemacd/err-hint
brucemacd/qwen2_5
brucemacd/logprobs
brucemacd/new_runner_graph_bench
progress-flicker
brucemacd/forward-test
brucemacd/go_qwen2
pdevine/gemma2
jmorganca/add-missing-symlink-eval
mxyng/next-debug
parth/set-context-size-openai
brucemacd/next-bpe-bench
brucemacd/next-bpe-test
brucemacd/new_runner_e2e
brucemacd/new_runner_qwen2
pdevine/convert-cohere2
brucemacd/convert-cli
parth/log-probs
mxyng/next-mlx
mxyng/cmd-history
parth/templating
parth/tokenize-detokenize
brucemacd/check-key-register
bmizerany/grammar
jmorganca/vendor-081b29bd
mxyng/func-checks
jmorganca/fix-null-format
parth/fix-default-to-warn-json
jmorganca/qwen2vl
jmorganca/no-concat
parth/cmd-cleanup-SO
brucemacd/check-key-register-structured-err
parth/openai-stream-usage
parth/fix-referencing-so
stream-tools-stop
jmorganca/degin-1
brucemacd/install-path-clean
brucemacd/push-name-validation
brucemacd/browser-key-register
jmorganca/openai-fix-first-message
jmorganca/fix-proxy
jessegross/sample
parth/disallow-streaming-tools
dhiltgen/remove_submodule
jmorganca/ga
jmorganca/mllama
pdevine/newlines
pdevine/geems-2b
jmorganca/llama-bump
mxyng/modelname-7
mxyng/gin-slog
mxyng/modelname-6
jyan/convert-prog
jyan/quant5
paligemma-support
pdevine/import-docs
jmorganca/openai-context
jyan/paligemma
jyan/p2
jyan/palitest
bmizerany/embedspeedup
jmorganca/llama-vit
brucemacd/allow-ollama
royh/ep-methods
royh/whisper
mxyng/api-models
mxyng/fix-memory
jyan/q4_4/8
jyan/ollama-v
royh/stream-tools
roy-embed-parallel
bmizerany/hrm
revert-5963-revert-5924-mxyng/llama3.1-rope
royh/embed-viz
jyan/local2
jyan/auth
jyan/local
jyan/parse-temp
jmorganca/template-mistral
jyan/reord-g
royh-openai-suffixdocs
royh-imgembed
royh-embed-parallel
jyan/quant4
royh-precision
jyan/progress
pdevine/fix-template
jyan/quant3
pdevine/ggla
mxyng/update-registry-domain
jmorganca/ggml-static
mxyng/create-context
jyan/v0.146
mxyng/layers-from-files
build_dist
bmizerany/noseek
royh-ls
royh-name
timeout
mxyng/server-timestamp
bmizerany/nosillyggufslurps
royh-params
jmorganca/llama-cpp-7c26775
royh-openai-delete
royh-show-rigid
jmorganca/enable-fa
jmorganca/no-error-template
jyan/format
royh-testdelete
bmizerany/fastverify
language_support
pdevine/ps-glitches
brucemacd/tokenize
bruce/iq-quants
bmizerany/filepathwithcoloninhost
mxyng/split-bin
bmizerany/client-registry
jmorganca/if-none-match
native
jmorganca/native
jmorganca/batch-embeddings
jmorganca/initcmake
jmorganca/mm
pdevine/showggmlinfo
modenameenforcealphanum
bmizerany/modenameenforcealphanum
jmorganca/done-reason
jmorganca/llama-cpp-8960fe8
ollama.com
bmizerany/filepathnobuild
bmizerany/types/model/defaultfix
rmdisplaylong
nogogen
bmizerany/x
modelfile-readme
bmizerany/replacecolon
jmorganca/limit
jmorganca/execstack
jmorganca/replace-assets
mxyng/tune-concurrency
jmorganca/testing
whitespace-detection
jmorganca/options
upgrade-all
scratch
cuda-search
mattw/airenamer
mattw/allmodelsonhuggingface
mattw/quantcontext
mattw/whatneedstorun
brucemacd/llama-mem-calc
mattw/faq-context
mattw/communitylinks
mattw/noprune
mattw/python-functioncalling
rename
mxyng/install
pulse
remove-first
editor
mattw/selfqueryingretrieval
cgo
mattw/howtoquant
api
matt/streamingapi
format-config
mxyng/extra-args
shell
update-nous-hermes
cp-model
upload-progress
fix-unknown-model
fix-model-names
delete-fix
insecure-registry
ls
deletemodels
progressbar
readme-updates
license-layers
skip-list
list-models
modelpath
matt/examplemodelfiles
distribution
go-opts
v0.23.1
v0.23.1-rc0
v0.23.0
v0.23.0-rc0
v0.22.1
v0.22.1-rc1
v0.22.1-rc0
v0.22.0
v0.22.0-rc1
v0.21.3-rc0
v0.21.2-rc1
v0.21.2
v0.21.2-rc0
v0.21.1
v0.21.1-rc1
v0.21.1-rc0
v0.21.0
v0.21.0-rc1
v0.21.0-rc0
v0.20.8-rc0
v0.20.7
v0.20.7-rc1
v0.20.7-rc0
v0.20.6
v0.20.6-rc1
v0.20.6-rc0
v0.20.5
v0.20.5-rc2
v0.20.5-rc1
v0.20.5-rc0
v0.20.4
v0.20.4-rc2
v0.20.4-rc1
v0.20.4-rc0
v0.20.3
v0.20.3-rc0
v0.20.2
v0.20.1
v0.20.1-rc2
v0.20.1-rc1
v0.20.1-rc0
v0.20.0
v0.20.0-rc1
v0.20.0-rc0
v0.19.0
v0.19.0-rc2
v0.19.0-rc1
v0.19.0-rc0
v0.18.4-rc1
v0.18.4-rc0
v0.18.3
v0.18.3-rc2
v0.18.3-rc1
v0.18.3-rc0
v0.18.2
v0.18.2-rc1
v0.18.2-rc0
v0.18.1
v0.18.1-rc1
v0.18.1-rc0
v0.18.0
v0.18.0-rc2
v0.18.0-rc1
v0.18.0-rc0
v0.17.8-rc4
v0.17.8-rc3
v0.17.8-rc2
v0.17.8-rc1
v0.17.8-rc0
v0.17.7
v0.17.7-rc2
v0.17.7-rc1
v0.17.7-rc0
v0.17.6
v0.17.5
v0.17.4
v0.17.3
v0.17.2
v0.17.1
v0.17.1-rc2
v0.17.1-rc1
v0.17.1-rc0
v0.17.0
v0.17.0-rc2
v0.17.0-rc1
v0.17.0-rc0
v0.16.3
v0.16.3-rc2
v0.16.3-rc1
v0.16.3-rc0
v0.16.2
v0.16.2-rc0
v0.16.1
v0.16.0
v0.16.0-rc2
v0.16.0-rc0
v0.16.0-rc1
v0.15.6
v0.15.5
v0.15.5-rc5
v0.15.5-rc4
v0.15.5-rc3
v0.15.5-rc2
v0.15.5-rc1
v0.15.5-rc0
v0.15.4
v0.15.3
v0.15.2
v0.15.1
v0.15.1-rc1
v0.15.1-rc0
v0.15.0-rc6
v0.15.0
v0.15.0-rc5
v0.15.0-rc4
v0.15.0-rc3
v0.15.0-rc2
v0.15.0-rc1
v0.15.0-rc0
v0.14.3
v0.14.3-rc3
v0.14.3-rc2
v0.14.3-rc1
v0.14.3-rc0
v0.14.2
v0.14.2-rc1
v0.14.2-rc0
v0.14.1
v0.14.0-rc11
v0.14.0
v0.14.0-rc10
v0.14.0-rc9
v0.14.0-rc8
v0.14.0-rc7
v0.14.0-rc6
v0.14.0-rc5
v0.14.0-rc4
v0.14.0-rc3
v0.14.0-rc2
v0.14.0-rc1
v0.14.0-rc0
v0.13.5
v0.13.5-rc1
v0.13.5-rc0
v0.13.4-rc2
v0.13.4
v0.13.4-rc1
v0.13.4-rc0
v0.13.3
v0.13.3-rc1
v0.13.3-rc0
v0.13.2
v0.13.2-rc2
v0.13.2-rc1
v0.13.2-rc0
v0.13.1
v0.13.1-rc2
v0.13.1-rc1
v0.13.1-rc0
v0.13.0
v0.13.0-rc0
v0.12.11
v0.12.11-rc1
v0.12.11-rc0
v0.12.10
v0.12.10-rc1
v0.12.10-rc0
v0.12.9-rc0
v0.12.9
v0.12.8
v0.12.8-rc0
v0.12.7
v0.12.7-rc1
v0.12.7-rc0
v0.12.7-citest0
v0.12.6
v0.12.6-rc1
v0.12.6-rc0
v0.12.5
v0.12.5-rc0
v0.12.4
v0.12.4-rc7
v0.12.4-rc6
v0.12.4-rc5
v0.12.4-rc4
v0.12.4-rc3
v0.12.4-rc2
v0.12.4-rc1
v0.12.4-rc0
v0.12.3
v0.12.2
v0.12.2-rc0
v0.12.1
v0.12.1-rc1
v0.12.1-rc2
v0.12.1-rc0
v0.12.0
v0.12.0-rc1
v0.12.0-rc0
v0.11.11
v0.11.11-rc3
v0.11.11-rc2
v0.11.11-rc1
v0.11.11-rc0
v0.11.10
v0.11.9
v0.11.9-rc0
v0.11.8
v0.11.8-rc0
v0.11.7-rc1
v0.11.7-rc0
v0.11.7
v0.11.6
v0.11.6-rc0
v0.11.5-rc4
v0.11.5-rc3
v0.11.5
v0.11.5-rc5
v0.11.5-rc2
v0.11.5-rc1
v0.11.5-rc0
v0.11.4
v0.11.4-rc0
v0.11.3
v0.11.3-rc0
v0.11.2
v0.11.1
v0.11.0-rc0
v0.11.0-rc1
v0.11.0-rc2
v0.11.0
v0.10.2-int1
v0.10.1
v0.10.0
v0.10.0-rc4
v0.10.0-rc3
v0.10.0-rc2
v0.10.0-rc1
v0.10.0-rc0
v0.9.7-rc1
v0.9.7-rc0
v0.9.6
v0.9.6-rc0
v0.9.6-ci0
v0.9.5
v0.9.4-rc5
v0.9.4-rc6
v0.9.4
v0.9.4-rc3
v0.9.4-rc4
v0.9.4-rc1
v0.9.4-rc2
v0.9.4-rc0
v0.9.3
v0.9.3-rc5
v0.9.4-citest0
v0.9.3-rc4
v0.9.3-rc3
v0.9.3-rc2
v0.9.3-rc1
v0.9.3-rc0
v0.9.2
v0.9.1
v0.9.1-rc1
v0.9.1-rc0
v0.9.1-ci1
v0.9.1-ci0
v0.9.0
v0.9.0-rc0
v0.8.0
v0.8.0-rc0
v0.7.1-rc2
v0.7.1
v0.7.1-rc1
v0.7.1-rc0
v0.7.0
v0.7.0-rc1
v0.7.0-rc0
v0.6.9-rc0
v0.6.8
v0.6.8-rc0
v0.6.7
v0.6.7-rc2
v0.6.7-rc1
v0.6.7-rc0
v0.6.6
v0.6.6-rc2
v0.6.6-rc1
v0.6.6-rc0
v0.6.5-rc1
v0.6.5
v0.6.5-rc0
v0.6.4-rc0
v0.6.4
v0.6.3-rc1
v0.6.3
v0.6.3-rc0
v0.6.2
v0.6.2-rc0
v0.6.1
v0.6.1-rc0
v0.6.0-rc0
v0.6.0
v0.5.14-rc0
v0.5.13
v0.5.13-rc6
v0.5.13-rc5
v0.5.13-rc4
v0.5.13-rc3
v0.5.13-rc2
v0.5.13-rc1
v0.5.13-rc0
v0.5.12
v0.5.12-rc1
v0.5.12-rc0
v0.5.11
v0.5.10
v0.5.9
v0.5.9-rc0
v0.5.8-rc13
v0.5.8
v0.5.8-rc12
v0.5.8-rc11
v0.5.8-rc10
v0.5.8-rc9
v0.5.8-rc8
v0.5.8-rc7
v0.5.8-rc6
v0.5.8-rc5
v0.5.8-rc4
v0.5.8-rc3
v0.5.8-rc2
v0.5.8-rc1
v0.5.8-rc0
v0.5.7
v0.5.6
v0.5.5
v0.5.5-rc0
v0.5.4
v0.5.3
v0.5.3-rc0
v0.5.2
v0.5.2-rc3
v0.5.2-rc2
v0.5.2-rc1
v0.5.2-rc0
v0.5.1
v0.5.0
v0.5.0-rc1
v0.4.8-rc0
v0.4.7
v0.4.6
v0.4.5
v0.4.4
v0.4.3
v0.4.3-rc0
v0.4.2
v0.4.2-rc1
v0.4.2-rc0
v0.4.1
v0.4.1-rc0
v0.4.0
v0.4.0-rc8
v0.4.0-rc7
v0.4.0-rc6
v0.4.0-rc5
v0.4.0-rc4
v0.4.0-rc3
v0.4.0-rc2
v0.4.0-rc1
v0.4.0-rc0
v0.4.0-ci3
v0.3.14
v0.3.14-rc0
v0.3.13
v0.3.12
v0.3.12-rc5
v0.3.12-rc4
v0.3.12-rc3
v0.3.12-rc2
v0.3.12-rc1
v0.3.11
v0.3.11-rc4
v0.3.11-rc3
v0.3.11-rc2
v0.3.11-rc1
v0.3.10
v0.3.10-rc1
v0.3.9
v0.3.8
v0.3.7
v0.3.7-rc6
v0.3.7-rc5
v0.3.7-rc4
v0.3.7-rc3
v0.3.7-rc2
v0.3.7-rc1
v0.3.6
v0.3.5
v0.3.4
v0.3.3
v0.3.2
v0.3.1
v0.3.0
v0.2.8
v0.2.8-rc2
v0.2.8-rc1
v0.2.7
v0.2.6
v0.2.5
v0.2.4
v0.2.3
v0.2.2
v0.2.2-rc2
v0.2.2-rc1
v0.2.1
v0.2.0
v0.1.49-rc14
v0.1.49-rc13
v0.1.49-rc12
v0.1.49-rc11
v0.1.49-rc10
v0.1.49-rc9
v0.1.49-rc8
v0.1.49-rc7
v0.1.49-rc6
v0.1.49-rc4
v0.1.49-rc5
v0.1.49-rc3
v0.1.49-rc2
v0.1.49-rc1
v0.1.48
v0.1.47
v0.1.46
v0.1.45-rc5
v0.1.45
v0.1.45-rc4
v0.1.45-rc3
v0.1.45-rc2
v0.1.45-rc1
v0.1.44
v0.1.43
v0.1.42
v0.1.41
v0.1.40
v0.1.40-rc1
v0.1.39
v0.1.39-rc2
v0.1.39-rc1
v0.1.38
v0.1.37
v0.1.36
v0.1.35
v0.1.35-rc1
v0.1.34
v0.1.34-rc1
v0.1.33
v0.1.33-rc7
v0.1.33-rc6
v0.1.33-rc5
v0.1.33-rc4
v0.1.33-rc3
v0.1.33-rc2
v0.1.33-rc1
v0.1.32
v0.1.32-rc2
v0.1.32-rc1
v0.1.31
v0.1.30
v0.1.29
v0.1.28
v0.1.27
v0.1.26
v0.1.25
v0.1.24
v0.1.23
v0.1.22
v0.1.21
v0.1.20
v0.1.19
v0.1.18
v0.1.17
v0.1.16
v0.1.15
v0.1.14
v0.1.13
v0.1.12
v0.1.11
v0.1.10
v0.1.9
v0.1.8
v0.1.7
v0.1.6
v0.1.5
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
v0.0.21
v0.0.20
v0.0.19
v0.0.18
v0.0.17
v0.0.16
v0.0.15
v0.0.14
v0.0.13
v0.0.12
v0.0.11
v0.0.10
v0.0.9
v0.0.8
v0.0.7
v0.0.6
v0.0.5
v0.0.4
v0.0.3
v0.0.2
v0.0.1
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
Mirrored from GitHub Pull Request
No Label
bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/ollama#52908
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @AlbertoSinigaglia on GitHub (Mar 15, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9782
What is the issue?
This is the model I'm trying to load:
Which is pretty small, however, I'm getting:
Which would be fine, if only I didn't have 80Gb of VRAM free (running
nvitop):The first 2 GPUs are pretty much free. Ollama is not restricted to any of these GPUs, indeed usually it also manages to shard the big models across these three GPUs. At this point, I'm thinking that it wants to allocate the model in the only GPU that is already occupied...
Side notes:
OS
Ubuntu 22.04
GPU
3x
NVIDIA RTX A6000 48GbCPU
AMD Ryzen Threadripper PRO 5995WX 64-CoresOllama version
ollama version is 0.5.13@AlbertoSinigaglia commented on GitHub (Mar 15, 2025):
I've updated to the new 0.6.0, which solved the issue, though also 0.5.13 was running fine until this morning, so I feel that it's not quite a "static" bug, but more of a "Running" bug... let you know if that happens again
@rick-github commented on GitHub (Mar 16, 2025):
If you still have the log, could you include some of the lines from before the crash? It might give some insight into what it was doing when the alloc failed.
@demiurger commented on GitHub (Mar 16, 2025):
Have the same issue: model starts, if prompt is short it works, if promt is like few sentence ollama crashes with OOM.
Have 2 1080Ti 11GB. Same results with 0.6.0
app.log
server.log
@AlbertoSinigaglia commented on GitHub (Mar 16, 2025):
Mmm not sure, what should i run to get them? (Or in which folder are they stored?)
@rick-github commented on GitHub (Mar 16, 2025):
Where ever you found the
Error: llama runner process has terminated: CUDA error: out of memoryline. A few lines above that will either showGINlines, or hopefully some lines starting withggmlwhich may give insight into what was happening. either that or just include the output ofsudo journalctl -u ollama --no-pager --since=yesterday@rick-github commented on GitHub (Mar 16, 2025):
@demiurger Something strange is happening.
ollama needs 23G to host the model. You don't have enough VRAM to hold it all, so ollama allocates 18.4G of the available 18.6G of VRAM and offloads 49 of 63 layers to the GPU.
It successfully loads the model and completes a
generateinference.But then decides it suddenly needs another 13G of VRAM. It's like some allocator deep in ollama/llama.cpp got MB and GB mixed up. Do you recall what the query was that caused the OOM?
@AlbertoSinigaglia commented on GitHub (Mar 16, 2025):
@rick-github here's one of the logs
logs from ollama
@rick-github commented on GitHub (Mar 16, 2025):
The first model load in the log that caused the crash is not cas/nous-hermes-2-mistral-7b-dpo:latest (sha256 of 3f99518c1e2c1b2cee14c3cd7c110358ceb89cf2be0be0626d11ebd8571ff0ff).
The model requires 35G VRAM of 47G available, which would normally leave plenty of room for subsequent allocations.
However, it crashed during the model load, so this is a different failure to the one that @demiurger experienced. For this sort of error I usually recommend some mitigations, but 12G free should really be enough so it's not clear to me why this failed. Can you supply the model name?
The second model load is phi4:14b and works fine.
@AlbertoSinigaglia commented on GitHub (Mar 16, 2025):
@rick-github not sure which model name you are referring to,
cas/nous-hermes-2-mistral-7b-dpo:latestis the one I tried, but later I've also tried other models, with the same error (I've uploaded only one of the logs). Phi4:14b run withough problem because we always keep that in memory with a cronjob.Furthermore, that
cas/nous-hermes-2-mistral-7b-dpo:latestis extremely small:So it would for sure fit in the 35Gb of GPU mem remaining in cuda:0 and cuda:1
@demiurger commented on GitHub (Mar 16, 2025):
@rick-github I used CLI, and entered the following prompt:
C:\Users\hpc> ollama run gemma3:27b
As I said before, it always happens if prompt (or context, sorry I am just a user, not a pro) is more than a few words. If I say something like: "Hello" works, but after a few queries it stops working.
@rick-github commented on GitHub (Mar 16, 2025):
@AlbertoSinigaglia
I agree that cas/nous-hermes-2-mistral-7b-dpo:latest will fit in the available memory. The model that caused the crash in your log was not cas/nous-hermes-2-mistral-7b-dpo:latest, it was significantly larger with model weights of around 32G, not the 4.4G of nouse-hermes.
@AlbertoSinigaglia commented on GitHub (Mar 16, 2025):
@rick-github so i happen to still have the whole nvitop screen:

As you can see, the only loaded model was phi4, the 10Gb vram process. The 30Gb-ish model was either Qwen coder or QwQ-32B (most likely)
Not sure if this makes a difference, in case let me know and i'll upload a file with the whole log of yesterday morning (and not just one failure between the many I tried, to see if it was a single model that was causing that problem or if all of them had it)
@AlbertoSinigaglia commented on GitHub (Mar 16, 2025):
Also (maybe useless, but worth saying i guess), ollama usually shards models across gpus if they can't fit in a single one, so i'm not sure why it was happening... my only guess was that cuda:2 that was used almost to a saturation point by another user, and thar maybe ollama wanted some mem from that one lol
@nickheyer commented on GitHub (Mar 25, 2025):
Same thing is happening to me with a 48gb card and a relatively large model. I can "fix" this by setting num_gpu to a static number, but it kills performance and concurrency.
@tjwebb commented on GitHub (Apr 13, 2025):
I am also seeing this occasionally with 0.6.5 when loading gemma3:27b into an L40 with 48GB of VRAM. I cannot reproduce consistently.
@rick-github commented on GitHub (Apr 13, 2025):
Server logs will aid in diagnosis.
@JMuff22 commented on GitHub (Apr 21, 2025):
I have a similar issue running Deepseek 671B. The error only occurs if I set
OLLAMA_FLASH_ATTENTION=1on startup. Otherwise I have no issue. Does this help?Logs
time=2025-04-21T12:29:51.160+03:00 level=INFO source=sched.go:509 msg="updated VRAM based on existing loaded models" gpu=GPU-bc3bedc1-3249-b2ab-e15f-cf6ba3e4b329 library=cuda total="79.2 GiB" available="68.3 GiB" time=2025-04-21T12:29:51.160+03:00 level=INFO source=sched.go:509 msg="updated VRAM based on existing loaded models" gpu=GPU-76ddd618-f2df-feab-66aa-f85ce8e5ebc8 library=cuda total="79.2 GiB" available="35.6 GiB" time=2025-04-21T12:29:51.161+03:00 level=INFO source=sched.go:509 msg="updated VRAM based on existing loaded models" gpu=GPU-6d2eef79-9a59-4516-6456-5b14fbb8681e library=cuda total="79.2 GiB" available="57.6 GiB" time=2025-04-21T12:29:51.161+03:00 level=INFO source=sched.go:509 msg="updated VRAM based on existing loaded models" gpu=GPU-86ad5f8b-7ba9-7c70-20de-b6ef5d350cf9 library=cuda total="79.2 GiB" available="57.6 GiB" time=2025-04-21T12:29:51.161+03:00 level=INFO source=sched.go:509 msg="updated VRAM based on existing loaded models" gpu=GPU-530159d7-95ea-c8f8-7a4c-871a99cab07b library=cuda total="79.2 GiB" available="35.6 GiB" time=2025-04-21T12:29:51.161+03:00 level=INFO source=sched.go:509 msg="updated VRAM based on existing loaded models" gpu=GPU-11f7cd34-11a2-96ed-e7e8-452c48885d47 library=cuda total="79.2 GiB" available="78.7 GiB" time=2025-04-21T12:29:51.161+03:00 level=INFO source=sched.go:509 msg="updated VRAM based on existing loaded models" gpu=GPU-4b2d7293-9b58-7a25-c59a-57cba53a232b library=cuda total="79.2 GiB" available="78.7 GiB" time=2025-04-21T12:29:51.161+03:00 level=INFO source=sched.go:509 msg="updated VRAM based on existing loaded models" gpu=GPU-616e70b1-2ea8-0707-bf19-b814e054f853 library=cuda total="79.2 GiB" available="78.7 GiB" time=2025-04-21T12:29:51.161+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:29:54.249+03:00 level=WARN source=sched.go:648 msg="gpu VRAM usage didn't recover within timeout" seconds=10.313002755 model=~/.ollama/models/blobs/sha256-d83c18fb2a2ccca56d641920f01e6fe533dfb3fcf4b77c007c931497cd24a517 time=2025-04-21T12:29:57.363+03:00 level=WARN source=sched.go:648 msg="gpu VRAM usage didn't recover within timeout" seconds=13.426501454 model=~/.ollama/models/blobs/sha256-d83c18fb2a2ccca56d641920f01e6fe533dfb3fcf4b77c007c931497cd24a517 time=2025-04-21T12:30:00.475+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:30:03.583+03:00 level=WARN source=sched.go:648 msg="gpu VRAM usage didn't recover within timeout" seconds=19.647088021 model=~/.ollama/models/blobs/sha256-d83c18fb2a2ccca56d641920f01e6fe533dfb3fcf4b77c007c931497cd24a517 time=2025-04-21T12:30:06.844+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:30:09.978+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:30:13.119+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:30:16.357+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:30:19.620+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:30:22.744+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:30:25.949+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:30:29.094+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:30:32.193+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:30:35.261+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:30:38.534+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:30:41.687+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:30:44.811+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:30:48.041+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:30:51.271+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:30:54.416+03:00 level=INFO source=sched.go:732 msg="new model will fit in available VRAM, loading" model=~/.ollama/models/blobs/sha256-d83c18fb2a2ccca56d641920f01e6fe533dfb3fcf4b77c007c931497cd24a517 library=cuda parallel=4 required="449.4 GiB" time=2025-04-21T12:30:57.583+03:00 level=INFO source=server.go:105 msg="system memory" total="1007.4 GiB" free="990.2 GiB" free_swap="9.0 GiB" time=2025-04-21T12:30:57.583+03:00 level=WARN source=ggml.go:152 msg="key not found" key=deepseek2.vision.block_count default=0 time=2025-04-21T12:31:00.791+03:00 level=INFO source=server.go:138 msg=offload library=cuda layers.requested=-1 layers.model=62 layers.offload=62 layers.split=10,11,11,8,7,7,4,4 memory.available="[78.7 GiB 78.7 GiB 78.7 GiB 68.3 GiB 57.6 GiB 57.6 GiB 35.6 GiB 35.6 GiB]" memory.gpu_overhead="0 B" memory.required.full="449.4 GiB" memory.required.partial="449.4 GiB" memory.required.kv="38.1 GiB" memory.required.allocations="[70.7 GiB 69.6 GiB 75.6 GiB 61.3 GiB 54.6 GiB 52.8 GiB 32.8 GiB 31.9 GiB]" memory.weights.total="376.2 GiB" memory.weights.repeating="375.5 GiB" memory.weights.nonrepeating="725.0 MiB" memory.graph.full="3.0 GiB" memory.graph.partial="3.0 GiB" time=2025-04-21T12:31:00.791+03:00 level=WARN source=server.go:178 msg="flash attention enabled but not supported by model" llama_model_loader: loaded meta data with 42 key-value pairs and 1025 tensors from ~/.ollama/models/blobs/sha256-d83c18fb2a2ccca56d641920f01e6fe533dfb3fcf4b77c007c931497cd24a517 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = deepseek2 llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.size_label str = 256x20B llama_model_loader: - kv 3: deepseek2.block_count u32 = 61 llama_model_loader: - kv 4: deepseek2.context_length u32 = 163840 llama_model_loader: - kv 5: deepseek2.embedding_length u32 = 7168 llama_model_loader: - kv 6: deepseek2.feed_forward_length u32 = 18432 llama_model_loader: - kv 7: deepseek2.attention.head_count u32 = 128 llama_model_loader: - kv 8: deepseek2.attention.head_count_kv u32 = 128 llama_model_loader: - kv 9: deepseek2.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 10: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 11: deepseek2.expert_used_count u32 = 8 llama_model_loader: - kv 12: deepseek2.leading_dense_block_count u32 = 3 llama_model_loader: - kv 13: deepseek2.vocab_size u32 = 129280 llama_model_loader: - kv 14: deepseek2.attention.q_lora_rank u32 = 1536 llama_model_loader: - kv 15: deepseek2.attention.kv_lora_rank u32 = 512 llama_model_loader: - kv 16: deepseek2.attention.key_length u32 = 192 llama_model_loader: - kv 17: deepseek2.attention.value_length u32 = 128 llama_model_loader: - kv 18: deepseek2.expert_feed_forward_length u32 = 2048 llama_model_loader: - kv 19: deepseek2.expert_count u32 = 256 llama_model_loader: - kv 20: deepseek2.expert_shared_count u32 = 1 llama_model_loader: - kv 21: deepseek2.expert_weights_scale f32 = 2.500000 llama_model_loader: - kv 22: deepseek2.expert_weights_norm bool = true llama_model_loader: - kv 23: deepseek2.expert_gating_func u32 = 2 llama_model_loader: - kv 24: deepseek2.rope.dimension_count u32 = 64 llama_model_loader: - kv 25: deepseek2.rope.scaling.type str = yarn llama_model_loader: - kv 26: deepseek2.rope.scaling.factor f32 = 40.000000 llama_model_loader: - kv 27: deepseek2.rope.scaling.original_context_length u32 = 4096 llama_model_loader: - kv 28: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000 llama_model_loader: - kv 29: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 30: tokenizer.ggml.pre str = deepseek-v3 llama_model_loader: - kv 31: tokenizer.ggml.tokens arr[str,129280] = ["<|begin▁of▁sentence|>", "<�... llama_model_loader: - kv 32: tokenizer.ggml.token_type arr[i32,129280] = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 33: tokenizer.ggml.merges arr[str,127741] = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e... llama_model_loader: - kv 34: tokenizer.ggml.bos_token_id u32 = 0 llama_model_loader: - kv 35: tokenizer.ggml.eos_token_id u32 = 1 llama_model_loader: - kv 36: tokenizer.ggml.padding_token_id u32 = 1 llama_model_loader: - kv 37: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 38: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 39: tokenizer.chat_template str = {% if not add_generation_prompt is de... llama_model_loader: - kv 40: general.quantization_version u32 = 2 llama_model_loader: - kv 41: general.file_type u32 = 15 llama_model_loader: - type f32: 361 tensors llama_model_loader: - type q4_K: 606 tensors llama_model_loader: - type q6_K: 58 tensors print_info: file format = GGUF V3 (latest) print_info: file type = Q4_K - Medium print_info: file size = 376.65 GiB (4.82 BPW) load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect load: special tokens cache size = 818 load: token to piece cache size = 0.8223 MB print_info: arch = deepseek2 print_info: vocab_only = 1 print_info: model type = ?B print_info: model params = 671.03 B print_info: general.name = n/a print_info: n_layer_dense_lead = 0 print_info: n_lora_q = 0 print_info: n_lora_kv = 0 print_info: n_ff_exp = 0 print_info: n_expert_shared = 0 print_info: expert_weights_scale = 0.0 print_info: expert_weights_norm = 0 print_info: expert_gating_func = unknown print_info: rope_yarn_log_mul = 0.0000 print_info: vocab type = BPE print_info: n_vocab = 129280 print_info: n_merges = 127741 print_info: BOS token = 0 '<|begin▁of▁sentence|>' print_info: EOS token = 1 '<|end▁of▁sentence|>' print_info: EOT token = 1 '<|end▁of▁sentence|>' print_info: PAD token = 1 '<|end▁of▁sentence|>' print_info: LF token = 201 'Ċ' print_info: FIM PRE token = 128801 '<|fim▁begin|>' print_info: FIM SUF token = 128800 '<|fim▁hole|>' print_info: FIM MID token = 128802 '<|fim▁end|>' print_info: EOG token = 1 '<|end▁of▁sentence|>' print_info: max token length = 256 llama_model_load: vocab only - skipping tensors time=2025-04-21T12:31:00.947+03:00 level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/bin/ollama runner --model ~/.ollama/models/blobs/sha256-d83c18fb2a2ccca56d641920f01e6fe533dfb3fcf4b77c007c931497cd24a517 --ctx-size 8192 --batch-size 512 --n-gpu-layers 62 --threads 96 --parallel 4 --tensor-split 10,11,11,8,7,7,4,4 --port 37411" time=2025-04-21T12:31:00.947+03:00 level=INFO source=sched.go:451 msg="loaded runners" count=6 time=2025-04-21T12:31:00.947+03:00 level=INFO source=server.go:580 msg="waiting for llama runner to start responding" time=2025-04-21T12:31:00.947+03:00 level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server error" time=2025-04-21T12:31:00.959+03:00 level=INFO source=runner.go:853 msg="starting go runner" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 8 CUDA devices: Device 0: NVIDIA H100 80GB HBM3, compute capability 9.0, VMM: yes Device 1: NVIDIA H100 80GB HBM3, compute capability 9.0, VMM: yes Device 2: NVIDIA H100 80GB HBM3, compute capability 9.0, VMM: yes Device 3: NVIDIA H100 80GB HBM3, compute capability 9.0, VMM: yes Device 4: NVIDIA H100 80GB HBM3, compute capability 9.0, VMM: yes Device 5: NVIDIA H100 80GB HBM3, compute capability 9.0, VMM: yes Device 6: NVIDIA H100 80GB HBM3, compute capability 9.0, VMM: yes Device 7: NVIDIA H100 80GB HBM3, compute capability 9.0, VMM: yes load_backend: loaded CUDA backend from /usr/lib/ollama/cuda_v12/libggml-cuda.so load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-icelake.so time=2025-04-21T12:31:03.531+03:00 level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 CUDA.2.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.2.USE_GRAPHS=1 CUDA.2.PEER_MAX_BATCH_SIZE=128 CUDA.3.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.3.USE_GRAPHS=1 CUDA.3.PEER_MAX_BATCH_SIZE=128 CUDA.4.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.4.USE_GRAPHS=1 CUDA.4.PEER_MAX_BATCH_SIZE=128 CUDA.5.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.5.USE_GRAPHS=1 CUDA.5.PEER_MAX_BATCH_SIZE=128 CUDA.6.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.6.USE_GRAPHS=1 CUDA.6.PEER_MAX_BATCH_SIZE=128 CUDA.7.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.7.USE_GRAPHS=1 CUDA.7.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) time=2025-04-21T12:31:03.532+03:00 level=INFO source=runner.go:913 msg="Server listening on 127.0.0.1:37411" time=2025-04-21T12:31:03.706+03:00 level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server loading model" llama_model_load_from_file_impl: using device CUDA0 (NVIDIA H100 80GB HBM3) - 80580 MiB free llama_model_load_from_file_impl: using device CUDA1 (NVIDIA H100 80GB HBM3) - 80580 MiB free llama_model_load_from_file_impl: using device CUDA2 (NVIDIA H100 80GB HBM3) - 80580 MiB free llama_model_load_from_file_impl: using device CUDA3 (NVIDIA H100 80GB HBM3) - 69911 MiB free llama_model_load_from_file_impl: using device CUDA4 (NVIDIA H100 80GB HBM3) - 59033 MiB free llama_model_load_from_file_impl: using device CUDA5 (NVIDIA H100 80GB HBM3) - 59033 MiB free llama_model_load_from_file_impl: using device CUDA6 (NVIDIA H100 80GB HBM3) - 37089 MiB free llama_model_load_from_file_impl: using device CUDA7 (NVIDIA H100 80GB HBM3) - 37089 MiB free llama_model_loader: loaded meta data with 42 key-value pairs and 1025 tensors from ~/.ollama/models/blobs/sha256-d83c18fb2a2ccca56d641920f01e6fe533dfb3fcf4b77c007c931497cd24a517 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = deepseek2 llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.size_label str = 256x20B llama_model_loader: - kv 3: deepseek2.block_count u32 = 61 llama_model_loader: - kv 4: deepseek2.context_length u32 = 163840 llama_model_loader: - kv 5: deepseek2.embedding_length u32 = 7168 llama_model_loader: - kv 6: deepseek2.feed_forward_length u32 = 18432 llama_model_loader: - kv 7: deepseek2.attention.head_count u32 = 128 llama_model_loader: - kv 8: deepseek2.attention.head_count_kv u32 = 128 llama_model_loader: - kv 9: deepseek2.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 10: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 11: deepseek2.expert_used_count u32 = 8 llama_model_loader: - kv 12: deepseek2.leading_dense_block_count u32 = 3 llama_model_loader: - kv 13: deepseek2.vocab_size u32 = 129280 llama_model_loader: - kv 14: deepseek2.attention.q_lora_rank u32 = 1536 llama_model_loader: - kv 15: deepseek2.attention.kv_lora_rank u32 = 512 llama_model_loader: - kv 16: deepseek2.attention.key_length u32 = 192 llama_model_loader: - kv 17: deepseek2.attention.value_length u32 = 128 llama_model_loader: - kv 18: deepseek2.expert_feed_forward_length u32 = 2048 llama_model_loader: - kv 19: deepseek2.expert_count u32 = 256 llama_model_loader: - kv 20: deepseek2.expert_shared_count u32 = 1 llama_model_loader: - kv 21: deepseek2.expert_weights_scale f32 = 2.500000 llama_model_loader: - kv 22: deepseek2.expert_weights_norm bool = true llama_model_loader: - kv 23: deepseek2.expert_gating_func u32 = 2 llama_model_loader: - kv 24: deepseek2.rope.dimension_count u32 = 64 llama_model_loader: - kv 25: deepseek2.rope.scaling.type str = yarn llama_model_loader: - kv 26: deepseek2.rope.scaling.factor f32 = 40.000000 llama_model_loader: - kv 27: deepseek2.rope.scaling.original_context_length u32 = 4096 llama_model_loader: - kv 28: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000 llama_model_loader: - kv 29: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 30: tokenizer.ggml.pre str = deepseek-v3 llama_model_loader: - kv 31: tokenizer.ggml.tokens arr[str,129280] = ["<|begin▁of▁sentence|>", "<�... llama_model_loader: - kv 32: tokenizer.ggml.token_type arr[i32,129280] = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 33: tokenizer.ggml.merges arr[str,127741] = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e... llama_model_loader: - kv 34: tokenizer.ggml.bos_token_id u32 = 0 llama_model_loader: - kv 35: tokenizer.ggml.eos_token_id u32 = 1 llama_model_loader: - kv 36: tokenizer.ggml.padding_token_id u32 = 1 llama_model_loader: - kv 37: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 38: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 39: tokenizer.chat_template str = {% if not add_generation_prompt is de... llama_model_loader: - kv 40: general.quantization_version u32 = 2 llama_model_loader: - kv 41: general.file_type u32 = 15 llama_model_loader: - type f32: 361 tensors llama_model_loader: - type q4_K: 606 tensors llama_model_loader: - type q6_K: 58 tensors print_info: file format = GGUF V3 (latest) print_info: file type = Q4_K - Medium print_info: file size = 376.65 GiB (4.82 BPW) load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect load: special tokens cache size = 818 load: token to piece cache size = 0.8223 MB print_info: arch = deepseek2 print_info: vocab_only = 0 print_info: n_ctx_train = 163840 print_info: n_embd = 7168 print_info: n_layer = 61 print_info: n_head = 128 print_info: n_head_kv = 128 print_info: n_rot = 64 print_info: n_swa = 0 print_info: n_embd_head_k = 192 print_info: n_embd_head_v = 128 print_info: n_gqa = 1 print_info: n_embd_k_gqa = 24576 print_info: n_embd_v_gqa = 16384 print_info: f_norm_eps = 0.0e+00 print_info: f_norm_rms_eps = 1.0e-06 print_info: f_clamp_kqv = 0.0e+00 print_info: f_max_alibi_bias = 0.0e+00 print_info: f_logit_scale = 0.0e+00 print_info: n_ff = 18432 print_info: n_expert = 256 print_info: n_expert_used = 8 print_info: causal attn = 1 print_info: pooling type = 0 print_info: rope type = 0 print_info: rope scaling = yarn print_info: freq_base_train = 10000.0 print_info: freq_scale_train = 0.025 print_info: n_ctx_orig_yarn = 4096 print_info: rope_finetuned = unknown print_info: ssm_d_conv = 0 print_info: ssm_d_inner = 0 print_info: ssm_d_state = 0 print_info: ssm_dt_rank = 0 print_info: ssm_dt_b_c_rms = 0 print_info: model type = 671B print_info: model params = 671.03 B print_info: general.name = n/a print_info: n_layer_dense_lead = 3 print_info: n_lora_q = 1536 print_info: n_lora_kv = 512 print_info: n_ff_exp = 2048 print_info: n_expert_shared = 1 print_info: expert_weights_scale = 2.5 print_info: expert_weights_norm = 1 print_info: expert_gating_func = sigmoid print_info: rope_yarn_log_mul = 0.1000 print_info: vocab type = BPE print_info: n_vocab = 129280 print_info: n_merges = 127741 print_info: BOS token = 0 '<|begin▁of▁sentence|>' print_info: EOS token = 1 '<|end▁of▁sentence|>' print_info: EOT token = 1 '<|end▁of▁sentence|>' print_info: PAD token = 1 '<|end▁of▁sentence|>' print_info: LF token = 201 'Ċ' print_info: FIM PRE token = 128801 '<|fim▁begin|>' print_info: FIM SUF token = 128800 '<|fim▁hole|>' print_info: FIM MID token = 128802 '<|fim▁end|>' print_info: EOG token = 1 '<|end▁of▁sentence|>' print_info: max token length = 256 load_tensors: loading model tensors, this can take a while... (mmap = true) load_tensors: offloading 61 repeating layers to GPU load_tensors: offloading output layer to GPU load_tensors: offloaded 62/62 layers to GPU load_tensors: CUDA0 model buffer size = 48928.09 MiB load_tensors: CUDA1 model buffer size = 70752.48 MiB load_tensors: CUDA2 model buffer size = 71680.09 MiB load_tensors: CUDA3 model buffer size = 52215.30 MiB load_tensors: CUDA4 model buffer size = 45108.64 MiB load_tensors: CUDA5 model buffer size = 46036.25 MiB load_tensors: CUDA6 model buffer size = 28426.68 MiB load_tensors: CUDA7 model buffer size = 22044.99 MiB load_tensors: CPU_Mapped model buffer size = 497.11 MiB llama_init_from_model: n_seq_max = 4 llama_init_from_model: n_ctx = 8192 llama_init_from_model: n_ctx_per_seq = 2048 llama_init_from_model: n_batch = 2048 llama_init_from_model: n_ubatch = 512 llama_init_from_model: flash_attn = 0 llama_init_from_model: freq_base = 10000.0 llama_init_from_model: freq_scale = 0.025 llama_init_from_model: n_ctx_per_seq (2048) < n_ctx_train (163840) -- the full capacity of the model will not be utilized llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 61, can_shift = 0 llama_kv_cache_init: CUDA0 KV buffer size = 6400.00 MiB llama_kv_cache_init: CUDA1 KV buffer size = 7040.00 MiB llama_kv_cache_init: CUDA2 KV buffer size = 7040.00 MiB llama_kv_cache_init: CUDA3 KV buffer size = 5120.00 MiB llama_kv_cache_init: CUDA4 KV buffer size = 4480.00 MiB llama_kv_cache_init: CUDA5 KV buffer size = 4480.00 MiB llama_kv_cache_init: CUDA6 KV buffer size = 2560.00 MiB llama_kv_cache_init: CUDA7 KV buffer size = 1920.00 MiB llama_init_from_model: KV self size = 39040.00 MiB, K (f16): 23424.00 MiB, V (f16): 15616.00 MiB llama_init_from_model: CUDA_Host output buffer size = 2.08 MiB llama_init_from_model: pipeline parallelism enabled (n_copies=4) ggml_backend_cuda_buffer_type_alloc_buffer: allocating 2322.01 MiB on device 2: cudaMalloc failed: out of memory ggml_gallocr_reserve_n: failed to allocate CUDA2 buffer of size 2434801664 ggml_backend_cuda_buffer_type_alloc_buffer: allocating 2322.01 MiB on device 2: cudaMalloc failed: out of memory ggml_gallocr_reserve_n: failed to allocate CUDA2 buffer of size 2434801664 llama_init_from_model: failed to allocate compute buffers panic: unable to create llama contextgoroutine 111 [running]:
github.com/ollama/ollama/runner/llamarunner.(*Server).loadModel(0xc000544360, {0x3e, 0x0, 0x1, 0x0, {0xc0005010c0, 0x8, 0x8}, 0xc00055e1a0, 0x0}, ...)
github.com/ollama/ollama/runner/llamarunner/runner.go:779 +0x369
created by github.com/ollama/ollama/runner/llamarunner.Execute in goroutine 1
github.com/ollama/ollama/runner/llamarunner/runner.go:887 +0xbd7
time=2025-04-21T12:32:01.411+03:00 level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server error"
time=2025-04-21T12:32:02.192+03:00 level=ERROR source=server.go:449 msg="llama runner terminated" error="exit status 2"
time=2025-04-21T12:32:02.413+03:00 level=ERROR source=sched.go:457 msg="error loading llama server" error="llama runner process has terminated: cudaMalloc failed: out of memory"
[GIN] 2025/04/21 - 12:32:02 | 500 | 2m18s | 127.0.0.1 | POST "/api/generate"
Generation error: llama runner process has terminated: cudaMalloc failed: out of memory (status code: 500)
@rick-github commented on GitHub (Apr 21, 2025):
Setting
OLLAMA_FLASH_ATTENTION=1should have no effect. It would be interesting to see the logs from whenOLLAMA_FLASH_ATTENTIONis not set.ollama OOMed allocating VRAM on device 2.
Device 2 had the largest allocation, 75.6G out of 78.7G available. It looks like the model failed during loading (rather than inference) so it's likely not a transient allocation that caused the failure. You can try reducing the occurrence of this using some of the mitigations listed here. Improving the memory estimation is a work in progress.
@carbolymer commented on GitHub (Apr 29, 2025):
I have RTX 3050 Ti mobile GPU with 4GB RAM. I can run larger models like
qwen3:14b(mostly on CPU) just fine, but this one for example errors out:Log
@rick-github commented on GitHub (Apr 29, 2025):
ollama allocated 3.4G of the 3.4G available, ie left no room for error. You can mitigate this by using some of the techniques shown here.