github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 08:02:14 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #8667] deepseek-r1:671b Q4_K_M: error="model requires more system memory (446.3 GiB) than is available #52127

New Issue
Closed
opened 2026-04-28 22:12:45 -05:00 by GiteaMirror · 32 comments
GiteaMirror commented 2026-04-28 22:12:45 -05:00
Owner
Copy Link

Originally created by @philippstoboy on GitHub (Jan 29, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/8667

Hey Ollama community,

I’m reaching out for some advice on running the DeepSeek-R1 671B model with Q4 quantization on my current setup, which has 40GB of RAM. I understand that this model employs a Mixture of Experts (MoE) architecture, meaning that during inference, only a subset of the model’s parameters (approximately 37 billion) is active at any given time. This design should, in theory, reduce the immediate memory requirements compared to loading all 671 billion parameters simultaneously. Based on calculations, the active 37B parameters in Q4 quantization should only require around 18.5GB of RAM, with an additional 20–30% memory overhead, resulting in a total memory requirement of ~25–30GB. In contrast, the Q8_0 version (8-bit quantization) would require approximately 37GB + overhead, totaling around 50GB. Even the full 671B parameters in Q4 quantization should only need around 325GB of RAM if loaded entirely, and Q8_0 would need 650GB. Given these figures, my expectation was that loading just the active 37B subset in Q4 should work within reasonable hardware configurations.

However, when I attempt to load the model using Ollama, I encounter an error indicating that the model requires more system memory (446.3 GiB) than is available (37.3 GiB). This is perplexing, given the MoE architecture’s supposed efficiency and the Q4 quantization.

I came across a discussion where a user successfully ran the Q8_0 GGUF version of DeepSeek-R1 671B on a CPU-only system with 256GB of RAM. They noted that with a context length of 32,092 tokens, the system utilized around 220GB of RAM, emphasizing that the KV cache can consume more RAM than the model itself. Reference: https://huggingface.co/deepseek-ai/DeepSeek-R1/discussions/19

Based on this, I would expect that running the Q4 quantized model with a context length of 32,092 tokens would require less memory than the Q8 version, potentially making it feasible to run on a system with 192GB of RAM(What I'm planning to upgrade to). However, the error message I received suggests otherwise, indicating a requirement of 446.3 GiB of system memory.

Thank you for your help.

Error Logs:

2025/01/29 14:31:32 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2025-01-29T14:31:32.660Z level=INFO source=images.go:432 msg="total blobs: 11"
time=2025-01-29T14:31:32.680Z level=INFO source=images.go:439 msg="total unused blobs removed: 0"
time=2025-01-29T14:31:32.699Z level=INFO source=routes.go:1238 msg="Listening on [::]:11434 (version 0.5.7-0-ga420a45-dirty)"
time=2025-01-29T14:31:32.701Z level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cuda_v12_avx cpu cpu_avx cpu_avx2 cuda_v11_avx]"
time=2025-01-29T14:31:32.701Z level=INFO source=gpu.go:226 msg="looking for compatible GPUs"
time=2025-01-29T14:31:32.729Z level=WARN source=amd_linux.go:61 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2025-01-29T14:31:32.730Z level=INFO source=amd_linux.go:404 msg="no compatible amdgpu devices detected"
time=2025-01-29T14:31:32.730Z level=INFO source=gpu.go:392 msg="no compatible GPUs were discovered"
time=2025-01-29T14:31:32.730Z level=INFO source=types.go:131 msg="inference compute" id=0 library=cpu variant=avx compute="" driver=0.0 name="" total="39.2 GiB" available="37.9 GiB"
time=2025-01-29T14:33:04.221Z level=INFO source=server.go:104 msg="system memory" total="39.2 GiB" free="37.3 GiB" free_swap="0 B"
time=2025-01-29T14:33:04.223Z level=WARN source=server.go:136 msg="model request too large for system" requested="446.3 GiB" available=40099082240 total="39.2 GiB" free="37.3 GiB" swap="0 B"
time=2025-01-29T14:33:04.223Z level=INFO source=sched.go:428 msg="NewLlamaServer failed" model=/root/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 error="model requires more system memory (446.3 GiB) than is available (37.3 GiB)"
[GIN-debug] [WARNING]
Originally created by @philippstoboy on GitHub (Jan 29, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/8667 Hey Ollama community, I’m reaching out for some advice on running the DeepSeek-R1 671B model with Q4 quantization on my current setup, which has 40GB of RAM. I understand that this model employs a Mixture of Experts (MoE) architecture, meaning that during inference, only a subset of the model’s parameters (approximately 37 billion) is active at any given time. This design should, in theory, reduce the immediate memory requirements compared to loading all 671 billion parameters simultaneously. Based on calculations, the active 37B parameters in Q4 quantization should only require around **18.5GB of RAM**, with an additional **20–30% memory overhead**, resulting in a total memory requirement of **~25–30GB**. In contrast, the Q8_0 version (8-bit quantization) would require approximately **37GB + overhead**, totaling around **50GB**. Even the full 671B parameters in Q4 quantization should only need around **325GB of RAM** if loaded entirely, and Q8_0 would need **650GB**. Given these figures, my expectation was that loading just the active 37B subset in Q4 should work within reasonable hardware configurations. However, when I attempt to load the model using Ollama, I encounter an error indicating that the model requires more system memory (446.3 GiB) than is available (37.3 GiB). This is perplexing, given the MoE architecture’s supposed efficiency and the Q4 quantization. I came across a discussion where a user successfully ran the Q8_0 GGUF version of DeepSeek-R1 671B on a CPU-only system with 256GB of RAM. They noted that with a context length of 32,092 tokens, the system utilized around 220GB of RAM, emphasizing that the KV cache can consume more RAM than the model itself. Reference: https://huggingface.co/deepseek-ai/DeepSeek-R1/discussions/19 Based on this, I would expect that running the Q4 quantized model with a context length of 32,092 tokens would require less memory than the Q8 version, potentially making it feasible to run on a system with 192GB of RAM(What I'm planning to upgrade to). However, the error message I received suggests otherwise, indicating a requirement of 446.3 GiB of system memory. Thank you for your help. Error Logs: ``` 2025/01/29 14:31:32 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2025-01-29T14:31:32.660Z level=INFO source=images.go:432 msg="total blobs: 11" time=2025-01-29T14:31:32.680Z level=INFO source=images.go:439 msg="total unused blobs removed: 0" time=2025-01-29T14:31:32.699Z level=INFO source=routes.go:1238 msg="Listening on [::]:11434 (version 0.5.7-0-ga420a45-dirty)" time=2025-01-29T14:31:32.701Z level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cuda_v12_avx cpu cpu_avx cpu_avx2 cuda_v11_avx]" time=2025-01-29T14:31:32.701Z level=INFO source=gpu.go:226 msg="looking for compatible GPUs" time=2025-01-29T14:31:32.729Z level=WARN source=amd_linux.go:61 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2025-01-29T14:31:32.730Z level=INFO source=amd_linux.go:404 msg="no compatible amdgpu devices detected" time=2025-01-29T14:31:32.730Z level=INFO source=gpu.go:392 msg="no compatible GPUs were discovered" time=2025-01-29T14:31:32.730Z level=INFO source=types.go:131 msg="inference compute" id=0 library=cpu variant=avx compute="" driver=0.0 name="" total="39.2 GiB" available="37.9 GiB" time=2025-01-29T14:33:04.221Z level=INFO source=server.go:104 msg="system memory" total="39.2 GiB" free="37.3 GiB" free_swap="0 B" time=2025-01-29T14:33:04.223Z level=WARN source=server.go:136 msg="model request too large for system" requested="446.3 GiB" available=40099082240 total="39.2 GiB" free="37.3 GiB" swap="0 B" time=2025-01-29T14:33:04.223Z level=INFO source=sched.go:428 msg="NewLlamaServer failed" model=/root/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 error="model requires more system memory (446.3 GiB) than is available (37.3 GiB)" [GIN-debug] [WARNING] ```
GiteaMirror commented 2026-04-28 22:12:50 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 29, 2025):

MoE reduces the computation required, but the model weights still need to be loaded in order to select the expert. What you are looking for is SwapMoE which, to my knowledge, is not implemented in any mainstream open source inference engine.

You can work around the "more system memory" restriction by creating a bunch of free swap, or tricking ollama into thinking that you have extra resources and relying on mmap to make the model available: https://github.com/ollama/ollama/issues/8654

<!-- gh-comment-id:2622107267 --> @rick-github commented on GitHub (Jan 29, 2025): MoE reduces the computation required, but the model weights still need to be loaded in order to select the expert. What you are looking for is [SwapMoE](https://aclanthology.org/2024.acl-long.363.pdf) which, to my knowledge, is not implemented in any mainstream open source inference engine. You can work around the "more system memory" restriction by creating a bunch of free swap, or tricking ollama into thinking that you have extra resources and relying on mmap to make the model available: https://github.com/ollama/ollama/issues/8654
GiteaMirror commented 2026-04-28 22:12:51 -05:00
Author
Owner
Copy Link

@philippstoboy commented on GitHub (Jan 29, 2025):

MoE reduces the computation required, but the model weights still need to be loaded in order to select the expert. What you are looking for is SwapMoE which, to my knowledge, is not implemented in any mainstream open source inference engine.

You can work around the "more system memory" restriction by creating a bunch of free swap, or tricking ollama into thinking that you have extra resources and relying on mmap to make the model available: #8654

I just did what was mentioned in the linked Thread and now this comes up:

goroutine 22 [running]: github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc0001441b0, {0x0, 0x0, 0x0, 0x0, {0x0, 0x0, 0x0}, 0xc0001141a0, 0x0}, ...) github.com/ollama/ollama/llama/runner/runner.go:852 +0x3ad created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1 github.com/ollama/ollama/llama/runner/runner.go:970 +0xd0d time=2025-01-29T17:16:31.749Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error loading model: unable to allocate CPU buffer\nllama_load_model_from_file: failed to load model" [GIN] 2025/01/29 - 17:16:31 | 500 | 1.752744738s | 192.168.178.63 | POST "/api/chat"

<!-- gh-comment-id:2622294856 --> @philippstoboy commented on GitHub (Jan 29, 2025): > MoE reduces the computation required, but the model weights still need to be loaded in order to select the expert. What you are looking for is [SwapMoE](https://aclanthology.org/2024.acl-long.363.pdf) which, to my knowledge, is not implemented in any mainstream open source inference engine. > > You can work around the "more system memory" restriction by creating a bunch of free swap, or tricking ollama into thinking that you have extra resources and relying on mmap to make the model available: [#8654](https://github.com/ollama/ollama/issues/8654) I just did what was mentioned in the linked Thread and now this comes up: `goroutine 22 [running]: github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc0001441b0, {0x0, 0x0, 0x0, 0x0, {0x0, 0x0, 0x0}, 0xc0001141a0, 0x0}, ...) github.com/ollama/ollama/llama/runner/runner.go:852 +0x3ad created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1 github.com/ollama/ollama/llama/runner/runner.go:970 +0xd0d time=2025-01-29T17:16:31.749Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error loading model: unable to allocate CPU buffer\nllama_load_model_from_file: failed to load model" [GIN] 2025/01/29 - 17:16:31 | 500 | 1.752744738s | 192.168.178.63 | POST "/api/chat" `
GiteaMirror commented 2026-04-28 22:12:53 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 29, 2025):

More context will show how much it tried to allocate and might give some idea of how to work around it.

But I haven't tried that trick myself yet, so I don't know the fine points of getting it to work.

<!-- gh-comment-id:2622336159 --> @rick-github commented on GitHub (Jan 29, 2025): More context will show how much it tried to allocate and might give some idea of how to work around it. But I haven't tried that trick myself yet, so I don't know the fine points of getting it to work.
GiteaMirror commented 2026-04-28 22:12:56 -05:00
Author
Owner
Copy Link

@outis151 commented on GitHub (Jan 29, 2025):

Image
This is what the memory usage looks like when loading DeepSeek-R1 671B Q4 with mmap and the trick from my issue https://github.com/ollama/ollama/issues/8654. There is a short spike while loading the model, then the memory usage remains less than 12GiB during the generation.
Correction: The ram usage remains at that high value during generation. Misinterpreted the data

In any case its less than what ollama calculates
Note that loading the model takes longer using mmap and you may need to adjust the ollama timeout environment variable.

<!-- gh-comment-id:2622670391 --> @outis151 commented on GitHub (Jan 29, 2025): ![Image](https://github.com/user-attachments/assets/5fbe3b69-475c-4faf-beba-44bb270efa1a) This is what the memory usage looks like when loading DeepSeek-R1 671B Q4 with `mmap` and the trick from my issue https://github.com/ollama/ollama/issues/8654. There is a short spike while loading the model, then the memory usage remains less than 12GiB during the generation. **Correction**: The ram usage remains at that high value during generation. Misinterpreted the data In any case its less than what ollama calculates Note that loading the model takes longer using mmap and you may need to adjust the ollama timeout environment variable.
GiteaMirror commented 2026-04-28 22:13:01 -05:00
Author
Owner
Copy Link

@philippstoboy commented on GitHub (Jan 29, 2025):

Hey, my upgrade of 192GB of Memory just arrived. I'm not quite sure how to correctly implement the things mentioned in Your Issue. Could you go into a bit more detail? I "faked" my memory so that ollama thinks I have 500GB and the only output now is this: root@bc8941192791:/# ollama run deepseek-r1:671b --verbose Error: llama runner process has terminated: error loading model: unable to allocate CPU buffer llama_load_model_from_file: failed to load model root@bc8941192791:/#

I'm confident that the Memory faking worked as this was the output before I did it:

root@bc8941192791:/# ollama run deepseek-r1:671b Error: model requires more system memory (433.6 GiB) than is available (186.6 GiB)

As of mmap I'm not certain if I was able to enable it.

Thanks for the help!

<!-- gh-comment-id:2622820960 --> @philippstoboy commented on GitHub (Jan 29, 2025): Hey, my upgrade of 192GB of Memory just arrived. I'm not quite sure how to correctly implement the things mentioned in [Your Issue](https://github.com/ollama/ollama/issues/8654#issuecomment-2621589918). Could you go into a bit more detail? I "faked" my memory so that ollama thinks I have 500GB and the only output now is this: `root@bc8941192791:/# ollama run deepseek-r1:671b --verbose Error: llama runner process has terminated: error loading model: unable to allocate CPU buffer llama_load_model_from_file: failed to load model root@bc8941192791:/#` I'm confident that the Memory faking worked as this was the output before I did it: `root@bc8941192791:/# ollama run deepseek-r1:671b Error: model requires more system memory (433.6 GiB) than is available (186.6 GiB)` As of `mmap` I'm not certain if I was able to enable it. Thanks for the help!
GiteaMirror commented 2026-04-28 22:13:08 -05:00
Author
Owner
Copy Link

@outis151 commented on GitHub (Jan 29, 2025):

confirm you have enabled mmap by trying with a smaller model and checking that it is not loaded into RAM and that only a tiny fraction of RAM is used as compared to without mmap

<!-- gh-comment-id:2622866779 --> @outis151 commented on GitHub (Jan 29, 2025): confirm you have enabled mmap by trying with a smaller model and checking that it is not loaded into RAM and that only a tiny fraction of RAM is used as compared to without mmap
GiteaMirror commented 2026-04-28 22:13:13 -05:00
Author
Owner
Copy Link

@philippstoboy commented on GitHub (Jan 29, 2025):

confirm you have enabled mmap by trying with a smaller model and checking that it is not loaded into RAM and that only a tiny fraction of RAM is used as compared to without mmap

So I ran this command to test with a smaller model with mmap:
curl http://localhost:11434/api/generate -d '{ "model": "llava:latest", "use_mmap": true }'

And here is the ram result:

Image

docker is the ollama usage here

Now if I just run llava:latest without mmap:
curl http://localhost:11434/api/generate -d '{ "model": "llava:latest" }'

Result:

Image

docker is the ollama usage here

As you can see, it seems to be the same usage. I even have an enviorment Variable :
NVIDIA_VISIBLE_DEVICES=all HOSTNAME=bc8941192791 HOST_CONTAINERNAME=ollama PWD=/ NVIDIA_DRIVER_CAPABILITIES=compute,utility TZ=America/Los_Angeles USE_MMAP=1 HOME=/root LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36: OLLAMA_USE_MMAP=1 OLLAMA_TIMEOUT=30000 OLLAMA_ORIGINS=* LESSCLOSE=/usr/bin/lesspipe %s %s OLLAMA_HOST=0.0.0.0 TERM=xterm LESSOPEN=| /usr/bin/lesspipe %s HOST_OS=Unraid SHLVL=1 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOST_HOSTNAME=Tower OLDPWD=/root _=/usr/bin/printenv

Not sure what to do now.

<!-- gh-comment-id:2622886582 --> @philippstoboy commented on GitHub (Jan 29, 2025): > confirm you have enabled mmap by trying with a smaller model and checking that it is not loaded into RAM and that only a tiny fraction of RAM is used as compared to without mmap So I ran this command to test with a smaller model with mmap: `curl http://localhost:11434/api/generate -d '{ "model": "llava:latest", "use_mmap": true }'` And here is the ram result: <img width="296" alt="Image" src="https://github.com/user-attachments/assets/7e387d91-b5e6-48dc-8641-57ef836b6010" /> `docker is the ollama usage here` Now if I just run `llava:latest` without mmap: `curl http://localhost:11434/api/generate -d '{ "model": "llava:latest" }'` Result: <img width="296" alt="Image" src="https://github.com/user-attachments/assets/1603d232-7078-4300-90d0-939031dc9288" /> `docker is the ollama usage here` As you can see, it seems to be the same usage. I even have an enviorment Variable : `NVIDIA_VISIBLE_DEVICES=all HOSTNAME=bc8941192791 HOST_CONTAINERNAME=ollama PWD=/ NVIDIA_DRIVER_CAPABILITIES=compute,utility TZ=America/Los_Angeles USE_MMAP=1 HOME=/root LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36: OLLAMA_USE_MMAP=1 OLLAMA_TIMEOUT=30000 OLLAMA_ORIGINS=* LESSCLOSE=/usr/bin/lesspipe %s %s OLLAMA_HOST=0.0.0.0 TERM=xterm LESSOPEN=| /usr/bin/lesspipe %s HOST_OS=Unraid SHLVL=1 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOST_HOSTNAME=Tower OLDPWD=/root _=/usr/bin/printenv` Not sure what to do now.
GiteaMirror commented 2026-04-28 22:13:16 -05:00
Author
Owner
Copy Link

@philippstoboy commented on GitHub (Jan 29, 2025):

And btw this is when I try to run deepseek:

root@bc8941192791:/# curl http://localhost:11434/api/generate -d '{ "model": "deepseek-r1:671b","use_mmap": true }' {"error":"llama runner process has terminated: error loading model: unable to allocate CPU buffer\nllama_load_model_from_file: failed to load model"}root@bc8941192791:/#

and this is in the live log:

`llama_model_loader: - kv 34: tokenizer.ggml.bos_token_id u32 = 0
llama_model_loader: - kv 35: tokenizer.ggml.eos_token_id u32 = 1
llama_model_loader: - kv 36: tokenizer.ggml.padding_token_id u32 = 1
llama_model_loader: - kv 37: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 38: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 39: tokenizer.chat_template str = {% if not add_generation_prompt is de...
llama_model_loader: - kv 40: general.quantization_version u32 = 2
llama_model_loader: - kv 41: general.file_type u32 = 15
llama_model_loader: - type f32: 361 tensors
llama_model_loader: - type q4_K: 606 tensors
llama_model_loader: - type q6_K: 58 tensors
llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 818
llm_load_vocab: token to piece cache size = 0.8223 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = deepseek2
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 129280
llm_load_print_meta: n_merges = 127741
llm_load_print_meta: vocab_only = 0
llm_load_print_meta: n_ctx_train = 163840
llm_load_print_meta: n_embd = 7168
llm_load_print_meta: n_layer = 61
llm_load_print_meta: n_head = 128
llm_load_print_meta: n_head_kv = 128
llm_load_print_meta: n_rot = 64
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_embd_head_k = 192
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 1
llm_load_print_meta: n_embd_k_gqa = 24576
llm_load_print_meta: n_embd_v_gqa = 16384
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-06
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 18432
llm_load_print_meta: n_expert = 256
llm_load_print_meta: n_expert_used = 8
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = yarn
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 0.025
llm_load_print_meta: n_ctx_orig_yarn = 4096
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: ssm_dt_b_c_rms = 0
llm_load_print_meta: model type = 671B
llm_load_print_meta: model ftype = Q4_K - Medium
llm_load_print_meta: model params = 671.03 B
llm_load_print_meta: model size = 376.65 GiB (4.82 BPW)
llm_load_print_meta: general.name = n/a
llm_load_print_meta: BOS token = 0 '<｜begin▁of▁sentence｜>'
llm_load_print_meta: EOS token = 1 '<｜end▁of▁sentence｜>'
llm_load_print_meta: EOT token = 1 '<｜end▁of▁sentence｜>'
llm_load_print_meta: PAD token = 1 '<｜end▁of▁sentence｜>'
llm_load_print_meta: LF token = 131 'Ä'
llm_load_print_meta: FIM PRE token = 128801 '<｜fim▁begin｜>'
llm_load_print_meta: FIM SUF token = 128800 '<｜fim▁hole｜>'
llm_load_print_meta: FIM MID token = 128802 '<｜fim▁end｜>'
llm_load_print_meta: EOG token = 1 '<｜end▁of▁sentence｜>'
llm_load_print_meta: max token length = 256
llm_load_print_meta: n_layer_dense_lead = 3
llm_load_print_meta: n_lora_q = 1536
llm_load_print_meta: n_lora_kv = 512
llm_load_print_meta: n_ff_exp = 2048
llm_load_print_meta: n_expert_shared = 1
llm_load_print_meta: expert_weights_scale = 2.5
llm_load_print_meta: expert_weights_norm = 1
llm_load_print_meta: expert_gating_func = sigmoid
llm_load_print_meta: rope_yarn_log_mul = 0.1000
ggml_aligned_malloc: insufficient memory (attempted to allocate 385689.62 MB)
ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 404424884224
llama_model_load: error loading model: unable to allocate CPU buffer
llama_load_model_from_file: failed to load model
panic: unable to load model: /root/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9

goroutine 7 [running]:
github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc0000c41b0, {0x0, 0x0, 0x0, 0x0, {0x0, 0x0, 0x0}, 0xc0000241c0, 0x0}, ...)
github.com/ollama/ollama/llama/runner/runner.go:852 +0x3ad
created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1
github.com/ollama/ollama/llama/runner/runner.go:970 +0xd0d
time=2025-01-29T21:29:13.915Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error loading model: unable to allocate CPU buffer\nllama_load_model_from_file: failed to load model"
[GIN] 2025/01/29 - 21:29:13 | 500 | 2.021031495s | 127.0.0.1 | POST "/api/generate"
`

<!-- gh-comment-id:2622897980 --> @philippstoboy commented on GitHub (Jan 29, 2025): And btw this is when I try to run deepseek: `root@bc8941192791:/# curl http://localhost:11434/api/generate -d '{ "model": "deepseek-r1:671b","use_mmap": true }' {"error":"llama runner process has terminated: error loading model: unable to allocate CPU buffer\nllama_load_model_from_file: failed to load model"}root@bc8941192791:/# ` and this is in the live log: `llama_model_loader: - kv 34: tokenizer.ggml.bos_token_id u32 = 0 llama_model_loader: - kv 35: tokenizer.ggml.eos_token_id u32 = 1 llama_model_loader: - kv 36: tokenizer.ggml.padding_token_id u32 = 1 llama_model_loader: - kv 37: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 38: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 39: tokenizer.chat_template str = {% if not add_generation_prompt is de... llama_model_loader: - kv 40: general.quantization_version u32 = 2 llama_model_loader: - kv 41: general.file_type u32 = 15 llama_model_loader: - type f32: 361 tensors llama_model_loader: - type q4_K: 606 tensors llama_model_loader: - type q6_K: 58 tensors llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect llm_load_vocab: special tokens cache size = 818 llm_load_vocab: token to piece cache size = 0.8223 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = deepseek2 llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 129280 llm_load_print_meta: n_merges = 127741 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 163840 llm_load_print_meta: n_embd = 7168 llm_load_print_meta: n_layer = 61 llm_load_print_meta: n_head = 128 llm_load_print_meta: n_head_kv = 128 llm_load_print_meta: n_rot = 64 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 192 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 1 llm_load_print_meta: n_embd_k_gqa = 24576 llm_load_print_meta: n_embd_v_gqa = 16384 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-06 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 18432 llm_load_print_meta: n_expert = 256 llm_load_print_meta: n_expert_used = 8 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = yarn llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 0.025 llm_load_print_meta: n_ctx_orig_yarn = 4096 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = 671B llm_load_print_meta: model ftype = Q4_K - Medium llm_load_print_meta: model params = 671.03 B llm_load_print_meta: model size = 376.65 GiB (4.82 BPW) llm_load_print_meta: general.name = n/a llm_load_print_meta: BOS token = 0 '<｜begin▁of▁sentence｜>' llm_load_print_meta: EOS token = 1 '<｜end▁of▁sentence｜>' llm_load_print_meta: EOT token = 1 '<｜end▁of▁sentence｜>' llm_load_print_meta: PAD token = 1 '<｜end▁of▁sentence｜>' llm_load_print_meta: LF token = 131 'Ä' llm_load_print_meta: FIM PRE token = 128801 '<｜fim▁begin｜>' llm_load_print_meta: FIM SUF token = 128800 '<｜fim▁hole｜>' llm_load_print_meta: FIM MID token = 128802 '<｜fim▁end｜>' llm_load_print_meta: EOG token = 1 '<｜end▁of▁sentence｜>' llm_load_print_meta: max token length = 256 llm_load_print_meta: n_layer_dense_lead = 3 llm_load_print_meta: n_lora_q = 1536 llm_load_print_meta: n_lora_kv = 512 llm_load_print_meta: n_ff_exp = 2048 llm_load_print_meta: n_expert_shared = 1 llm_load_print_meta: expert_weights_scale = 2.5 llm_load_print_meta: expert_weights_norm = 1 llm_load_print_meta: expert_gating_func = sigmoid llm_load_print_meta: rope_yarn_log_mul = 0.1000 ggml_aligned_malloc: insufficient memory (attempted to allocate 385689.62 MB) ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 404424884224 llama_model_load: error loading model: unable to allocate CPU buffer llama_load_model_from_file: failed to load model panic: unable to load model: /root/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 goroutine 7 [running]: github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc0000c41b0, {0x0, 0x0, 0x0, 0x0, {0x0, 0x0, 0x0}, 0xc0000241c0, 0x0}, ...) github.com/ollama/ollama/llama/runner/runner.go:852 +0x3ad created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1 github.com/ollama/ollama/llama/runner/runner.go:970 +0xd0d time=2025-01-29T21:29:13.915Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error loading model: unable to allocate CPU buffer\nllama_load_model_from_file: failed to load model" [GIN] 2025/01/29 - 21:29:13 | 500 | 2.021031495s | 127.0.0.1 | POST "/api/generate" `
GiteaMirror commented 2026-04-28 22:13:17 -05:00
Author
Owner
Copy Link

@philippstoboy commented on GitHub (Jan 30, 2025):

@outis151 ?

<!-- gh-comment-id:2624526745 --> @philippstoboy commented on GitHub (Jan 30, 2025): @outis151 ?
GiteaMirror commented 2026-04-28 22:13:18 -05:00
Author
Owner
Copy Link

@guiguid commented on GitHub (Feb 1, 2025):

Hi,
Same issue, mmap from SSD is not used with ollama but woks with llama-cli.

llama-cli --model /srv/ollama/blobs/sha256-f4212639f8b6e105df9c2feebc2f8ebe6c1bb5cac3e721051b097a6bca76c183 --cache-type-k q4_0 --threads 24 --prio 2 --temp 0.6 --ctx-size 8192 --seed 3407 --n-gpu-layers 7  -no-cnv --prompt "<｜User｜>Quelle est la meilleure orientation et inclinaison pour les panneaux solaires installés à Blaye.<｜Assistant｜>"
[...]
load_tensors: offloading 7 repeating layers to GPU
load_tensors: offloaded 7/62 layers to GPU
load_tensors:   CPU_Mapped model buffer size = 216099,06 MiB
llama_init_from_model: n_seq_max     = 1
llama_init_from_model: n_ctx         = 8192
llama_init_from_model: n_ctx_per_seq = 8192
llama_init_from_model: n_batch       = 2048
llama_init_from_model: n_ubatch      = 512
llama_init_from_model: flash_attn    = 0
llama_init_from_model: freq_base     = 10000,0
llama_init_from_model: freq_scale    = 0,025
llama_init_from_model: n_ctx_per_seq (8192) < n_ctx_train (163840) -- the full capacity of the model will not be utilized
llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'q4_0', type_v = 'f16', n_layer = 61, can_shift = 0
llama_kv_cache_init:        CPU KV buffer size = 22204,00 MiB
llama_init_from_model: KV self size  = 22204,00 MiB, K (q4_0): 6588,00 MiB, V (f16): 15616,00 MiB
llama_init_from_model:        CPU  output buffer size =     0,49 MiB
llama_init_from_model:        CPU compute buffer size =  2218,01 MiB
[...]
llama_perf_sampler_print:    sampling time =      99,06 ms /   985 runs   (    0,10 ms per token,  9943,47 tokens per second)
llama_perf_context_print:        load time =   73522,94 ms
llama_perf_context_print: prompt eval time =  171156,36 ms /    27 tokens ( 6339,12 ms per token,     0,16 tokens per second)
llama_perf_context_print:        eval time = 18254717,61 ms /   957 runs   (19074,94 ms per token,     0,05 tokens per second)
llama_perf_context_print:       total time = 18426536,71 ms /   984 tokens
[...]
❯ llama-cli --version
register_backend: registered backend BLAS (1 devices)
register_device: registered device BLAS (OpenBLAS)
register_backend: registered backend RPC (0 devices)
register_backend: registered backend CPU (1 devices)
register_device: registered device CPU (AMD Ryzen 9 7900 12-Core Processor)
version: 4598 (27d135c97)
built with cc (GCC) 14.2.1 20240910 for x86_64-pc-linux-gnu

And with Ollama :

❯ curl localhost:11434/api/generate -d '{"model":"Huzderu/deepseek-r1-671b-2.51bit:latest","options":{"num_gpu":0,"use_mmap":true}}'
{"error":"model requires more system memory (257.6 GiB) than is available (60.2 GiB)"}

❯ ollama -v
ollama version is 0.5.7

❯ ollama show Huzderu/deepseek-r1-671b-2.51bit:latest
  Model
    architecture        deepseek2    
    parameters          671.0B       
    context length      163840       
    embedding length    7168         
    quantization        Q2_K         

  Parameters
    stop    "<｜begin▁of▁sentence｜>"    
    stop    "<｜end▁of▁sentence｜>"      
    stop    "<｜User｜>"                 
    stop    "<｜Assistant｜>"            

  License
    MIT License                    
    Copyright (c) 2023 DeepSeek
<!-- gh-comment-id:2629073907 --> @guiguid commented on GitHub (Feb 1, 2025): Hi, Same issue, mmap from SSD is not used with ollama but woks with llama-cli. ```bash llama-cli --model /srv/ollama/blobs/sha256-f4212639f8b6e105df9c2feebc2f8ebe6c1bb5cac3e721051b097a6bca76c183 --cache-type-k q4_0 --threads 24 --prio 2 --temp 0.6 --ctx-size 8192 --seed 3407 --n-gpu-layers 7 -no-cnv --prompt "<｜User｜>Quelle est la meilleure orientation et inclinaison pour les panneaux solaires installés à Blaye.<｜Assistant｜>" [...] load_tensors: offloading 7 repeating layers to GPU load_tensors: offloaded 7/62 layers to GPU load_tensors: CPU_Mapped model buffer size = 216099,06 MiB llama_init_from_model: n_seq_max = 1 llama_init_from_model: n_ctx = 8192 llama_init_from_model: n_ctx_per_seq = 8192 llama_init_from_model: n_batch = 2048 llama_init_from_model: n_ubatch = 512 llama_init_from_model: flash_attn = 0 llama_init_from_model: freq_base = 10000,0 llama_init_from_model: freq_scale = 0,025 llama_init_from_model: n_ctx_per_seq (8192) < n_ctx_train (163840) -- the full capacity of the model will not be utilized llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'q4_0', type_v = 'f16', n_layer = 61, can_shift = 0 llama_kv_cache_init: CPU KV buffer size = 22204,00 MiB llama_init_from_model: KV self size = 22204,00 MiB, K (q4_0): 6588,00 MiB, V (f16): 15616,00 MiB llama_init_from_model: CPU output buffer size = 0,49 MiB llama_init_from_model: CPU compute buffer size = 2218,01 MiB [...] llama_perf_sampler_print: sampling time = 99,06 ms / 985 runs ( 0,10 ms per token, 9943,47 tokens per second) llama_perf_context_print: load time = 73522,94 ms llama_perf_context_print: prompt eval time = 171156,36 ms / 27 tokens ( 6339,12 ms per token, 0,16 tokens per second) llama_perf_context_print: eval time = 18254717,61 ms / 957 runs (19074,94 ms per token, 0,05 tokens per second) llama_perf_context_print: total time = 18426536,71 ms / 984 tokens [...] ❯ llama-cli --version register_backend: registered backend BLAS (1 devices) register_device: registered device BLAS (OpenBLAS) register_backend: registered backend RPC (0 devices) register_backend: registered backend CPU (1 devices) register_device: registered device CPU (AMD Ryzen 9 7900 12-Core Processor) version: 4598 (27d135c97) built with cc (GCC) 14.2.1 20240910 for x86_64-pc-linux-gnu ``` And with Ollama : ```bash ❯ curl localhost:11434/api/generate -d '{"model":"Huzderu/deepseek-r1-671b-2.51bit:latest","options":{"num_gpu":0,"use_mmap":true}}' {"error":"model requires more system memory (257.6 GiB) than is available (60.2 GiB)"} ❯ ollama -v ollama version is 0.5.7 ❯ ollama show Huzderu/deepseek-r1-671b-2.51bit:latest Model architecture deepseek2 parameters 671.0B context length 163840 embedding length 7168 quantization Q2_K Parameters stop "<｜begin▁of▁sentence｜>" stop "<｜end▁of▁sentence｜>" stop "<｜User｜>" stop "<｜Assistant｜>" License MIT License Copyright (c) 2023 DeepSeek ``` ```
GiteaMirror commented 2026-04-28 22:13:22 -05:00
Author
Owner
Copy Link

@sammcj commented on GitHub (Feb 5, 2025):

Have been trying to run R1 with Ollama for the last week or so, came to a similar conclusion as mentioned here:

  • DeepSeek v3/r1 architecture not supporting Flash Attention and as such K/V context quantisation is where the insane vRAM requirements are coming from.
  • vRAM usage from context sizes 8K -> 54K~ scale consistently, but from about 60K+ memory usage increases significantly faster.
  • Ollama does not let you disable mmap, if it did - it would likely help quite a bit as it does when using llama.cpp directly.
  • The default num_batch of 512 can be reduced to 64 to reduce vRAM overheads at the expense of performance.

For example, I have access to a machine with 8x Nvidia H100 with 80GB vRAM each - 640GB of vRAM - and I cannot run an IQ3_XS quant with a reasonably useful context size of 64k from vRAM it will only be able to load 40 of the 62 layers onto the GPUs.

<!-- gh-comment-id:2635746890 --> @sammcj commented on GitHub (Feb 5, 2025): Have been trying to run R1 with Ollama for the last week or so, came to a similar conclusion as mentioned here: - DeepSeek v3/r1 architecture not supporting Flash Attention and as such K/V context quantisation is where the insane vRAM requirements are coming from. - vRAM usage from context sizes 8K -> 54K~ scale consistently, but from about 60K+ memory usage increases significantly faster. - Ollama does not let you disable mmap, if it did - it would likely help quite a bit as it does when using llama.cpp directly. - The default num_batch of 512 can be reduced to 64 to reduce vRAM overheads at the expense of performance. For example, I have access to a machine with 8x Nvidia H100 with 80GB vRAM each - 640GB of vRAM - and I cannot run an IQ3_XS quant with a reasonably useful context size of 64k from vRAM it will only be able to load 40 of the 62 layers onto the GPUs.
GiteaMirror commented 2026-04-28 22:13:28 -05:00
Author
Owner
Copy Link

@outis151 commented on GitHub (Feb 5, 2025):

I suggest checking again. For testing purposes I have even managed to load the full deepseek R1 (700 gigabytes) on a machine with just CPU and 16GB of RAM.

  • MMAP enabled to avoid loading the entire model at once into RAM but instead read from disk dynamically
  • 60GB of swap to fit in the parts that cannot be cached from disk (this is the mainly the stuff that isn't model weights. More is necessary as you choose higher context sizes)
<!-- gh-comment-id:2636397623 --> @outis151 commented on GitHub (Feb 5, 2025): I suggest checking again. For testing purposes I have even managed to load the full deepseek R1 (700 gigabytes) on a machine with just CPU and 16GB of RAM. - MMAP enabled to avoid loading the entire model at once into RAM but instead read from disk dynamically - 60GB of swap to fit in the parts that cannot be cached from disk (this is the mainly the stuff that isn't model weights. More is necessary as you choose higher context sizes)
GiteaMirror commented 2026-04-28 22:13:32 -05:00
Author
Owner
Copy Link

@HongZhang-Robotechlab commented on GitHub (Feb 7, 2025):

I suggest checking again. For testing purposes I have even managed to load the full deepseek R1 (700 gigabytes) on a machine with just CPU and 16GB of RAM.

  • MMAP enabled to avoid loading the entire model at once into RAM but instead read from disk dynamically
  • 60GB of swap to fit in the parts that cannot be cached from disk (this is the mainly the stuff that isn't model weights. More is necessary as you choose higher context sizes)

I'm trying to load full R1 with Ollama on a Windows 11 Home 32GB RAM 800GB SSD Nvidia RTX4060 GPU. It downloaded the model to local drive successfully but when running, it has this same error as below:

Image

Does Windows have similar operation as MMAP enabled?

<!-- gh-comment-id:2641728701 --> @HongZhang-Robotechlab commented on GitHub (Feb 7, 2025): > I suggest checking again. For testing purposes I have even managed to load the full deepseek R1 (700 gigabytes) on a machine with just CPU and 16GB of RAM. > > * MMAP enabled to avoid loading the entire model at once into RAM but instead read from disk dynamically > * 60GB of swap to fit in the parts that cannot be cached from disk (this is the mainly the stuff that isn't model weights. More is necessary as you choose higher context sizes) I'm trying to load full R1 with Ollama on a Windows 11 Home 32GB RAM 800GB SSD Nvidia RTX4060 GPU. It downloaded the model to local drive successfully but when running, it has this same error as below: <img width="746" alt="Image" src="https://github.com/user-attachments/assets/ac71d84a-71d1-4e3b-a7ca-42ba9662ced3" /> Does Windows have similar operation as MMAP enabled?
GiteaMirror commented 2026-04-28 22:13:35 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 7, 2025):

On windows and linux you currently need to pre-allocate enough swap to hold the model. If https://github.com/ollama/ollama/pull/8895 is approved, all you need to do is set "use_mmap":true.

<!-- gh-comment-id:2642451314 --> @rick-github commented on GitHub (Feb 7, 2025): On windows and linux you currently need to pre-allocate enough swap to hold the model. If https://github.com/ollama/ollama/pull/8895 is approved, all you need to do is set `"use_mmap":true`.
GiteaMirror commented 2026-04-28 22:13:36 -05:00
Author
Owner
Copy Link

@philippstoboy commented on GitHub (Feb 9, 2025):

Hello everyone. I managed to make it run on a machine with 192GB on ram. But only 40.5GB Are being used when running the Model. If anyone is interested I could Share you how it's done.

<!-- gh-comment-id:2646543751 --> @philippstoboy commented on GitHub (Feb 9, 2025): Hello everyone. I managed to make it run on a machine with 192GB on ram. But only 40.5GB Are being used when running the Model. If anyone is interested I could Share you how it's done.
GiteaMirror commented 2026-04-28 22:13:37 -05:00
Author
Owner
Copy Link

@HongZhang-Robotechlab commented on GitHub (Feb 9, 2025):

On windows and linux you currently need to pre-allocate enough swap to hold the model. If #8895 is approved, all you need to do is set "use_mmap":true.

@rick-github Thank you for addressing the issue on Windows with the ollama commit. By the way, how to set "use_mmap":true in Windows command console for ollama?

<!-- gh-comment-id:2646610768 --> @HongZhang-Robotechlab commented on GitHub (Feb 9, 2025): > On windows and linux you currently need to pre-allocate enough swap to hold the model. If [#8895](https://github.com/ollama/ollama/pull/8895) is approved, all you need to do is set `"use_mmap":true`. @rick-github Thank you for addressing the issue on Windows with the ollama commit. By the way, how to set `"use_mmap":true` in Windows command console for ollama?
GiteaMirror commented 2026-04-28 22:13:38 -05:00
Author
Owner
Copy Link

@HongZhang-Robotechlab commented on GitHub (Feb 9, 2025):

Hello everyone. I managed to make it run on a machine with 192GB on ram. But only 40.5GB Are being used when running the Model. If anyone is interested I could Share you how it's done.

Nice! I'm very curious how to make it work.

<!-- gh-comment-id:2646611602 --> @HongZhang-Robotechlab commented on GitHub (Feb 9, 2025): > Hello everyone. I managed to make it run on a machine with 192GB on ram. But only 40.5GB Are being used when running the Model. If anyone is interested I could Share you how it's done. Nice! I'm very curious how to make it work.
GiteaMirror commented 2026-04-28 22:13:39 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 9, 2025):

use_mmap is true by default, ollama only sets it false when it thinks there are not enough resources. You can override with the /set parameter command.

$ ollama run qwen2.5:0.5b
>>> /set parameter use_mmap true
Set parameter 'use_mmap' to 'true'
>>> hello
Hello! How can I assist you today? Feel free to ask me any questions or share your thoughts with me.
<!-- gh-comment-id:2646612470 --> @rick-github commented on GitHub (Feb 9, 2025): `use_mmap` is true by default, ollama only sets it false when it thinks there are not enough resources. You can override with the `/set parameter` command. ```console $ ollama run qwen2.5:0.5b >>> /set parameter use_mmap true Set parameter 'use_mmap' to 'true' >>> hello Hello! How can I assist you today? Feel free to ask me any questions or share your thoughts with me. ```
GiteaMirror commented 2026-04-28 22:13:39 -05:00
Author
Owner
Copy Link

@HongZhang-Robotechlab commented on GitHub (Feb 9, 2025):

$ ollama run qwen2.5:0.5b

/set parameter use_mmap true
Set parameter 'use_mmap' to 'true'
hello
Hello! How can I assist you today? Feel free to ask me any questions or share your thoughts with me.

Thank you! In my situation, it could the case that ollama thinks there are not enough RAM for 671b, so I can't run "ollama run deepseek-r1:671b". Do I have a way to set it when or before running the command? Or if I got ollama with your code change in https://github.com/ollama/ollama/pull/8895, I will not need to worry about it?

<!-- gh-comment-id:2646621603 --> @HongZhang-Robotechlab commented on GitHub (Feb 9, 2025): > $ ollama run qwen2.5:0.5b > >>> /set parameter use_mmap true > Set parameter 'use_mmap' to 'true' > >>> hello > Hello! How can I assist you today? Feel free to ask me any questions or share your thoughts with me. Thank you! In my situation, it could the case that ollama thinks there are not enough RAM for 671b, so I can't run "_ollama run deepseek-r1:671b_". Do I have a way to set it when or before running the command? Or if I got ollama with your code change in https://github.com/ollama/ollama/pull/8895, I will not need to worry about it?
GiteaMirror commented 2026-04-28 22:13:40 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 9, 2025):

echo FROM deepseek-r1:671b > Modelfile
echo PARAMETER use_mmap true >> Modelfile
ollama create deepseek-r1:671b-mmap

#8895 requires that use_mmap is explicitly set, so something like the above or setting "options":{"use_mmap":true} in the API call would be required.

<!-- gh-comment-id:2646626235 --> @rick-github commented on GitHub (Feb 9, 2025): ```console echo FROM deepseek-r1:671b > Modelfile echo PARAMETER use_mmap true >> Modelfile ollama create deepseek-r1:671b-mmap ``` #8895 requires that `use_mmap` is explicitly set, so something like the above or setting `"options":{"use_mmap":true}` in the API call would be required.
GiteaMirror commented 2026-04-28 22:13:41 -05:00
Author
Owner
Copy Link

@glide-the commented on GitHub (Feb 10, 2025):

I tried quantized inference in llama.cpp, and the settings can be started normally.

https://unsloth.ai/blog/deepseekr1-dynamic#running%20r1
--threads 16 \
--n-gpu-layers 7 \

But it was observed that the modelfile configuration was deleted in ollama.

e54a3c7fcd

 
FROM ./merged_file.gguf



TEMPLATE """{{- if .System }}{{ .System }}{{ end }}
{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1}}
{{- if eq .Role "user" }}<｜User｜>{{ .Content }}
{{- else if eq .Role "assistant" }}<｜Assistant｜>{{ .Content }}{{- if not $last }}<｜end▁of▁sentence｜>{{- end }}
{{- end }}
{{- if and $last (ne .Role "assistant") }}<｜Assistant｜>{{- end }}
{{- end }}"""
PARAMETER stop <｜begin▁of▁sentence｜>
PARAMETER stop <｜end▁of▁sentence｜>
PARAMETER stop <｜User｜>
PARAMETER stop <｜Assistant｜>
PARAMETER use_mmap true
PARAMETER num_gpu 7
PARAMETER num_ctx 8192

LICENSE """MIT License

Copyright (c) 2023 DeepSeek

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""

ollama run deepseek-r1:merge1.58


Error: timed out waiting for llama runner to start - progress 0.00 -

After I tried to configure it in /api/generate, it appeared, success

curl http://localhost:11434/api/generate -d '{"model": "deepseek-r1:merge1.58", "use_mmap": true, "num_gpu": 7, "keep_alive": "24h"}'
{"model":"deepseek-r1:merge1.58","created_at":"2025-02-10T13:01:02.770051259Z","response":"","done":true,"done_reason":"load"}
<!-- gh-comment-id:2647923888 --> @glide-the commented on GitHub (Feb 10, 2025): I tried quantized inference in llama.cpp, and the settings can be started normally. ``` https://unsloth.ai/blog/deepseekr1-dynamic#running%20r1 --threads 16 \ --n-gpu-layers 7 \ ``` But it was observed that the modelfile configuration was deleted in ollama. https://github.com/ollama/ollama/commit/e54a3c7fcd3a66486c3946a7944f3d7ce2daff6c ``` FROM ./merged_file.gguf TEMPLATE """{{- if .System }}{{ .System }}{{ end }} {{- range $i, $_ := .Messages }} {{- $last := eq (len (slice $.Messages $i)) 1}} {{- if eq .Role "user" }}<｜User｜>{{ .Content }} {{- else if eq .Role "assistant" }}<｜Assistant｜>{{ .Content }}{{- if not $last }}<｜end▁of▁sentence｜>{{- end }} {{- end }} {{- if and $last (ne .Role "assistant") }}<｜Assistant｜>{{- end }} {{- end }}""" PARAMETER stop <｜begin▁of▁sentence｜> PARAMETER stop <｜end▁of▁sentence｜> PARAMETER stop <｜User｜> PARAMETER stop <｜Assistant｜> PARAMETER use_mmap true PARAMETER num_gpu 7 PARAMETER num_ctx 8192 LICENSE """MIT License Copyright (c) 2023 DeepSeek Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. """ ``` ollama run deepseek-r1:merge1.58 ``` Error: timed out waiting for llama runner to start - progress 0.00 - ``` After I tried to configure it in /api/generate, it appeared, success ``` curl http://localhost:11434/api/generate -d '{"model": "deepseek-r1:merge1.58", "use_mmap": true, "num_gpu": 7, "keep_alive": "24h"}' {"model":"deepseek-r1:merge1.58","created_at":"2025-02-10T13:01:02.770051259Z","response":"","done":true,"done_reason":"load"} ```
GiteaMirror commented 2026-04-28 22:13:41 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 10, 2025):

Set OLLAMA_LOAD_TIMEOUT.

<!-- gh-comment-id:2647948887 --> @rick-github commented on GitHub (Feb 10, 2025): Set [`OLLAMA_LOAD_TIMEOUT`]( https://github.com/ollama/ollama/blob/38117fba83c8291e28abe564d56600c200ff4cab/envconfig/config.go#L243).
GiteaMirror commented 2026-04-28 22:13:41 -05:00
Author
Owner
Copy Link

@HongZhang-Robotechlab commented on GitHub (Feb 15, 2025):

echo FROM deepseek-r1:671b > Modelfile
echo PARAMETER use_mmap true >> Modelfile
ollama create deepseek-r1:671b-mmap
#8895 requires that use_mmap is explicitly set, so something like the above or setting "options":{"use_mmap":true} in the API call would be required.
Thank you! I tried to run the above 3 commands as below:
Image
Image
Image

It seems that the configuration was still not grasped by ollama.

I tried to download and build ollama by following the Development guide at link https://github.com/ollama/ollama/blob/main/docs/development.md, but don't know what to do after this:
Image

What are the following commands to make it work with deepseek-r1?

<!-- gh-comment-id:2661010902 --> @HongZhang-Robotechlab commented on GitHub (Feb 15, 2025): > echo FROM deepseek-r1:671b > Modelfile > echo PARAMETER use_mmap true >> Modelfile > ollama create deepseek-r1:671b-mmap > [#8895](https://github.com/ollama/ollama/pull/8895) requires that `use_mmap` is explicitly set, so something like the above or setting `"options":{"use_mmap":true}` in the API call would be required. Thank you! I tried to run the above 3 commands as below: <img width="660" alt="Image" src="https://github.com/user-attachments/assets/393aa73d-20b4-43ef-a1f9-40390cec7267" /> <img width="446" alt="Image" src="https://github.com/user-attachments/assets/24654b78-d18e-4c05-83cd-7e23dc17d7a0" /> <img width="583" alt="Image" src="https://github.com/user-attachments/assets/9d1529d9-0771-49eb-b22b-abd0acaadb0b" /> It seems that the configuration was still not grasped by ollama. I tried to download and build ollama by following the Development guide at link https://github.com/ollama/ollama/blob/main/docs/development.md, but don't know what to do after this: <img width="763" alt="Image" src="https://github.com/user-attachments/assets/57b62a6b-46c3-4b16-80bc-b95300e4bbb2" /> What are the following commands to make it work with deepseek-r1?
GiteaMirror commented 2026-04-28 22:13:42 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 15, 2025):

#8895 is not merged yet, you need to manually patch it in to the code before you build:

curl -sL https://github.com/ollama/ollama/pull/8895.patch | git apply
<!-- gh-comment-id:2661021324 --> @rick-github commented on GitHub (Feb 15, 2025): #8895 is not merged yet, you need to manually patch it in to the code before you build: ``` curl -sL https://github.com/ollama/ollama/pull/8895.patch | git apply ```
GiteaMirror commented 2026-04-28 22:13:42 -05:00
Author
Owner
Copy Link

@HongZhang-Robotechlab commented on GitHub (Feb 15, 2025):

#8895 is not merged yet, you need to manually patch it in to the code before you build:

curl -sL https://github.com/ollama/ollama/pull/8895.patch | git apply

Done and rebuild ollama:
Image

Then run "go run . serve" as the development.md instructed:
Image

What shall I do next?

<!-- gh-comment-id:2661033276 --> @HongZhang-Robotechlab commented on GitHub (Feb 15, 2025): > [#8895](https://github.com/ollama/ollama/pull/8895) is not merged yet, you need to manually patch it in to the code before you build: > > ``` > curl -sL https://github.com/ollama/ollama/pull/8895.patch | git apply > ``` Done and rebuild ollama: <img width="825" alt="Image" src="https://github.com/user-attachments/assets/e8fa68bd-a0cb-4849-81fe-012c60ea87e3" /> Then run "go run . serve" as the development.md instructed: <img width="1259" alt="Image" src="https://github.com/user-attachments/assets/0bff4a2c-86bf-4a5e-b449-57024c49db28" /> What shall I do next?
GiteaMirror commented 2026-04-28 22:13:43 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 15, 2025):

In a different terminal window:

ollama run deepseek-r1:671b-mmap
<!-- gh-comment-id:2661034482 --> @rick-github commented on GitHub (Feb 15, 2025): In a different terminal window: ``` ollama run deepseek-r1:671b-mmap ```
GiteaMirror commented 2026-04-28 22:13:43 -05:00
Author
Owner
Copy Link

@HongZhang-Robotechlab commented on GitHub (Feb 16, 2025):

In a different terminal window:

ollama run deepseek-r1:671b-mmap

This problem is solved!. Thank you for your step-by-step guiding.
However, it still failed with GPU error as below:
Image
With the details from the first terminal window, I can switch to the gear to https://github.com/ollama/ollama/issues/8447

<!-- gh-comment-id:2661456242 --> @HongZhang-Robotechlab commented on GitHub (Feb 16, 2025): > In a different terminal window: > > ``` > ollama run deepseek-r1:671b-mmap > ``` This problem is solved!. Thank you for your step-by-step guiding. However, it still failed with GPU error as below: <img width="555" alt="Image" src="https://github.com/user-attachments/assets/6b3a86c2-30d3-4f59-9212-af5984d7a457" /> With the details from the first terminal window, I can switch to the gear to https://github.com/ollama/ollama/issues/8447
GiteaMirror commented 2026-04-28 22:13:44 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 16, 2025):

It's trying to allocate memory on the GPU. You only have 6.9G free on the GPU and ollama is trying to offload 3 layers (see layers.offload in the log). Reduce the number of layers being offloaded.

echo PARAMETER num_gpu 2 >> Modelfile
ollama create deepseek-r1:671b-mmap
<!-- gh-comment-id:2661460367 --> @rick-github commented on GitHub (Feb 16, 2025): It's trying to allocate memory on the GPU. You only have 6.9G free on the GPU and ollama is trying to offload 3 layers (see `layers.offload` in the log). Reduce the number of layers being offloaded. ``` echo PARAMETER num_gpu 2 >> Modelfile ollama create deepseek-r1:671b-mmap ```
GiteaMirror commented 2026-04-28 22:13:44 -05:00
Author
Owner
Copy Link

@HongZhang-Robotechlab commented on GitHub (Feb 16, 2025):

It's trying to allocate memory on the GPU. You only have 6.9G free on the GPU and ollama is trying to offload 3 layers (see layers.offload in the log). Reduce the number of layers being offloaded.

echo PARAMETER num_gpu 2 >> Modelfile
ollama create deepseek-r1:671b-mmap

It works now with the log as below:
llm_load_tensors: offloading 2 repeating layers to GPU
llm_load_tensors: offloaded 2/62 layers to GPU
llm_load_tensors: CUDA0 model buffer size = 14213.34 MiB
llm_load_tensors: CPU_Mapped model buffer size = 371476.29 MiB
llama_new_context_with_model: n_seq_max = 4
llama_new_context_with_model: n_ctx = 8192
llama_new_context_with_model: n_ctx_per_seq = 2048
llama_new_context_with_model: n_batch = 2048
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 0.025
llama_new_context_with_model: n_ctx_per_seq (2048) < n_ctx_train (163840) -- the full capacity of the model will not be utilized
llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 61, can_shift = 0
llama_kv_cache_init: CUDA0 KV buffer size = 1280.00 MiB
llama_kv_cache_init: CPU KV buffer size = 37760.00 MiB
llama_new_context_with_model: KV self size = 39040.00 MiB, K (f16): 23424.00 MiB, V (f16): 15616.00 MiB
llama_new_context_with_model: CPU output buffer size = 2.08 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 5039.50 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 96.01 MiB
llama_new_context_with_model: graph nodes = 5025
llama_new_context_with_model: graph splits = 1110 (with bs=512), 3 (with bs=1)
time=2025-02-16T09:50:46.717-05:00 level=INFO source=server.go:597 msg="llama runner started in 92.49 seconds"
[GIN] 2025/02/16 - 09:50:46 | 200 | 1m32s | 127.0.0.1 | POST "/api/generate"

But looks a little bit slow: Since 9:51 I asked the question until now 12:45. what I got is as below:
Image

The fan of CPU sounds pretty calm. Is it supposed to work this slow to running a big model on basic hardware like my computer?

<!-- gh-comment-id:2661544452 --> @HongZhang-Robotechlab commented on GitHub (Feb 16, 2025): > It's trying to allocate memory on the GPU. You only have 6.9G free on the GPU and ollama is trying to offload 3 layers (see `layers.offload` in the log). Reduce the number of layers being offloaded. > > ``` > echo PARAMETER num_gpu 2 >> Modelfile > ollama create deepseek-r1:671b-mmap > ``` It works now with the log as below: _llm_load_tensors: offloading 2 repeating layers to GPU llm_load_tensors: offloaded 2/62 layers to GPU llm_load_tensors: CUDA0 model buffer size = 14213.34 MiB llm_load_tensors: CPU_Mapped model buffer size = 371476.29 MiB llama_new_context_with_model: n_seq_max = 4 llama_new_context_with_model: n_ctx = 8192 llama_new_context_with_model: n_ctx_per_seq = 2048 llama_new_context_with_model: n_batch = 2048 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 0.025 llama_new_context_with_model: n_ctx_per_seq (2048) < n_ctx_train (163840) -- the full capacity of the model will not be utilized llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 61, can_shift = 0 llama_kv_cache_init: CUDA0 KV buffer size = 1280.00 MiB llama_kv_cache_init: CPU KV buffer size = 37760.00 MiB llama_new_context_with_model: KV self size = 39040.00 MiB, K (f16): 23424.00 MiB, V (f16): 15616.00 MiB llama_new_context_with_model: CPU output buffer size = 2.08 MiB llama_new_context_with_model: CUDA0 compute buffer size = 5039.50 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 96.01 MiB llama_new_context_with_model: graph nodes = 5025 llama_new_context_with_model: graph splits = 1110 (with bs=512), 3 (with bs=1) time=2025-02-16T09:50:46.717-05:00 level=INFO source=server.go:597 msg="llama runner started in 92.49 seconds" [GIN] 2025/02/16 - 09:50:46 | 200 | 1m32s | 127.0.0.1 | POST "/api/generate"_ But looks a little bit slow: Since 9:51 I asked the question until now 12:45. what I got is as below: <img width="808" alt="Image" src="https://github.com/user-attachments/assets/0cec030e-a55b-442d-994c-c937ab8c73c3" /> The fan of CPU sounds pretty calm. Is it supposed to work this slow to running a big model on basic hardware like my computer?
GiteaMirror commented 2026-04-28 22:13:45 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 16, 2025):

Only 2 layers of 62 have been offloaded to the GPU. The rest of the model weights are mmap'ed so the bandwidth reading those weights will determine the rate at which tokens can be generated. 317GB of the model weights are mmap'ed. An average SSD can read date at about 550MB/s. So one token takes (317 / 0.55) = 576 seconds to generate. If you have the model on NVMe using PCIe 4.0, you get about 7000MB/s read rate, so one token takes (317 / 7) = 45 seconds. The page cache of the filesystem will help somewhat by having some of the model in RAM. You don't indicate what sort of system you have but you say "basic hardware", so let's assume something with DDR4 RAM. That's about 20GB/s read rate, so if the model was solely in RAM, one token would take (317 / 20) = 16 seconds. So best case speed you will get is 0.0625 tokens/second, worst case is 0.00173611 tokens/second.

Running a model requires VRAM. Running a large model requires a lot of VRAM.

<!-- gh-comment-id:2661552858 --> @rick-github commented on GitHub (Feb 16, 2025): Only 2 layers of 62 have been offloaded to the GPU. The rest of the model weights are mmap'ed so the bandwidth reading those weights will determine the rate at which tokens can be generated. 317GB of the model weights are mmap'ed. An average SSD can read date at about 550MB/s. So one token takes (317 / 0.55) = 576 seconds to generate. If you have the model on NVMe using PCIe 4.0, you get about 7000MB/s read rate, so one token takes (317 / 7) = 45 seconds. The page cache of the filesystem will help somewhat by having some of the model in RAM. You don't indicate what sort of system you have but you say "basic hardware", so let's assume something with DDR4 RAM. That's about 20GB/s read rate, so if the model was solely in RAM, one token would take (317 / 20) = 16 seconds. So best case speed you will get is 0.0625 tokens/second, worst case is 0.00173611 tokens/second. Running a model requires VRAM. Running a large model requires a lot of VRAM.
GiteaMirror commented 2026-04-28 22:13:45 -05:00
Author
Owner
Copy Link

@HongZhang-Robotechlab commented on GitHub (Mar 2, 2025):

It's trying to allocate memory on the GPU. You only have 6.9G free on the GPU and ollama is trying to offload 3 layers (see layers.offload in the log). Reduce the number of layers being offloaded.

echo PARAMETER num_gpu 2 >> Modelfile
ollama create deepseek-r1:671b-mmap

Since ollama can detect the available GPU memory and model layer size, could it adjust the value of num_gpu automatically instead manually set it in Modelfile?

<!-- gh-comment-id:2692941755 --> @HongZhang-Robotechlab commented on GitHub (Mar 2, 2025): > It's trying to allocate memory on the GPU. You only have 6.9G free on the GPU and ollama is trying to offload 3 layers (see `layers.offload` in the log). Reduce the number of layers being offloaded. > > ``` > echo PARAMETER num_gpu 2 >> Modelfile > ollama create deepseek-r1:671b-mmap > ``` Since ollama can detect the available GPU memory and model layer size, could it adjust the value of num_gpu automatically instead manually set it in Modelfile?
GiteaMirror commented 2026-04-28 22:13:45 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 3, 2025):

It tries to calculate num_gpu automatically but it gets it wrong, that's why it's manually overridden in he Modelfile. #9243 to improve the accuracy of the num_gpu estimation.

<!-- gh-comment-id:2693444913 --> @rick-github commented on GitHub (Mar 3, 2025): It tries to calculate `num_gpu` automatically but it gets it wrong, that's why it's manually overridden in he Modelfile. #9243 to improve the accuracy of the `num_gpu` estimation.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#52127
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?