github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 21:25:46 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #9181] How to offload all layers to GPU? #52491

New Issue
Closed
opened 2026-04-28 23:26:40 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-04-28 23:26:40 -05:00
Owner
Copy Link

Originally created by @cyberluke on GitHub (Feb 18, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9181

What is the issue?

Ollama offloads only half layers to gpu, half to cpu on 4x L4 (4x 24GB) !

Compiled current Github version on Lightning AI Studio

2025/02/18 02:20:54 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES:0,1,2,3,4,5,6,7 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:2562047h47m16.854775807s OLLAMA_MAX_LOADED_MODELS:5 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/teamspace/studios/this_studio/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:true OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] POST   /api/pull                 --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers)
[GIN-debug] POST   /api/generate             --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers)
[GIN-debug] POST   /api/chat                 --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers)
[GIN-debug] POST   /api/embed                --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers)
[GIN-debug] POST   /api/embeddings           --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers)
[GIN-debug] POST   /api/create               --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers)
[GIN-debug] POST   /api/push                 --> github.com/ollama/ollama/server.(*Server).PushHandler-fm (5 handlers)
[GIN-debug] POST   /api/copy                 --> github.com/ollama/ollama/server.(*Server).CopyHandler-fm (5 handlers)
[GIN-debug] DELETE /api/delete               --> github.com/ollama/ollama/server.(*Server).DeleteHandler-fm (5 handlers)
[GIN-debug] POST   /api/show                 --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (5 handlers)
[GIN-debug] POST   /api/blobs/:digest        --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers)
[GIN-debug] HEAD   /api/blobs/:digest        --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers)
[GIN-debug] GET    /api/ps                   --> github.com/ollama/ollama/server.(*Server).PsHandler-fm (5 handlers)
[GIN-debug] POST   /v1/chat/completions      --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers)
[GIN-debug] POST   /v1/completions           --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (6 handlers)
[GIN-debug] POST   /v1/embeddings            --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (6 handlers)
[GIN-debug] GET    /v1/models                --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (6 handlers)
[GIN-debug] GET    /v1/models/:model         --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (6 handlers)
[GIN-debug] GET    /                         --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] GET    /api/tags                 --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
[GIN-debug] GET    /api/version              --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
[GIN-debug] HEAD   /                         --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] HEAD   /api/tags                 --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
[GIN-debug] HEAD   /api/version              --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
time=2025-02-18T02:20:54.306Z level=INFO source=routes.go:1238 msg="Listening on [::]:11434 (version 0.0.0)"
time=2025-02-18T02:20:54.306Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-02-18T02:20:54.864Z level=INFO source=types.go:130 msg="inference compute" id=GPU-0ff4bdf0-9e73-ce87-e5ba-38960bca4915 library=cuda variant=v12 compute=8.9 driver=12.2 name="NVIDIA L4" total="22.2 GiB" available="22.0 GiB"
time=2025-02-18T02:20:54.864Z level=INFO source=types.go:130 msg="inference compute" id=GPU-6190f995-e05c-2456-a242-cc78a5a8460c library=cuda variant=v12 compute=8.9 driver=12.2 name="NVIDIA L4" total="22.2 GiB" available="22.0 GiB"
time=2025-02-18T02:20:54.864Z level=INFO source=types.go:130 msg="inference compute" id=GPU-023327d0-2990-1f83-8ecc-9baee53a7467 library=cuda variant=v12 compute=8.9 driver=12.2 name="NVIDIA L4" total="22.2 GiB" available="22.0 GiB"
time=2025-02-18T02:20:54.864Z level=INFO source=types.go:130 msg="inference compute" id=GPU-797fa0e5-a022-e02b-b15b-e0bc44f2fc2d library=cuda variant=v12 compute=8.9 driver=12.2 name="NVIDIA L4" total="22.2 GiB" available="22.0 GiB"
time=2025-02-18T02:21:25.260Z level=INFO source=server.go:97 msg="system memory" total="181.8 GiB" free="159.4 GiB" free_swap="32.0 GiB"
time=2025-02-18T02:21:25.646Z level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=81 layers.offload=44 layers.split=11,11,11,11 memory.available="[22.0 GiB 22.0 GiB 22.0 GiB 22.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="127.4 GiB" memory.required.partial="87.1 GiB" memory.required.kv="19.5 GiB" memory.required.allocations="[21.8 GiB 21.8 GiB 21.8 GiB 21.8 GiB]" memory.weights.total="87.3 GiB" memory.weights.repeating="86.2 GiB" memory.weights.nonrepeating="1.0 GiB" memory.graph.full="8.2 GiB" memory.graph.partial="8.2 GiB"
time=2025-02-18T02:21:25.646Z level=INFO source=server.go:182 msg="enabling flash attention"
time=2025-02-18T02:21:25.646Z level=WARN source=server.go:190 msg="kv cache type not supported by model" type=""
time=2025-02-18T02:21:25.647Z level=INFO source=server.go:380 msg="starting llama server" cmd="/teamspace/studios/this_studio/ollama/ollama runner --model /teamspace/studios/this_studio/.ollama/models/blobs/sha256-feef62aa06ab4162ebd3b9af4ff8383a37bf9544a7d30a3fe4623c8398bd1a28 --ctx-size 64000 --batch-size 512 --n-gpu-layers 44 --threads 24 --flash-attn --parallel 1 --tensor-split 11,11,11,11 --port 50137"
time=2025-02-18T02:21:25.648Z level=INFO source=sched.go:450 msg="loaded runners" count=1
time=2025-02-18T02:21:25.648Z level=INFO source=server.go:557 msg="waiting for llama runner to start responding"
time=2025-02-18T02:21:25.648Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error"
time=2025-02-18T02:21:25.662Z level=INFO source=runner.go:932 msg="starting go runner"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 CUDA devices:
  Device 0: NVIDIA L4, compute capability 8.9, VMM: yes
  Device 1: NVIDIA L4, compute capability 8.9, VMM: yes
  Device 2: NVIDIA L4, compute capability 8.9, VMM: yes
  Device 3: NVIDIA L4, compute capability 8.9, VMM: yes
load_backend: loaded CUDA backend from /teamspace/studios/this_studio/ollama/build/lib/ollama/libggml-cuda.so
load_backend: loaded CPU backend from /teamspace/studios/this_studio/ollama/build/lib/ollama/libggml-cpu-haswell.so
time=2025-02-18T02:21:25.946Z level=INFO source=runner.go:935 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | CUDA : ARCHS = 890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : LLAMAFILE = 1 | cgo(gcc)" threads=24
time=2025-02-18T02:21:25.947Z level=INFO source=runner.go:993 msg="Server listening on 127.0.0.1:50137"
llama_load_model_from_file: using device CUDA0 (NVIDIA L4) - 22510 MiB free
llama_load_model_from_file: using device CUDA1 (NVIDIA L4) - 22510 MiB free
llama_load_model_from_file: using device CUDA2 (NVIDIA L4) - 22510 MiB free
llama_load_model_from_file: using device CUDA3 (NVIDIA L4) - 22510 MiB free

export OLLAMA_MODELS="$HOME/.ollama/models"
export CMAKE_BIN="$HOME/cmake-3.31.5-linux-x86_64/bin"
export CMAKE_ROOT="$HOME/cmake-3.31.5-linux-x86_64/share/cmake-3.31"

export WEBUI_NAME="NANOTRIK.AI ALPHA"
export OLLAMA_ACCELERATE=1
export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_NOPRUNE=1
export OLLAMA_MAX_LOADED_MODELS=5
export OLLAMA_LOAD_TIMEOUT=0
export OLLAMA_NOHISTORY=0
export OLLAMA_KEEP_ALIVE=-1
#export OLLAMA_DISABLE_CPU=1
export OLLAMA_DEBUG=0
export OLLAMA_ORIGINS="*"
export OLLAMA_HOST="http://0.0.0.0:11434/"

Full GPU allocation
export CUDA_VISIBLE_DEVICES=0,1,2,3 # All 8 GPUs
export OLLAMA_NUM_GPU_LAYERS=9999 # Force full offload
export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_BATCH_SIZE=8192
export OLLAMA_GPUMEMORY=24000MB # 22GB per GPU (adjust for your VRAM)
export OLLAMA_GPUSPLIT="24,24,24,24,24,24,24,24"

~ nvidia-smi
Tue Feb 18 02:22:57 2025
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.230.02 Driver Version: 535.230.02 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA L4 Off | 00000000:38:00.0 Off | 0 |
| N/A 53C P0 50W / 72W | 13853MiB / 23034MiB | 96% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA L4 Off | 00000000:3A:00.0 Off | 0 |
| N/A 43C P0 27W / 72W | 12727MiB / 23034MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA L4 Off | 00000000:3C:00.0 Off | 0 |
| N/A 41C P0 26W / 72W | 12727MiB / 23034MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 3 NVIDIA L4 Off | 00000000:3E:00.0 Off | 0 |
| N/A 42C P0 27W / 72W | 12727MiB / 23034MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

Relevant log output

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

github latest

Originally created by @cyberluke on GitHub (Feb 18, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9181 ### What is the issue? Ollama offloads only half layers to gpu, half to cpu on 4x L4 (4x 24GB) ! Compiled current Github version on Lightning AI Studio ``` 2025/02/18 02:20:54 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES:0,1,2,3,4,5,6,7 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:2562047h47m16.854775807s OLLAMA_MAX_LOADED_MODELS:5 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/teamspace/studios/this_studio/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:true OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached. [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production. - using env: export GIN_MODE=release - using code: gin.SetMode(gin.ReleaseMode) [GIN-debug] POST /api/pull --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers) [GIN-debug] POST /api/generate --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers) [GIN-debug] POST /api/chat --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers) [GIN-debug] POST /api/embed --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers) [GIN-debug] POST /api/embeddings --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers) [GIN-debug] POST /api/create --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers) [GIN-debug] POST /api/push --> github.com/ollama/ollama/server.(*Server).PushHandler-fm (5 handlers) [GIN-debug] POST /api/copy --> github.com/ollama/ollama/server.(*Server).CopyHandler-fm (5 handlers) [GIN-debug] DELETE /api/delete --> github.com/ollama/ollama/server.(*Server).DeleteHandler-fm (5 handlers) [GIN-debug] POST /api/show --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (5 handlers) [GIN-debug] POST /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers) [GIN-debug] HEAD /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers) [GIN-debug] GET /api/ps --> github.com/ollama/ollama/server.(*Server).PsHandler-fm (5 handlers) [GIN-debug] POST /v1/chat/completions --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers) [GIN-debug] POST /v1/completions --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (6 handlers) [GIN-debug] POST /v1/embeddings --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (6 handlers) [GIN-debug] GET /v1/models --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (6 handlers) [GIN-debug] GET /v1/models/:model --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (6 handlers) [GIN-debug] GET / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers) [GIN-debug] GET /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers) [GIN-debug] GET /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers) [GIN-debug] HEAD / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers) [GIN-debug] HEAD /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers) [GIN-debug] HEAD /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers) time=2025-02-18T02:20:54.306Z level=INFO source=routes.go:1238 msg="Listening on [::]:11434 (version 0.0.0)" time=2025-02-18T02:20:54.306Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-02-18T02:20:54.864Z level=INFO source=types.go:130 msg="inference compute" id=GPU-0ff4bdf0-9e73-ce87-e5ba-38960bca4915 library=cuda variant=v12 compute=8.9 driver=12.2 name="NVIDIA L4" total="22.2 GiB" available="22.0 GiB" time=2025-02-18T02:20:54.864Z level=INFO source=types.go:130 msg="inference compute" id=GPU-6190f995-e05c-2456-a242-cc78a5a8460c library=cuda variant=v12 compute=8.9 driver=12.2 name="NVIDIA L4" total="22.2 GiB" available="22.0 GiB" time=2025-02-18T02:20:54.864Z level=INFO source=types.go:130 msg="inference compute" id=GPU-023327d0-2990-1f83-8ecc-9baee53a7467 library=cuda variant=v12 compute=8.9 driver=12.2 name="NVIDIA L4" total="22.2 GiB" available="22.0 GiB" time=2025-02-18T02:20:54.864Z level=INFO source=types.go:130 msg="inference compute" id=GPU-797fa0e5-a022-e02b-b15b-e0bc44f2fc2d library=cuda variant=v12 compute=8.9 driver=12.2 name="NVIDIA L4" total="22.2 GiB" available="22.0 GiB" time=2025-02-18T02:21:25.260Z level=INFO source=server.go:97 msg="system memory" total="181.8 GiB" free="159.4 GiB" free_swap="32.0 GiB" time=2025-02-18T02:21:25.646Z level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=81 layers.offload=44 layers.split=11,11,11,11 memory.available="[22.0 GiB 22.0 GiB 22.0 GiB 22.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="127.4 GiB" memory.required.partial="87.1 GiB" memory.required.kv="19.5 GiB" memory.required.allocations="[21.8 GiB 21.8 GiB 21.8 GiB 21.8 GiB]" memory.weights.total="87.3 GiB" memory.weights.repeating="86.2 GiB" memory.weights.nonrepeating="1.0 GiB" memory.graph.full="8.2 GiB" memory.graph.partial="8.2 GiB" time=2025-02-18T02:21:25.646Z level=INFO source=server.go:182 msg="enabling flash attention" time=2025-02-18T02:21:25.646Z level=WARN source=server.go:190 msg="kv cache type not supported by model" type="" time=2025-02-18T02:21:25.647Z level=INFO source=server.go:380 msg="starting llama server" cmd="/teamspace/studios/this_studio/ollama/ollama runner --model /teamspace/studios/this_studio/.ollama/models/blobs/sha256-feef62aa06ab4162ebd3b9af4ff8383a37bf9544a7d30a3fe4623c8398bd1a28 --ctx-size 64000 --batch-size 512 --n-gpu-layers 44 --threads 24 --flash-attn --parallel 1 --tensor-split 11,11,11,11 --port 50137" time=2025-02-18T02:21:25.648Z level=INFO source=sched.go:450 msg="loaded runners" count=1 time=2025-02-18T02:21:25.648Z level=INFO source=server.go:557 msg="waiting for llama runner to start responding" time=2025-02-18T02:21:25.648Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error" time=2025-02-18T02:21:25.662Z level=INFO source=runner.go:932 msg="starting go runner" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 4 CUDA devices: Device 0: NVIDIA L4, compute capability 8.9, VMM: yes Device 1: NVIDIA L4, compute capability 8.9, VMM: yes Device 2: NVIDIA L4, compute capability 8.9, VMM: yes Device 3: NVIDIA L4, compute capability 8.9, VMM: yes load_backend: loaded CUDA backend from /teamspace/studios/this_studio/ollama/build/lib/ollama/libggml-cuda.so load_backend: loaded CPU backend from /teamspace/studios/this_studio/ollama/build/lib/ollama/libggml-cpu-haswell.so time=2025-02-18T02:21:25.946Z level=INFO source=runner.go:935 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | CUDA : ARCHS = 890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : LLAMAFILE = 1 | cgo(gcc)" threads=24 time=2025-02-18T02:21:25.947Z level=INFO source=runner.go:993 msg="Server listening on 127.0.0.1:50137" llama_load_model_from_file: using device CUDA0 (NVIDIA L4) - 22510 MiB free llama_load_model_from_file: using device CUDA1 (NVIDIA L4) - 22510 MiB free llama_load_model_from_file: using device CUDA2 (NVIDIA L4) - 22510 MiB free llama_load_model_from_file: using device CUDA3 (NVIDIA L4) - 22510 MiB free export OLLAMA_MODELS="$HOME/.ollama/models" export CMAKE_BIN="$HOME/cmake-3.31.5-linux-x86_64/bin" export CMAKE_ROOT="$HOME/cmake-3.31.5-linux-x86_64/share/cmake-3.31" export WEBUI_NAME="NANOTRIK.AI ALPHA" export OLLAMA_ACCELERATE=1 export OLLAMA_FLASH_ATTENTION=1 export OLLAMA_NOPRUNE=1 export OLLAMA_MAX_LOADED_MODELS=5 export OLLAMA_LOAD_TIMEOUT=0 export OLLAMA_NOHISTORY=0 export OLLAMA_KEEP_ALIVE=-1 #export OLLAMA_DISABLE_CPU=1 export OLLAMA_DEBUG=0 export OLLAMA_ORIGINS="*" export OLLAMA_HOST="http://0.0.0.0:11434/" Full GPU allocation export CUDA_VISIBLE_DEVICES=0,1,2,3 # All 8 GPUs export OLLAMA_NUM_GPU_LAYERS=9999 # Force full offload export OLLAMA_FLASH_ATTENTION=1 export OLLAMA_BATCH_SIZE=8192 export OLLAMA_GPUMEMORY=24000MB # 22GB per GPU (adjust for your VRAM) export OLLAMA_GPUSPLIT="24,24,24,24,24,24,24,24" ~ nvidia-smi Tue Feb 18 02:22:57 2025 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.230.02 Driver Version: 535.230.02 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA L4 Off | 00000000:38:00.0 Off | 0 | | N/A 53C P0 50W / 72W | 13853MiB / 23034MiB | 96% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ | 1 NVIDIA L4 Off | 00000000:3A:00.0 Off | 0 | | N/A 43C P0 27W / 72W | 12727MiB / 23034MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ | 2 NVIDIA L4 Off | 00000000:3C:00.0 Off | 0 | | N/A 41C P0 26W / 72W | 12727MiB / 23034MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ | 3 NVIDIA L4 Off | 00000000:3E:00.0 Off | 0 | | N/A 42C P0 27W / 72W | 12727MiB / 23034MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| +---------------------------------------------------------------------------------------+ ``` ### Relevant log output ```shell ``` ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version github latest
GiteaMirror added the bug label 2026-04-28 23:26:40 -05:00
GiteaMirror commented 2026-04-28 23:26:42 -05:00
Author
Owner
Copy Link

@cyberluke commented on GitHub (Feb 18, 2025):

BETTER LOG:

llm_load_tensors: offloading 44 repeating layers to GPU
llm_load_tensors: offloaded 44/81 layers to GPU

like where is hardcoded 44 layers only to GPU??????????

time=2025-02-18T02:29:51.195Z level=INFO source=server.go:97 msg="system memory" total="181.8 GiB" free="159.3 GiB" free_swap="32.0 GiB"
time=2025-02-18T02:29:51.627Z level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=81 layers.offload=44 layers.split=11,11,11,11 memory.available="[22.0 GiB 22.0 GiB 22.0 GiB 22.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="127.4 GiB" memory.required.partial="87.1 GiB" memory.required.kv="19.5 GiB" memory.required.allocations="[21.8 GiB 21.8 GiB 21.8 GiB 21.8 GiB]" memory.weights.total="87.3 GiB" memory.weights.repeating="86.2 GiB" memory.weights.nonrepeating="1.0 GiB" memory.graph.full="8.2 GiB" memory.graph.partial="8.2 GiB"
time=2025-02-18T02:29:51.627Z level=INFO source=server.go:182 msg="enabling flash attention"
time=2025-02-18T02:29:51.627Z level=WARN source=server.go:190 msg="kv cache type not supported by model" type=""
time=2025-02-18T02:29:51.627Z level=INFO source=server.go:380 msg="starting llama server" cmd="/teamspace/studios/this_studio/ollama/ollama runner --model /teamspace/studios/this_studio/.ollama/models/blobs/sha256-feef62aa06ab4162ebd3b9af4ff8383a37bf9544a7d30a3fe4623c8398bd1a28 --ctx-size 64000 --batch-size 512 --n-gpu-layers 44 --threads 24 --flash-attn --parallel 1 --tensor-split 11,11,11,11 --port 50143"
time=2025-02-18T02:29:51.628Z level=INFO source=sched.go:450 msg="loaded runners" count=1
time=2025-02-18T02:29:51.628Z level=INFO source=server.go:557 msg="waiting for llama runner to start responding"
time=2025-02-18T02:29:51.628Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error"
time=2025-02-18T02:29:51.644Z level=INFO source=runner.go:932 msg="starting go runner"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 CUDA devices:
  Device 0: NVIDIA L4, compute capability 8.9, VMM: yes
  Device 1: NVIDIA L4, compute capability 8.9, VMM: yes
  Device 2: NVIDIA L4, compute capability 8.9, VMM: yes
  Device 3: NVIDIA L4, compute capability 8.9, VMM: yes
load_backend: loaded CUDA backend from /teamspace/studios/this_studio/ollama/build/lib/ollama/libggml-cuda.so
load_backend: loaded CPU backend from /teamspace/studios/this_studio/ollama/build/lib/ollama/libggml-cpu-haswell.so
time=2025-02-18T02:29:52.013Z level=INFO source=runner.go:935 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | CUDA : ARCHS = 890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : LLAMAFILE = 1 | cgo(gcc)" threads=24
time=2025-02-18T02:29:52.014Z level=INFO source=runner.go:993 msg="Server listening on 127.0.0.1:50143"
llama_load_model_from_file: using device CUDA0 (NVIDIA L4) - 22510 MiB free
llama_load_model_from_file: using device CUDA1 (NVIDIA L4) - 22510 MiB free
llama_load_model_from_file: using device CUDA2 (NVIDIA L4) - 22510 MiB free
llama_load_model_from_file: using device CUDA3 (NVIDIA L4) - 22510 MiB free
time=2025-02-18T02:29:52.130Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model"
llama_model_loader: loaded meta data with 30 key-value pairs and 724 tensors from /teamspace/studios/this_studio/.ollama/models/blobs/sha256-feef62aa06ab4162ebd3b9af4ff8383a37bf9544a7d30a3fe4623c8398bd1a28 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = DeepSeek R1 Distill Llama 70B
llama_model_loader: - kv   3:                           general.basename str              = DeepSeek-R1-Distill-Llama
llama_model_loader: - kv   4:                         general.size_label str              = 70B
llama_model_loader: - kv   5:                          llama.block_count u32              = 80
llama_model_loader: - kv   6:                       llama.context_length u32              = 131072
llama_model_loader: - kv   7:                     llama.embedding_length u32              = 8192
llama_model_loader: - kv   8:                  llama.feed_forward_length u32              = 28672
llama_model_loader: - kv   9:                 llama.attention.head_count u32              = 64
llama_model_loader: - kv  10:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  11:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  12:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  13:                 llama.attention.key_length u32              = 128
llama_model_loader: - kv  14:               llama.attention.value_length u32              = 128
llama_model_loader: - kv  15:                          general.file_type u32              = 7
llama_model_loader: - kv  16:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  17:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  18:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  19:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  20:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  21:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  22:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  23:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  24:                tokenizer.ggml.eos_token_id u32              = 128001
llama_model_loader: - kv  25:            tokenizer.ggml.padding_token_id u32              = 128001
llama_model_loader: - kv  26:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  27:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  28:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
llama_model_loader: - kv  29:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  162 tensors
llama_model_loader: - type q8_0:  562 tensors
llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 256
llm_load_vocab: token to piece cache size = 0.7999 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 128256
llm_load_print_meta: n_merges         = 280147
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 131072
llm_load_print_meta: n_embd           = 8192
llm_load_print_meta: n_layer          = 80
llm_load_print_meta: n_head           = 64
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 8
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 28672
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 500000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 131072
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = 70B
llm_load_print_meta: model ftype      = Q8_0
llm_load_print_meta: model params     = 70.55 B
llm_load_print_meta: model size       = 69.82 GiB (8.50 BPW) 
llm_load_print_meta: general.name     = DeepSeek R1 Distill Llama 70B
llm_load_print_meta: BOS token        = 128000 '<｜begin▁of▁sentence｜>'
llm_load_print_meta: EOS token        = 128001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: EOT token        = 128009 '<|eot_id|>'
llm_load_print_meta: EOM token        = 128008 '<|eom_id|>'
llm_load_print_meta: PAD token        = 128001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: LF token         = 128 'Ä'
llm_load_print_meta: EOG token        = 128001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: EOG token        = 128008 '<|eom_id|>'
llm_load_print_meta: EOG token        = 128009 '<|eot_id|>'
llm_load_print_meta: max token length = 256
llm_load_tensors: offloading 44 repeating layers to GPU
llm_load_tensors: offloaded 44/81 layers to GPU
llm_load_tensors:   CPU_Mapped model buffer size = 33343.53 MiB
llm_load_tensors:        CUDA0 model buffer size =  9537.69 MiB
llm_load_tensors:        CUDA1 model buffer size =  9537.69 MiB
llm_load_tensors:        CUDA2 model buffer size =  9537.69 MiB
llm_load_tensors:        CUDA3 model buffer size =  9537.69 MiB
llama_new_context_with_model: n_seq_max     = 1
llama_new_context_with_model: n_ctx         = 64000
llama_new_context_with_model: n_ctx_per_seq = 64000
llama_new_context_with_model: n_batch       = 512
llama_new_context_with_model: n_ubatch      = 512
llama_new_context_with_model: flash_attn    = 1
llama_new_context_with_model: freq_base     = 500000.0
llama_new_context_with_model: freq_scale    = 1
llama_new_context_with_model: n_ctx_per_seq (64000) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_kv_cache_init: kv_size = 64000, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 80, can_shift = 1
llama_kv_cache_init:        CPU KV buffer size =  9000.00 MiB
llama_kv_cache_init:      CUDA0 KV buffer size =  2750.00 MiB
llama_kv_cache_init:      CUDA1 KV buffer size =  2750.00 MiB
llama_kv_cache_init:      CUDA2 KV buffer size =  2750.00 MiB
llama_kv_cache_init:      CUDA3 KV buffer size =  2750.00 MiB
llama_new_context_with_model: KV self size  = 20000.00 MiB, K (f16): 10000.00 MiB, V (f16): 10000.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.52 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =  1331.12 MiB
llama_new_context_with_model:      CUDA1 compute buffer size =   206.50 MiB
llama_new_context_with_model:      CUDA2 compute buffer size =   206.50 MiB
llama_new_context_with_model:      CUDA3 compute buffer size =   206.50 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =   141.01 MiB
llama_new_context_with_model: graph nodes  = 2247
llama_new_context_with_model: graph splits = 404 (with bs=512), 6 (with bs=1)
time=2025-02-18T02:30:06.180Z level=INFO source=server.go:596 msg="llama runner started in 14.55 seconds"
[GIN] 2025/02/18 - 02:30:06 | 200 | 20.436756556s |       127.0.0.1 | POST     "/api/generate"
<!-- gh-comment-id:2664469611 --> @cyberluke commented on GitHub (Feb 18, 2025): BETTER LOG: llm_load_tensors: offloading 44 repeating layers to GPU llm_load_tensors: offloaded 44/81 layers to GPU like where is hardcoded 44 layers only to GPU?????????? ``` time=2025-02-18T02:29:51.195Z level=INFO source=server.go:97 msg="system memory" total="181.8 GiB" free="159.3 GiB" free_swap="32.0 GiB" time=2025-02-18T02:29:51.627Z level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=81 layers.offload=44 layers.split=11,11,11,11 memory.available="[22.0 GiB 22.0 GiB 22.0 GiB 22.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="127.4 GiB" memory.required.partial="87.1 GiB" memory.required.kv="19.5 GiB" memory.required.allocations="[21.8 GiB 21.8 GiB 21.8 GiB 21.8 GiB]" memory.weights.total="87.3 GiB" memory.weights.repeating="86.2 GiB" memory.weights.nonrepeating="1.0 GiB" memory.graph.full="8.2 GiB" memory.graph.partial="8.2 GiB" time=2025-02-18T02:29:51.627Z level=INFO source=server.go:182 msg="enabling flash attention" time=2025-02-18T02:29:51.627Z level=WARN source=server.go:190 msg="kv cache type not supported by model" type="" time=2025-02-18T02:29:51.627Z level=INFO source=server.go:380 msg="starting llama server" cmd="/teamspace/studios/this_studio/ollama/ollama runner --model /teamspace/studios/this_studio/.ollama/models/blobs/sha256-feef62aa06ab4162ebd3b9af4ff8383a37bf9544a7d30a3fe4623c8398bd1a28 --ctx-size 64000 --batch-size 512 --n-gpu-layers 44 --threads 24 --flash-attn --parallel 1 --tensor-split 11,11,11,11 --port 50143" time=2025-02-18T02:29:51.628Z level=INFO source=sched.go:450 msg="loaded runners" count=1 time=2025-02-18T02:29:51.628Z level=INFO source=server.go:557 msg="waiting for llama runner to start responding" time=2025-02-18T02:29:51.628Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error" time=2025-02-18T02:29:51.644Z level=INFO source=runner.go:932 msg="starting go runner" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 4 CUDA devices: Device 0: NVIDIA L4, compute capability 8.9, VMM: yes Device 1: NVIDIA L4, compute capability 8.9, VMM: yes Device 2: NVIDIA L4, compute capability 8.9, VMM: yes Device 3: NVIDIA L4, compute capability 8.9, VMM: yes load_backend: loaded CUDA backend from /teamspace/studios/this_studio/ollama/build/lib/ollama/libggml-cuda.so load_backend: loaded CPU backend from /teamspace/studios/this_studio/ollama/build/lib/ollama/libggml-cpu-haswell.so time=2025-02-18T02:29:52.013Z level=INFO source=runner.go:935 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | CUDA : ARCHS = 890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : LLAMAFILE = 1 | cgo(gcc)" threads=24 time=2025-02-18T02:29:52.014Z level=INFO source=runner.go:993 msg="Server listening on 127.0.0.1:50143" llama_load_model_from_file: using device CUDA0 (NVIDIA L4) - 22510 MiB free llama_load_model_from_file: using device CUDA1 (NVIDIA L4) - 22510 MiB free llama_load_model_from_file: using device CUDA2 (NVIDIA L4) - 22510 MiB free llama_load_model_from_file: using device CUDA3 (NVIDIA L4) - 22510 MiB free time=2025-02-18T02:29:52.130Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model" llama_model_loader: loaded meta data with 30 key-value pairs and 724 tensors from /teamspace/studios/this_studio/.ollama/models/blobs/sha256-feef62aa06ab4162ebd3b9af4ff8383a37bf9544a7d30a3fe4623c8398bd1a28 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = DeepSeek R1 Distill Llama 70B llama_model_loader: - kv 3: general.basename str = DeepSeek-R1-Distill-Llama llama_model_loader: - kv 4: general.size_label str = 70B llama_model_loader: - kv 5: llama.block_count u32 = 80 llama_model_loader: - kv 6: llama.context_length u32 = 131072 llama_model_loader: - kv 7: llama.embedding_length u32 = 8192 llama_model_loader: - kv 8: llama.feed_forward_length u32 = 28672 llama_model_loader: - kv 9: llama.attention.head_count u32 = 64 llama_model_loader: - kv 10: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 11: llama.rope.freq_base f32 = 500000.000000 llama_model_loader: - kv 12: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 13: llama.attention.key_length u32 = 128 llama_model_loader: - kv 14: llama.attention.value_length u32 = 128 llama_model_loader: - kv 15: general.file_type u32 = 7 llama_model_loader: - kv 16: llama.vocab_size u32 = 128256 llama_model_loader: - kv 17: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 18: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 19: tokenizer.ggml.pre str = llama-bpe llama_model_loader: - kv 20: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 21: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 22: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... llama_model_loader: - kv 23: tokenizer.ggml.bos_token_id u32 = 128000 llama_model_loader: - kv 24: tokenizer.ggml.eos_token_id u32 = 128001 llama_model_loader: - kv 25: tokenizer.ggml.padding_token_id u32 = 128001 llama_model_loader: - kv 26: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 27: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 28: tokenizer.chat_template str = {% if not add_generation_prompt is de... llama_model_loader: - kv 29: general.quantization_version u32 = 2 llama_model_loader: - type f32: 162 tensors llama_model_loader: - type q8_0: 562 tensors llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect llm_load_vocab: special tokens cache size = 256 llm_load_vocab: token to piece cache size = 0.7999 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 128256 llm_load_print_meta: n_merges = 280147 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 131072 llm_load_print_meta: n_embd = 8192 llm_load_print_meta: n_layer = 80 llm_load_print_meta: n_head = 64 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 8 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 28672 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 500000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 131072 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = 70B llm_load_print_meta: model ftype = Q8_0 llm_load_print_meta: model params = 70.55 B llm_load_print_meta: model size = 69.82 GiB (8.50 BPW) llm_load_print_meta: general.name = DeepSeek R1 Distill Llama 70B llm_load_print_meta: BOS token = 128000 '<｜begin▁of▁sentence｜>' llm_load_print_meta: EOS token = 128001 '<｜end▁of▁sentence｜>' llm_load_print_meta: EOT token = 128009 '<|eot_id|>' llm_load_print_meta: EOM token = 128008 '<|eom_id|>' llm_load_print_meta: PAD token = 128001 '<｜end▁of▁sentence｜>' llm_load_print_meta: LF token = 128 'Ä' llm_load_print_meta: EOG token = 128001 '<｜end▁of▁sentence｜>' llm_load_print_meta: EOG token = 128008 '<|eom_id|>' llm_load_print_meta: EOG token = 128009 '<|eot_id|>' llm_load_print_meta: max token length = 256 llm_load_tensors: offloading 44 repeating layers to GPU llm_load_tensors: offloaded 44/81 layers to GPU llm_load_tensors: CPU_Mapped model buffer size = 33343.53 MiB llm_load_tensors: CUDA0 model buffer size = 9537.69 MiB llm_load_tensors: CUDA1 model buffer size = 9537.69 MiB llm_load_tensors: CUDA2 model buffer size = 9537.69 MiB llm_load_tensors: CUDA3 model buffer size = 9537.69 MiB llama_new_context_with_model: n_seq_max = 1 llama_new_context_with_model: n_ctx = 64000 llama_new_context_with_model: n_ctx_per_seq = 64000 llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 1 llama_new_context_with_model: freq_base = 500000.0 llama_new_context_with_model: freq_scale = 1 llama_new_context_with_model: n_ctx_per_seq (64000) < n_ctx_train (131072) -- the full capacity of the model will not be utilized llama_kv_cache_init: kv_size = 64000, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 80, can_shift = 1 llama_kv_cache_init: CPU KV buffer size = 9000.00 MiB llama_kv_cache_init: CUDA0 KV buffer size = 2750.00 MiB llama_kv_cache_init: CUDA1 KV buffer size = 2750.00 MiB llama_kv_cache_init: CUDA2 KV buffer size = 2750.00 MiB llama_kv_cache_init: CUDA3 KV buffer size = 2750.00 MiB llama_new_context_with_model: KV self size = 20000.00 MiB, K (f16): 10000.00 MiB, V (f16): 10000.00 MiB llama_new_context_with_model: CPU output buffer size = 0.52 MiB llama_new_context_with_model: CUDA0 compute buffer size = 1331.12 MiB llama_new_context_with_model: CUDA1 compute buffer size = 206.50 MiB llama_new_context_with_model: CUDA2 compute buffer size = 206.50 MiB llama_new_context_with_model: CUDA3 compute buffer size = 206.50 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 141.01 MiB llama_new_context_with_model: graph nodes = 2247 llama_new_context_with_model: graph splits = 404 (with bs=512), 6 (with bs=1) time=2025-02-18T02:30:06.180Z level=INFO source=server.go:596 msg="llama runner started in 14.55 seconds" [GIN] 2025/02/18 - 02:30:06 | 200 | 20.436756556s | 127.0.0.1 | POST "/api/generate" ```
GiteaMirror commented 2026-04-28 23:26:43 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 18, 2025):

time=2025-02-18T02:29:51.627Z level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=81 
  layers.offload=44 layers.split=11,11,11,11 memory.available="[22.0 GiB 22.0 GiB 22.0 GiB 22.0 GiB]" memory.gpu_overhead="0 B" 
  memory.required.full="127.4 GiB" memory.required.partial="87.1 GiB" memory.required.kv="19.5 GiB" 
  memory.required.allocations="[21.8 GiB 21.8 GiB 21.8 GiB 21.8 GiB]" memory.weights.total="87.3 GiB" 
  memory.weights.repeating="86.2 GiB" memory.weights.nonrepeating="1.0 GiB" memory.graph.full="8.2 GiB" 
  memory.graph.partial="8.2 GiB"

You have 22GiB per card (memory.available). ollama is using 21.GiB of that (memory.required.allocations). There is only enough VRAM to hold 11 layers per card (layers.split). You have a large context window (64000) which requires 19.5GiB (memory.required.kv). If you want to fit more layers on the GPU, reduce num_ctx. Or you can use a smaller model, or the same model but more quantized, eg q4_K_M.

<!-- gh-comment-id:2664945988 --> @rick-github commented on GitHub (Feb 18, 2025): ``` time=2025-02-18T02:29:51.627Z level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=81 layers.offload=44 layers.split=11,11,11,11 memory.available="[22.0 GiB 22.0 GiB 22.0 GiB 22.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="127.4 GiB" memory.required.partial="87.1 GiB" memory.required.kv="19.5 GiB" memory.required.allocations="[21.8 GiB 21.8 GiB 21.8 GiB 21.8 GiB]" memory.weights.total="87.3 GiB" memory.weights.repeating="86.2 GiB" memory.weights.nonrepeating="1.0 GiB" memory.graph.full="8.2 GiB" memory.graph.partial="8.2 GiB" ``` You have 22GiB per card (`memory.available`). ollama is using 21.GiB of that (`memory.required.allocations`). There is only enough VRAM to hold 11 layers per card (`layers.split`). You have a large context window (64000) which requires 19.5GiB (`memory.required.kv`). If you want to fit more layers on the GPU, reduce `num_ctx`. Or you can use a smaller model, or the same model but more quantized, eg q4_K_M.
GiteaMirror commented 2026-04-28 23:26:44 -05:00
Author
Owner
Copy Link

@cyberluke commented on GitHub (Feb 19, 2025):

Not true, even I use 8x 24GB that is 192GB VRAM, it still sucks. Look at server.go in your source, you have bugs. Sorry I'm busy now. What you have in the logs and what you send to llama runner are two different things!!

<!-- gh-comment-id:2670028870 --> @cyberluke commented on GitHub (Feb 19, 2025): Not true, even I use 8x 24GB that is 192GB VRAM, it still sucks. Look at server.go in your source, you have bugs. Sorry I'm busy now. What you have in the logs and what you send to llama runner are two different things!!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#52491
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?