[GH-ISSUE #8923] llm_load_tensors: tensor 'token_embd.weight' (q4_K) (and 0 others) cannot be used with preferred buffer type CPU_AARCH64, using CPU instead #52299

New Issue

Closed

opened 2026-04-28 22:56:40 -05:00 by GiteaMirror · 3 comments

GiteaMirror commented 2026-04-28 22:56:40 -05:00

Owner

Copy Link

Originally created by @LwengGitHub on GitHub (Feb 7, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/8923

How to solve this problem? Intel(R)Xeon(R)Platinum 8336C CPU @ 2.30GHZ。Is it a Bug Or Must used CPU_AARCH64 ?

----------------------------------Log-Info---------------------------------

2025/02/07 09:38:48 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES:2 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:30m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2025-02-07T09:38:48.330Z level=INFO source=images.go:432 msg="total blobs: 5"
time=2025-02-07T09:38:48.353Z level=INFO source=images.go:439 msg="total unused blobs removed: 0"
time=2025-02-07T09:38:48.380Z level=INFO source=routes.go:1238 msg="Listening on [::]:11434 (version 0.5.7-0-ga420a45-dirty)"
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.

• using env: export GIN_MODE=release
• using code: gin.SetMode(gin.ReleaseMode)

[GIN-debug] POST /api/pull --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers)
[GIN-debug] POST /api/generate --> github.com/ollama/ollama/server.(Server).GenerateHandler-fm (5 handlers)
[GIN-debug] POST /api/chat --> github.com/ollama/ollama/server.(Server).ChatHandler-fm (5 handlers)
[GIN-debug] POST /api/embed --> github.com/ollama/ollama/server.(Server).EmbedHandler-fm (5 handlers)
[GIN-debug] POST /api/embeddings --> github.com/ollama/ollama/server.(Server).EmbeddingsHandler-fm (5 handlers)
[GIN-debug] POST /api/create --> github.com/ollama/ollama/server.(Server).CreateHandler-fm (5 handlers)
[GIN-debug] POST /api/push --> github.com/ollama/ollama/server.(Server).PushHandler-fm (5 handlers)
[GIN-debug] POST /api/copy --> github.com/ollama/ollama/server.(Server).CopyHandler-fm (5 handlers)
[GIN-debug] DELETE /api/delete --> github.com/ollama/ollama/server.(Server).DeleteHandler-fm (5 handlers)
[GIN-debug] POST /api/show --> github.com/ollama/ollama/server.(Server).ShowHandler-fm (5 handlers)
[GIN-debug] POST /api/blobs/:digest --> github.com/ollama/ollama/server.(Server).CreateBlobHandler-fm (5 handlers)
[GIN-debug] HEAD /api/blobs/:digest --> github.com/ollama/ollama/server.(Server).HeadBlobHandler-fm (5 handlers)
[GIN-debug] GET /api/ps --> github.com/ollama/ollama/server.(Server).PsHandler-fm (5 handlers)
[GIN-debug] POST /v1/chat/completions --> github.com/ollama/ollama/server.(Server).ChatHandler-fm (6 handlers)
[GIN-debug] POST /v1/completions --> github.com/ollama/ollama/server.(Server).GenerateHandler-fm (6 handlers)
[GIN-debug] POST /v1/embeddings --> github.com/ollama/ollama/server.(Server).EmbedHandler-fm (6 handlers)
[GIN-debug] GET /v1/models --> github.com/ollama/ollama/server.(Server).ListHandler-fm (6 handlers)
[GIN-debug] GET /v1/models/:model --> github.com/ollama/ollama/server.(Server).ShowHandler-fm (6 handlers)
[GIN-debug] GET / --> github.com/ollama/ollama/server.(Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] GET /api/tags --> github.com/ollama/ollama/server.(Server).ListHandler-fm (5 handlers)
[GIN-debug] GET /api/version --> github.com/ollama/ollama/server.(Server).GenerateRoutes.func2 (5 handlers)
[GIN-debug] HEAD / --> github.com/ollama/ollama/server.(Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] HEAD /api/tags --> github.com/ollama/ollama/server.(Server).ListHandler-fm (5 handlers)
[GIN-debug] HEAD /api/version --> github.com/ollama/ollama/server.(Server).GenerateRoutes.func2 (5 handlers)
time=2025-02-07T09:38:48.380Z level=DEBUG source=common.go:80 msg="runners located" dir=/usr/lib/ollama/runners
time=2025-02-07T09:38:48.381Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server
time=2025-02-07T09:38:48.381Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server
time=2025-02-07T09:38:48.381Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v11_avx/ollama_llama_server
time=2025-02-07T09:38:48.381Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v12_avx/ollama_llama_server
time=2025-02-07T09:38:48.381Z level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cpu_avx cpu_avx2 cuda_v11_avx cuda_v12_avx cpu]"
time=2025-02-07T09:38:48.381Z level=DEBUG source=routes.go:1268 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2025-02-07T09:38:48.381Z level=DEBUG source=sched.go:105 msg="starting llm scheduler"
time=2025-02-07T09:38:48.381Z level=INFO source=gpu.go:226 msg="looking for compatible GPUs"
time=2025-02-07T09:38:48.386Z level=DEBUG source=gpu.go:99 msg="searching for GPU discovery libraries for NVIDIA"
time=2025-02-07T09:38:48.386Z level=DEBUG source=gpu.go:517 msg="Searching for GPU library" name=libcuda.so
time=2025-02-07T09:38:48.386Z level=DEBUG source=gpu.go:543 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so /usr/lib/ollama/libcuda.so /usr/local/nvidia/lib/libcuda.so /usr/local/nvidia/lib64/libcuda.so /usr/local/cuda/targets//lib/libcuda.so /usr/lib/-linux-gnu/nvidia/current/libcuda.so /usr/lib/-linux-gnu/libcuda.so /usr/lib/wsl/lib/libcuda.so /usr/lib/wsl/drivers//libcuda.so /opt/cuda/lib/libcuda.so /usr/local/cuda/lib/libcuda.so /usr/lib/libcuda.so /usr/local/lib/libcuda.so]"
time=2025-02-07T09:38:48.387Z level=DEBUG source=gpu.go:576 msg="discovered GPU libraries" paths="[/usr/lib/x86_64-linux-gnu/libcuda.so.1 /usr/lib/wsl/drivers/nv_dispi.inf_amd64_adf5a840df867035/libcuda.so.1.1]"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1
dlsym: cuInit - 0x7fad9dc20ec0
dlsym: cuDriverGetVersion - 0x7fad9dc20e80
dlsym: cuDeviceGetCount - 0x7fad9dc20efd
dlsym: cuDeviceGet - 0x7fad9dc20ef7
dlsym: cuDeviceGetAttribute - 0x7fad9dc20de0
dlsym: cuDeviceGetUuid - 0x7fad9dc20f09
dlsym: cuDeviceGetName - 0x7fad9dc20f03
dlsym: cuCtxCreate_v3 - 0x7fad9dc20f7b
dlsym: cuMemGetInfo_v2 - 0x7fad9dc2107d
dlsym: cuCtxDestroy - 0x7fad9dc20f8d
calling cuInit
calling cuDriverGetVersion
raw version 0x2f26
CUDA driver version: 12.7
calling cuDeviceGetCount
device count 1
time=2025-02-07T09:38:48.475Z level=DEBUG source=gpu.go:134 msg="detected GPUs" count=1 library=/usr/lib/x86_64-linux-gnu/libcuda.so.1
[GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8] CUDA totalMem 24563 mb
[GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8] CUDA freeMem 23042 mb
[GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8] Compute Capability 8.9
time=2025-02-07T09:38:48.649Z level=DEBUG source=amd_linux.go:421 msg="amdgpu driver not detected /sys/module/amdgpu"
releasing cuda driver library
time=2025-02-07T09:38:48.650Z level=INFO source=types.go:131 msg="inference compute" id=GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8 library=cuda variant=v12 compute=8.9 driver=12.7 name="NVIDIA GeForce RTX 4090 D" total="24.0 GiB" available="22.5 GiB"
time=2025-02-07T09:39:04.454Z level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="62.6 GiB" before.free="44.5 GiB" before.free_swap="16.0 GiB" now.total="62.6 GiB" now.free="44.5 GiB" now.free_swap="16.0 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1
dlsym: cuInit - 0x7fad9dc20ec0
dlsym: cuDriverGetVersion - 0x7fad9dc20e80
dlsym: cuDeviceGetCount - 0x7fad9dc20efd
dlsym: cuDeviceGet - 0x7fad9dc20ef7
dlsym: cuDeviceGetAttribute - 0x7fad9dc20de0
dlsym: cuDeviceGetUuid - 0x7fad9dc20f09
dlsym: cuDeviceGetName - 0x7fad9dc20f03
dlsym: cuCtxCreate_v3 - 0x7fad9dc20f7b
dlsym: cuMemGetInfo_v2 - 0x7fad9dc2107d
dlsym: cuCtxDestroy - 0x7fad9dc20f8d
calling cuInit
calling cuDriverGetVersion
[GIN] 2025/02/07 - 09:38:55 | 200 | 31.54µs | 127.0.0.1 | HEAD "/"
[GIN] 2025/02/07 - 09:38:55 | 200 | 25.668935ms | 127.0.0.1 | GET "/api/tags"
[GIN] 2025/02/07 - 09:39:04 | 200 | 32.526µs | 127.0.0.1 | HEAD "/"
[GIN] 2025/02/07 - 09:39:04 | 200 | 103.398398ms | 127.0.0.1 | POST "/api/show"
raw version 0x2f26
CUDA driver version: 12.7
calling cuDeviceGetCount
device count 1
time=2025-02-07T09:39:04.586Z level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8 name="NVIDIA GeForce RTX 4090 D" overhead="0 B" before.total="24.0 GiB" before.free="22.5 GiB" now.total="24.0 GiB" now.free="22.5 GiB" now.used="1.5 GiB"
releasing cuda driver library
time=2025-02-07T09:39:04.586Z level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x5599decec4c0 gpu_count=1
time=2025-02-07T09:39:04.719Z level=DEBUG source=sched.go:224 msg="loading first model" model=/root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be
time=2025-02-07T09:39:04.719Z level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[22.5 GiB]"
time=2025-02-07T09:39:04.719Z level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="62.6 GiB" before.free="44.5 GiB" before.free_swap="16.0 GiB" now.total="62.6 GiB" now.free="44.5 GiB" now.free_swap="16.0 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1
dlsym: cuInit - 0x7fad9dc20ec0
dlsym: cuDriverGetVersion - 0x7fad9dc20e80
dlsym: cuDeviceGetCount - 0x7fad9dc20efd
dlsym: cuDeviceGet - 0x7fad9dc20ef7
dlsym: cuDeviceGetAttribute - 0x7fad9dc20de0
dlsym: cuDeviceGetUuid - 0x7fad9dc20f09
dlsym: cuDeviceGetName - 0x7fad9dc20f03
dlsym: cuCtxCreate_v3 - 0x7fad9dc20f7b
dlsym: cuMemGetInfo_v2 - 0x7fad9dc2107d
dlsym: cuCtxDestroy - 0x7fad9dc20f8d
calling cuInit
calling cuDriverGetVersion
raw version 0x2f26
CUDA driver version: 12.7
calling cuDeviceGetCount
device count 1
time=2025-02-07T09:39:04.841Z level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8 name="NVIDIA GeForce RTX 4090 D" overhead="0 B" before.total="24.0 GiB" before.free="22.5 GiB" now.total="24.0 GiB" now.free="22.5 GiB" now.used="1.5 GiB"
releasing cuda driver library
time=2025-02-07T09:39:04.841Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be gpu=GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8 parallel=1 available=24161288192 required="17.3 GiB"
time=2025-02-07T09:39:04.841Z level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="62.6 GiB" before.free="44.5 GiB" before.free_swap="16.0 GiB" now.total="62.6 GiB" now.free="44.5 GiB" now.free_swap="16.0 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1
dlsym: cuInit - 0x7fad9dc20ec0
dlsym: cuDriverGetVersion - 0x7fad9dc20e80
dlsym: cuDeviceGetCount - 0x7fad9dc20efd
dlsym: cuDeviceGet - 0x7fad9dc20ef7
dlsym: cuDeviceGetAttribute - 0x7fad9dc20de0
dlsym: cuDeviceGetUuid - 0x7fad9dc20f09
dlsym: cuDeviceGetName - 0x7fad9dc20f03
dlsym: cuCtxCreate_v3 - 0x7fad9dc20f7b
dlsym: cuMemGetInfo_v2 - 0x7fad9dc2107d
dlsym: cuCtxDestroy - 0x7fad9dc20f8d
calling cuInit
calling cuDriverGetVersion
raw version 0x2f26
CUDA driver version: 12.7
calling cuDeviceGetCount
device count 1
time=2025-02-07T09:39:04.960Z level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8 name="NVIDIA GeForce RTX 4090 D" overhead="0 B" before.total="24.0 GiB" before.free="22.5 GiB" now.total="24.0 GiB" now.free="22.5 GiB" now.used="1.5 GiB"
releasing cuda driver library
time=2025-02-07T09:39:04.960Z level=INFO source=server.go:104 msg="system memory" total="62.6 GiB" free="44.5 GiB" free_swap="16.0 GiB"
time=2025-02-07T09:39:04.960Z level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[22.5 GiB]"
time=2025-02-07T09:39:04.961Z level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="62.6 GiB" before.free="44.5 GiB" before.free_swap="16.0 GiB" now.total="62.6 GiB" now.free="44.5 GiB" now.free_swap="16.0 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1
dlsym: cuInit - 0x7fad9dc20ec0
dlsym: cuDriverGetVersion - 0x7fad9dc20e80
dlsym: cuDeviceGetCount - 0x7fad9dc20efd
dlsym: cuDeviceGet - 0x7fad9dc20ef7
dlsym: cuDeviceGetAttribute - 0x7fad9dc20de0
dlsym: cuDeviceGetUuid - 0x7fad9dc20f09
dlsym: cuDeviceGetName - 0x7fad9dc20f03
dlsym: cuCtxCreate_v3 - 0x7fad9dc20f7b
dlsym: cuMemGetInfo_v2 - 0x7fad9dc2107d
dlsym: cuCtxDestroy - 0x7fad9dc20f8d
calling cuInit
calling cuDriverGetVersion
raw version 0x2f26
CUDA driver version: 12.7
calling cuDeviceGetCount
device count 1
time=2025-02-07T09:39:05.074Z level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8 name="NVIDIA GeForce RTX 4090 D" overhead="0 B" before.total="24.0 GiB" before.free="22.5 GiB" now.total="24.0 GiB" now.free="22.5 GiB" now.used="1.5 GiB"
releasing cuda driver library
time=2025-02-07T09:39:05.074Z level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[22.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="17.3 GiB" memory.required.partial="17.3 GiB" memory.required.kv="8.0 GiB" memory.required.allocations="[17.3 GiB]" memory.weights.total="11.9 GiB" memory.weights.repeating="11.5 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="4.2 GiB" memory.graph.partial="4.4 GiB"
time=2025-02-07T09:39:05.074Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server
time=2025-02-07T09:39:05.074Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server
time=2025-02-07T09:39:05.074Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v11_avx/ollama_llama_server
time=2025-02-07T09:39:05.074Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v12_avx/ollama_llama_server
time=2025-02-07T09:39:05.074Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server
time=2025-02-07T09:39:05.075Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server
time=2025-02-07T09:39:05.075Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v11_avx/ollama_llama_server
time=2025-02-07T09:39:05.075Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v12_avx/ollama_llama_server
time=2025-02-07T09:39:05.075Z level=INFO source=server.go:223 msg="enabling flash attention"
time=2025-02-07T09:39:05.075Z level=WARN source=server.go:231 msg="kv cache type not supported by model" type=""
time=2025-02-07T09:39:05.075Z level=INFO source=server.go:376 msg="starting llama server" cmd="/usr/lib/ollama/runners/cuda_v12_avx/ollama_llama_server runner --model /root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be --ctx-size 65536 --batch-size 512 --n-gpu-layers 33 --verbose --threads 64 --flash-attn --parallel 1 --port 37207"
time=2025-02-07T09:39:05.075Z level=DEBUG source=server.go:393 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin CUDA_VISIBLE_DEVICES=GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8 LD_LIBRARY_PATH=/usr/lib/ollama:/usr/lib/ollama:/usr/lib/ollama/runners/cuda_v12_avx:/usr/local/nvidia/lib:/usr/local/nvidia/lib64]"
time=2025-02-07T09:39:05.075Z level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-02-07T09:39:05.075Z level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
time=2025-02-07T09:39:05.076Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
time=2025-02-07T09:39:05.107Z level=INFO source=runner.go:936 msg="starting go runner"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 4090 D, compute capability 8.9, VMM: yes
time=2025-02-07T09:39:05.197Z level=INFO source=runner.go:937 msg=system info="CUDA : ARCHS = 600,610,620,700,720,750,800,860,870,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=64
time=2025-02-07T09:39:05.197Z level=INFO source=.:0 msg="Server listening on 127.0.0.1:37207"
llama_load_model_from_file: using device CUDA0 (NVIDIA GeForce RTX 4090 D) - 23042 MiB free
time=2025-02-07T09:39:05.327Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server loading model"
llama_model_loader: loaded meta data with 28 key-value pairs and 292 tensors from /root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = DeepSeek R1 Distill Llama 8B
llama_model_loader: - kv 3: general.basename str = DeepSeek-R1-Distill-Llama
llama_model_loader: - kv 4: general.size_label str = 8B
llama_model_loader: - kv 5: llama.block_count u32 = 32
llama_model_loader: - kv 6: llama.context_length u32 = 131072
llama_model_loader: - kv 7: llama.embedding_length u32 = 4096
llama_model_loader: - kv 8: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 9: llama.attention.head_count u32 = 32
llama_model_loader: - kv 10: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 11: llama.rope.freq_base f32 = 500000.000000
llama_model_loader: - kv 12: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 13: general.file_type u32 = 15
llama_model_loader: - kv 14: llama.vocab_size u32 = 128256
llama_model_loader: - kv 15: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 16: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 17: tokenizer.ggml.pre str = llama-bpe
llama_model_loader: - kv 18: tokenizer.ggml.tokens arr[str,128256] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 19: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 20: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv 21: tokenizer.ggml.bos_token_id u32 = 128000
llama_model_loader: - kv 22: tokenizer.ggml.eos_token_id u32 = 128001
llama_model_loader: - kv 23: tokenizer.ggml.padding_token_id u32 = 128001
llama_model_loader: - kv 24: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 25: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 26: tokenizer.chat_template str = {% if not add_generation_prompt is de...
llama_model_loader: - kv 27: general.quantization_version u32 = 2
llama_model_loader: - type f32: 66 tensors
llama_model_loader: - type q4_K: 193 tensors
llama_model_loader: - type q6_K: 33 tensors
llm_load_vocab: control token: 128254 '<|reserved_special_token_246|>' is not marked as EOG
llm_load_vocab: control token: 128249 '<|reserved_special_token_241|>' is not marked as EOG
llm_load_vocab: control token: 128246 '<|reserved_special_token_238|>' is not marked as EOG
llm_load_vocab: control token: 128243 '<|reserved_special_token_235|>' is not marked as EOG
llm_load_vocab: control token: 128242 '<|reserved_special_token_234|>' is not marked as EOG
llm_load_vocab: control token: 128241 '<|reserved_special_token_233|>' is not marked as EOG
llm_load_vocab: control token: 128240 '<|reserved_special_token_232|>' is not marked as EOG
llm_load_vocab: control token: 128235 '<|reserved_special_token_227|>' is not marked as EOG
llm_load_vocab: control token: 128231 '<|reserved_special_token_223|>' is not marked as EOG
llm_load_vocab: control token: 128230 '<|reserved_special_token_222|>' is not marked as EOG
llm_load_vocab: control token: 128228 '<|reserved_special_token_220|>' is not marked as EOG
llm_load_vocab: control token: 128225 '<|reserved_special_token_217|>' is not marked as EOG
llm_load_vocab: control token: 128218 '<|reserved_special_token_210|>' is not marked as EOG
llm_load_vocab: control token: 128214 '<|reserved_special_token_206|>' is not marked as EOG
llm_load_vocab: control token: 128213 '<|reserved_special_token_205|>' is not marked as EOG
llm_load_vocab: control token: 128207 '<|reserved_special_token_199|>' is not marked as EOG
llm_load_vocab: control token: 128206 '<|reserved_special_token_198|>' is not marked as EOG
llm_load_vocab: control token: 128204 '<|reserved_special_token_196|>' is not marked as EOG
llm_load_vocab: control token: 128200 '<|reserved_special_token_192|>' is not marked as EOG
llm_load_vocab: control token: 128199 '<|reserved_special_token_191|>' is not marked as EOG
llm_load_vocab: control token: 128198 '<|reserved_special_token_190|>' is not marked as EOG
llm_load_vocab: control token: 128196 '<|reserved_special_token_188|>' is not marked as EOG
llm_load_vocab: control token: 128194 '<|reserved_special_token_186|>' is not marked as EOG
llm_load_vocab: control token: 128193 '<|reserved_special_token_185|>' is not marked as EOG
llm_load_vocab: control token: 128188 '<|reserved_special_token_180|>' is not marked as EOG
llm_load_vocab: control token: 128187 '<|reserved_special_token_179|>' is not marked as EOG
llm_load_vocab: control token: 128185 '<|reserved_special_token_177|>' is not marked as EOG
llm_load_vocab: control token: 128184 '<|reserved_special_token_176|>' is not marked as EOG
llm_load_vocab: control token: 128180 '<|reserved_special_token_172|>' is not marked as EOG
llm_load_vocab: control token: 128179 '<|reserved_special_token_171|>' is not marked as EOG
llm_load_vocab: control token: 128178 '<|reserved_special_token_170|>' is not marked as EOG
llm_load_vocab: control token: 128177 '<|reserved_special_token_169|>' is not marked as EOG
llm_load_vocab: control token: 128176 '<|reserved_special_token_168|>' is not marked as EOG
llm_load_vocab: control token: 128175 '<|reserved_special_token_167|>' is not marked as EOG
llm_load_vocab: control token: 128171 '<|reserved_special_token_163|>' is not marked as EOG
llm_load_vocab: control token: 128170 '<|reserved_special_token_162|>' is not marked as EOG
llm_load_vocab: control token: 128169 '<|reserved_special_token_161|>' is not marked as EOG
llm_load_vocab: control token: 128168 '<|reserved_special_token_160|>' is not marked as EOG
llm_load_vocab: control token: 128165 '<|reserved_special_token_157|>' is not marked as EOG
llm_load_vocab: control token: 128162 '<|reserved_special_token_154|>' is not marked as EOG
llm_load_vocab: control token: 128158 '<|reserved_special_token_150|>' is not marked as EOG
llm_load_vocab: control token: 128156 '<|reserved_special_token_148|>' is not marked as EOG
llm_load_vocab: control token: 128155 '<|reserved_special_token_147|>' is not marked as EOG
llm_load_vocab: control token: 128154 '<|reserved_special_token_146|>' is not marked as EOG
llm_load_vocab: control token: 128151 '<|reserved_special_token_143|>' is not marked as EOG
llm_load_vocab: control token: 128149 '<|reserved_special_token_141|>' is not marked as EOG
llm_load_vocab: control token: 128147 '<|reserved_special_token_139|>' is not marked as EOG
llm_load_vocab: control token: 128146 '<|reserved_special_token_138|>' is not marked as EOG
llm_load_vocab: control token: 128144 '<|reserved_special_token_136|>' is not marked as EOG
llm_load_vocab: control token: 128142 '<|reserved_special_token_134|>' is not marked as EOG
llm_load_vocab: control token: 128141 '<|reserved_special_token_133|>' is not marked as EOG
llm_load_vocab: control token: 128138 '<|reserved_special_token_130|>' is not marked as EOG
llm_load_vocab: control token: 128136 '<|reserved_special_token_128|>' is not marked as EOG
llm_load_vocab: control token: 128135 '<|reserved_special_token_127|>' is not marked as EOG
llm_load_vocab: control token: 128134 '<|reserved_special_token_126|>' is not marked as EOG
llm_load_vocab: control token: 128133 '<|reserved_special_token_125|>' is not marked as EOG
llm_load_vocab: control token: 128131 '<|reserved_special_token_123|>' is not marked as EOG
llm_load_vocab: control token: 128128 '<|reserved_special_token_120|>' is not marked as EOG
llm_load_vocab: control token: 128124 '<|reserved_special_token_116|>' is not marked as EOG
llm_load_vocab: control token: 128123 '<|reserved_special_token_115|>' is not marked as EOG
llm_load_vocab: control token: 128122 '<|reserved_special_token_114|>' is not marked as EOG
llm_load_vocab: control token: 128119 '<|reserved_special_token_111|>' is not marked as EOG
llm_load_vocab: control token: 128115 '<|reserved_special_token_107|>' is not marked as EOG
llm_load_vocab: control token: 128112 '<|reserved_special_token_104|>' is not marked as EOG
llm_load_vocab: control token: 128110 '<|reserved_special_token_102|>' is not marked as EOG
llm_load_vocab: control token: 128109 '<|reserved_special_token_101|>' is not marked as EOG
llm_load_vocab: control token: 128108 '<|reserved_special_token_100|>' is not marked as EOG
llm_load_vocab: control token: 128106 '<|reserved_special_token_98|>' is not marked as EOG
llm_load_vocab: control token: 128103 '<|reserved_special_token_95|>' is not marked as EOG
llm_load_vocab: control token: 128102 '<|reserved_special_token_94|>' is not marked as EOG
llm_load_vocab: control token: 128101 '<|reserved_special_token_93|>' is not marked as EOG
llm_load_vocab: control token: 128097 '<|reserved_special_token_89|>' is not marked as EOG
llm_load_vocab: control token: 128091 '<|reserved_special_token_83|>' is not marked as EOG
llm_load_vocab: control token: 128090 '<|reserved_special_token_82|>' is not marked as EOG
llm_load_vocab: control token: 128089 '<|reserved_special_token_81|>' is not marked as EOG
llm_load_vocab: control token: 128087 '<|reserved_special_token_79|>' is not marked as EOG
llm_load_vocab: control token: 128085 '<|reserved_special_token_77|>' is not marked as EOG
llm_load_vocab: control token: 128081 '<|reserved_special_token_73|>' is not marked as EOG
llm_load_vocab: control token: 128078 '<|reserved_special_token_70|>' is not marked as EOG
llm_load_vocab: control token: 128076 '<|reserved_special_token_68|>' is not marked as EOG
llm_load_vocab: control token: 128075 '<|reserved_special_token_67|>' is not marked as EOG
llm_load_vocab: control token: 128073 '<|reserved_special_token_65|>' is not marked as EOG
llm_load_vocab: control token: 128068 '<|reserved_special_token_60|>' is not marked as EOG
llm_load_vocab: control token: 128067 '<|reserved_special_token_59|>' is not marked as EOG
llm_load_vocab: control token: 128065 '<|reserved_special_token_57|>' is not marked as EOG
llm_load_vocab: control token: 128063 '<|reserved_special_token_55|>' is not marked as EOG
llm_load_vocab: control token: 128062 '<|reserved_special_token_54|>' is not marked as EOG
llm_load_vocab: control token: 128060 '<|reserved_special_token_52|>' is not marked as EOG
llm_load_vocab: control token: 128059 '<|reserved_special_token_51|>' is not marked as EOG
llm_load_vocab: control token: 128057 '<|reserved_special_token_49|>' is not marked as EOG
llm_load_vocab: control token: 128054 '<|reserved_special_token_46|>' is not marked as EOG
llm_load_vocab: control token: 128046 '<|reserved_special_token_38|>' is not marked as EOG
llm_load_vocab: control token: 128045 '<|reserved_special_token_37|>' is not marked as EOG
llm_load_vocab: control token: 128044 '<|reserved_special_token_36|>' is not marked as EOG
llm_load_vocab: control token: 128043 '<|reserved_special_token_35|>' is not marked as EOG
llm_load_vocab: control token: 128038 '<|reserved_special_token_30|>' is not marked as EOG
llm_load_vocab: control token: 128036 '<|reserved_special_token_28|>' is not marked as EOG
llm_load_vocab: control token: 128035 '<|reserved_special_token_27|>' is not marked as EOG
llm_load_vocab: control token: 128032 '<|reserved_special_token_24|>' is not marked as EOG
llm_load_vocab: control token: 128028 '<|reserved_special_token_20|>' is not marked as EOG
llm_load_vocab: control token: 128027 '<|reserved_special_token_19|>' is not marked as EOG
llm_load_vocab: control token: 128024 '<|reserved_special_token_16|>' is not marked as EOG
llm_load_vocab: control token: 128023 '<|reserved_special_token_15|>' is not marked as EOG
llm_load_vocab: control token: 128022 '<|reserved_special_token_14|>' is not marked as EOG
llm_load_vocab: control token: 128021 '<|reserved_special_token_13|>' is not marked as EOG
llm_load_vocab: control token: 128018 '<|reserved_special_token_10|>' is not marked as EOG
llm_load_vocab: control token: 128016 '<|reserved_special_token_8|>' is not marked as EOG
llm_load_vocab: control token: 128012 '<｜Assistant｜>' is not marked as EOG
llm_load_vocab: control token: 128011 '<｜User｜>' is not marked as EOG
llm_load_vocab: control token: 128005 '<|reserved_special_token_2|>' is not marked as EOG
llm_load_vocab: control token: 128004 '<|finetune_right_pad_id|>' is not marked as EOG
llm_load_vocab: control token: 128002 '<|reserved_special_token_0|>' is not marked as EOG
llm_load_vocab: control token: 128252 '<|reserved_special_token_244|>' is not marked as EOG
llm_load_vocab: control token: 128190 '<|reserved_special_token_182|>' is not marked as EOG
llm_load_vocab: control token: 128183 '<|reserved_special_token_175|>' is not marked as EOG
llm_load_vocab: control token: 128137 '<|reserved_special_token_129|>' is not marked as EOG
llm_load_vocab: control token: 128182 '<|reserved_special_token_174|>' is not marked as EOG
llm_load_vocab: control token: 128040 '<|reserved_special_token_32|>' is not marked as EOG
llm_load_vocab: control token: 128048 '<|reserved_special_token_40|>' is not marked as EOG
llm_load_vocab: control token: 128092 '<|reserved_special_token_84|>' is not marked as EOG
llm_load_vocab: control token: 128215 '<|reserved_special_token_207|>' is not marked as EOG
llm_load_vocab: control token: 128107 '<|reserved_special_token_99|>' is not marked as EOG
llm_load_vocab: control token: 128208 '<|reserved_special_token_200|>' is not marked as EOG
llm_load_vocab: control token: 128145 '<|reserved_special_token_137|>' is not marked as EOG
llm_load_vocab: control token: 128031 '<|reserved_special_token_23|>' is not marked as EOG
llm_load_vocab: control token: 128129 '<|reserved_special_token_121|>' is not marked as EOG
llm_load_vocab: control token: 128201 '<|reserved_special_token_193|>' is not marked as EOG
llm_load_vocab: control token: 128074 '<|reserved_special_token_66|>' is not marked as EOG
llm_load_vocab: control token: 128095 '<|reserved_special_token_87|>' is not marked as EOG
llm_load_vocab: control token: 128186 '<|reserved_special_token_178|>' is not marked as EOG
llm_load_vocab: control token: 128143 '<|reserved_special_token_135|>' is not marked as EOG
llm_load_vocab: control token: 128229 '<|reserved_special_token_221|>' is not marked as EOG
llm_load_vocab: control token: 128007 '<|end_header_id|>' is not marked as EOG
llm_load_vocab: control token: 128055 '<|reserved_special_token_47|>' is not marked as EOG
llm_load_vocab: control token: 128056 '<|reserved_special_token_48|>' is not marked as EOG
llm_load_vocab: control token: 128061 '<|reserved_special_token_53|>' is not marked as EOG
llm_load_vocab: control token: 128153 '<|reserved_special_token_145|>' is not marked as EOG
llm_load_vocab: control token: 128152 '<|reserved_special_token_144|>' is not marked as EOG
llm_load_vocab: control token: 128212 '<|reserved_special_token_204|>' is not marked as EOG
llm_load_vocab: control token: 128172 '<|reserved_special_token_164|>' is not marked as EOG
llm_load_vocab: control token: 128160 '<|reserved_special_token_152|>' is not marked as EOG
llm_load_vocab: control token: 128041 '<|reserved_special_token_33|>' is not marked as EOG
llm_load_vocab: control token: 128181 '<|reserved_special_token_173|>' is not marked as EOG
llm_load_vocab: control token: 128094 '<|reserved_special_token_86|>' is not marked as EOG
llm_load_vocab: control token: 128118 '<|reserved_special_token_110|>' is not marked as EOG
llm_load_vocab: control token: 128236 '<|reserved_special_token_228|>' is not marked as EOG
llm_load_vocab: control token: 128148 '<|reserved_special_token_140|>' is not marked as EOG
llm_load_vocab: control token: 128042 '<|reserved_special_token_34|>' is not marked as EOG
llm_load_vocab: control token: 128139 '<|reserved_special_token_131|>' is not marked as EOG
llm_load_vocab: control token: 128173 '<|reserved_special_token_165|>' is not marked as EOG
llm_load_vocab: control token: 128239 '<|reserved_special_token_231|>' is not marked as EOG
llm_load_vocab: control token: 128157 '<|reserved_special_token_149|>' is not marked as EOG
llm_load_vocab: control token: 128052 '<|reserved_special_token_44|>' is not marked as EOG
llm_load_vocab: control token: 128026 '<|reserved_special_token_18|>' is not marked as EOG
llm_load_vocab: control token: 128003 '<|reserved_special_token_1|>' is not marked as EOG
llm_load_vocab: control token: 128019 '<|reserved_special_token_11|>' is not marked as EOG
llm_load_vocab: control token: 128116 '<|reserved_special_token_108|>' is not marked as EOG
llm_load_vocab: control token: 128161 '<|reserved_special_token_153|>' is not marked as EOG
llm_load_vocab: control token: 128000 '<｜begin▁of▁sentence｜>' is not marked as EOG
llm_load_vocab: control token: 128226 '<|reserved_special_token_218|>' is not marked as EOG
llm_load_vocab: control token: 128159 '<|reserved_special_token_151|>' is not marked as EOG
llm_load_vocab: control token: 128088 '<|reserved_special_token_80|>' is not marked as EOG
llm_load_vocab: control token: 128163 '<|reserved_special_token_155|>' is not marked as EOG
llm_load_vocab: control token: 128113 '<|reserved_special_token_105|>' is not marked as EOG
llm_load_vocab: control token: 128250 '<|reserved_special_token_242|>' is not marked as EOG
llm_load_vocab: control token: 128125 '<|reserved_special_token_117|>' is not marked as EOG
llm_load_vocab: control token: 128053 '<|reserved_special_token_45|>' is not marked as EOG
llm_load_vocab: control token: 128224 '<|reserved_special_token_216|>' is not marked as EOG
llm_load_vocab: control token: 128247 '<|reserved_special_token_239|>' is not marked as EOG
llm_load_vocab: control token: 128251 '<|reserved_special_token_243|>' is not marked as EOG
llm_load_vocab: control token: 128216 '<|reserved_special_token_208|>' is not marked as EOG
llm_load_vocab: control token: 128006 '<|start_header_id|>' is not marked as EOG
llm_load_vocab: control token: 128211 '<|reserved_special_token_203|>' is not marked as EOG
llm_load_vocab: control token: 128077 '<|reserved_special_token_69|>' is not marked as EOG
llm_load_vocab: control token: 128237 '<|reserved_special_token_229|>' is not marked as EOG
llm_load_vocab: control token: 128086 '<|reserved_special_token_78|>' is not marked as EOG
llm_load_vocab: control token: 128001 '<｜end▁of▁sentence｜>' is not marked as EOG
llm_load_vocab: control token: 128227 '<|reserved_special_token_219|>' is not marked as EOG
llm_load_vocab: control token: 128058 '<|reserved_special_token_50|>' is not marked as EOG
llm_load_vocab: control token: 128100 '<|reserved_special_token_92|>' is not marked as EOG
llm_load_vocab: control token: 128209 '<|reserved_special_token_201|>' is not marked as EOG
llm_load_vocab: control token: 128084 '<|reserved_special_token_76|>' is not marked as EOG
llm_load_vocab: control token: 128071 '<|reserved_special_token_63|>' is not marked as EOG
llm_load_vocab: control token: 128070 '<|reserved_special_token_62|>' is not marked as EOG
llm_load_vocab: control token: 128049 '<|reserved_special_token_41|>' is not marked as EOG
llm_load_vocab: control token: 128197 '<|reserved_special_token_189|>' is not marked as EOG
llm_load_vocab: control token: 128072 '<|reserved_special_token_64|>' is not marked as EOG
llm_load_vocab: control token: 128223 '<|reserved_special_token_215|>' is not marked as EOG
llm_load_vocab: control token: 128217 '<|reserved_special_token_209|>' is not marked as EOG
llm_load_vocab: control token: 128111 '<|reserved_special_token_103|>' is not marked as EOG
llm_load_vocab: control token: 128203 '<|reserved_special_token_195|>' is not marked as EOG
llm_load_vocab: control token: 128051 '<|reserved_special_token_43|>' is not marked as EOG
llm_load_vocab: control token: 128030 '<|reserved_special_token_22|>' is not marked as EOG
llm_load_vocab: control token: 128117 '<|reserved_special_token_109|>' is not marked as EOG
llm_load_vocab: control token: 128010 '<|python_tag|>' is not marked as EOG
llm_load_vocab: control token: 128238 '<|reserved_special_token_230|>' is not marked as EOG
llm_load_vocab: control token: 128255 '<|reserved_special_token_247|>' is not marked as EOG
llm_load_vocab: control token: 128202 '<|reserved_special_token_194|>' is not marked as EOG
llm_load_vocab: control token: 128132 '<|reserved_special_token_124|>' is not marked as EOG
llm_load_vocab: control token: 128248 '<|reserved_special_token_240|>' is not marked as EOG
llm_load_vocab: control token: 128167 '<|reserved_special_token_159|>' is not marked as EOG
llm_load_vocab: control token: 128127 '<|reserved_special_token_119|>' is not marked as EOG
llm_load_vocab: control token: 128105 '<|reserved_special_token_97|>' is not marked as EOG
llm_load_vocab: control token: 128039 '<|reserved_special_token_31|>' is not marked as EOG
llm_load_vocab: control token: 128232 '<|reserved_special_token_224|>' is not marked as EOG
llm_load_vocab: control token: 128166 '<|reserved_special_token_158|>' is not marked as EOG
llm_load_vocab: control token: 128130 '<|reserved_special_token_122|>' is not marked as EOG
llm_load_vocab: control token: 128114 '<|reserved_special_token_106|>' is not marked as EOG
llm_load_vocab: control token: 128234 '<|reserved_special_token_226|>' is not marked as EOG
llm_load_vocab: control token: 128191 '<|reserved_special_token_183|>' is not marked as EOG
llm_load_vocab: control token: 128064 '<|reserved_special_token_56|>' is not marked as EOG
llm_load_vocab: control token: 128140 '<|reserved_special_token_132|>' is not marked as EOG
llm_load_vocab: control token: 128096 '<|reserved_special_token_88|>' is not marked as EOG
llm_load_vocab: control token: 128098 '<|reserved_special_token_90|>' is not marked as EOG
llm_load_vocab: control token: 128192 '<|reserved_special_token_184|>' is not marked as EOG
llm_load_vocab: control token: 128093 '<|reserved_special_token_85|>' is not marked as EOG
llm_load_vocab: control token: 128150 '<|reserved_special_token_142|>' is not marked as EOG
llm_load_vocab: control token: 128222 '<|reserved_special_token_214|>' is not marked as EOG
llm_load_vocab: control token: 128233 '<|reserved_special_token_225|>' is not marked as EOG
llm_load_vocab: control token: 128220 '<|reserved_special_token_212|>' is not marked as EOG
llm_load_vocab: control token: 128034 '<|reserved_special_token_26|>' is not marked as EOG
llm_load_vocab: control token: 128033 '<|reserved_special_token_25|>' is not marked as EOG
llm_load_vocab: control token: 128253 '<|reserved_special_token_245|>' is not marked as EOG
llm_load_vocab: control token: 128195 '<|reserved_special_token_187|>' is not marked as EOG
llm_load_vocab: control token: 128099 '<|reserved_special_token_91|>' is not marked as EOG
llm_load_vocab: control token: 128189 '<|reserved_special_token_181|>' is not marked as EOG
llm_load_vocab: control token: 128210 '<|reserved_special_token_202|>' is not marked as EOG
llm_load_vocab: control token: 128174 '<|reserved_special_token_166|>' is not marked as EOG
llm_load_vocab: control token: 128083 '<|reserved_special_token_75|>' is not marked as EOG
llm_load_vocab: control token: 128080 '<|reserved_special_token_72|>' is not marked as EOG
llm_load_vocab: control token: 128104 '<|reserved_special_token_96|>' is not marked as EOG
llm_load_vocab: control token: 128082 '<|reserved_special_token_74|>' is not marked as EOG
llm_load_vocab: control token: 128219 '<|reserved_special_token_211|>' is not marked as EOG
llm_load_vocab: control token: 128017 '<|reserved_special_token_9|>' is not marked as EOG
llm_load_vocab: control token: 128050 '<|reserved_special_token_42|>' is not marked as EOG
llm_load_vocab: control token: 128205 '<|reserved_special_token_197|>' is not marked as EOG
llm_load_vocab: control token: 128047 '<|reserved_special_token_39|>' is not marked as EOG
llm_load_vocab: control token: 128164 '<|reserved_special_token_156|>' is not marked as EOG
llm_load_vocab: control token: 128020 '<|reserved_special_token_12|>' is not marked as EOG
llm_load_vocab: control token: 128069 '<|reserved_special_token_61|>' is not marked as EOG
llm_load_vocab: control token: 128245 '<|reserved_special_token_237|>' is not marked as EOG
llm_load_vocab: control token: 128121 '<|reserved_special_token_113|>' is not marked as EOG
llm_load_vocab: control token: 128079 '<|reserved_special_token_71|>' is not marked as EOG
llm_load_vocab: control token: 128037 '<|reserved_special_token_29|>' is not marked as EOG
llm_load_vocab: control token: 128244 '<|reserved_special_token_236|>' is not marked as EOG
llm_load_vocab: control token: 128029 '<|reserved_special_token_21|>' is not marked as EOG
llm_load_vocab: control token: 128221 '<|reserved_special_token_213|>' is not marked as EOG
llm_load_vocab: control token: 128066 '<|reserved_special_token_58|>' is not marked as EOG
llm_load_vocab: control token: 128120 '<|reserved_special_token_112|>' is not marked as EOG
llm_load_vocab: control token: 128015 '<｜▁pad▁｜>' is not marked as EOG
llm_load_vocab: control token: 128025 '<|reserved_special_token_17|>' is not marked as EOG
llm_load_vocab: control token: 128126 '<|reserved_special_token_118|>' is not marked as EOG
llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 256
llm_load_vocab: token to piece cache size = 0.7999 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 128256
llm_load_print_meta: n_merges = 280147
llm_load_print_meta: vocab_only = 0
llm_load_print_meta: n_ctx_train = 131072
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 500000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn = 131072
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: ssm_dt_b_c_rms = 0
llm_load_print_meta: model type = 8B
llm_load_print_meta: model ftype = Q4_K - Medium
llm_load_print_meta: model params = 8.03 B
llm_load_print_meta: model size = 4.58 GiB (4.89 BPW)
llm_load_print_meta: general.name = DeepSeek R1 Distill Llama 8B
llm_load_print_meta: BOS token = 128000 '<｜begin▁of▁sentence｜>'
llm_load_print_meta: EOS token = 128001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: EOT token = 128009 '<|eot_id|>'
llm_load_print_meta: EOM token = 128008 '<|eom_id|>'
llm_load_print_meta: PAD token = 128001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: LF token = 128 'Ä'
llm_load_print_meta: EOG token = 128001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: EOG token = 128008 '<|eom_id|>'
llm_load_print_meta: EOG token = 128009 '<|eot_id|>'
llm_load_print_meta: max token length = 256
llm_load_tensors: tensor 'token_embd.weight' (q4_K) (and 0 others) cannot be used with preferred buffer type CPU_AARCH64, using CPU instead

Originally created by @LwengGitHub on GitHub (Feb 7, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/8923 How to solve this problem? Intel(R)Xeon(R)Platinum 8336C CPU @ 2.30GHZ。Is it a Bug Or Must used CPU_AARCH64 ? ----------------------------------Log-Info--------------------------------- 2025/02/07 09:38:48 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES:2 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:30m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2025-02-07T09:38:48.330Z level=INFO source=images.go:432 msg="total blobs: 5" time=2025-02-07T09:38:48.353Z level=INFO source=images.go:439 msg="total unused blobs removed: 0" time=2025-02-07T09:38:48.380Z level=INFO source=routes.go:1238 msg="Listening on [::]:11434 (version 0.5.7-0-ga420a45-dirty)" [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached. [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production. - using env: export GIN_MODE=release - using code: gin.SetMode(gin.ReleaseMode) [GIN-debug] POST /api/pull --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers) [GIN-debug] POST /api/generate --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers) [GIN-debug] POST /api/chat --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers) [GIN-debug] POST /api/embed --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers) [GIN-debug] POST /api/embeddings --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers) [GIN-debug] POST /api/create --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers) [GIN-debug] POST /api/push --> github.com/ollama/ollama/server.(*Server).PushHandler-fm (5 handlers) [GIN-debug] POST /api/copy --> github.com/ollama/ollama/server.(*Server).CopyHandler-fm (5 handlers) [GIN-debug] DELETE /api/delete --> github.com/ollama/ollama/server.(*Server).DeleteHandler-fm (5 handlers) [GIN-debug] POST /api/show --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (5 handlers) [GIN-debug] POST /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers) [GIN-debug] HEAD /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers) [GIN-debug] GET /api/ps --> github.com/ollama/ollama/server.(*Server).PsHandler-fm (5 handlers) [GIN-debug] POST /v1/chat/completions --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers) [GIN-debug] POST /v1/completions --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (6 handlers) [GIN-debug] POST /v1/embeddings --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (6 handlers) [GIN-debug] GET /v1/models --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (6 handlers) [GIN-debug] GET /v1/models/:model --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (6 handlers) [GIN-debug] GET / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers) [GIN-debug] GET /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers) [GIN-debug] GET /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers) [GIN-debug] HEAD / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers) [GIN-debug] HEAD /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers) [GIN-debug] HEAD /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers) time=2025-02-07T09:38:48.380Z level=DEBUG source=common.go:80 msg="runners located" dir=/usr/lib/ollama/runners time=2025-02-07T09:38:48.381Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server time=2025-02-07T09:38:48.381Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server time=2025-02-07T09:38:48.381Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v11_avx/ollama_llama_server time=2025-02-07T09:38:48.381Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v12_avx/ollama_llama_server time=2025-02-07T09:38:48.381Z level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cpu_avx cpu_avx2 cuda_v11_avx cuda_v12_avx cpu]" time=2025-02-07T09:38:48.381Z level=DEBUG source=routes.go:1268 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY" time=2025-02-07T09:38:48.381Z level=DEBUG source=sched.go:105 msg="starting llm scheduler" time=2025-02-07T09:38:48.381Z level=INFO source=gpu.go:226 msg="looking for compatible GPUs" time=2025-02-07T09:38:48.386Z level=DEBUG source=gpu.go:99 msg="searching for GPU discovery libraries for NVIDIA" time=2025-02-07T09:38:48.386Z level=DEBUG source=gpu.go:517 msg="Searching for GPU library" name=libcuda.so* time=2025-02-07T09:38:48.386Z level=DEBUG source=gpu.go:543 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /usr/lib/ollama/libcuda.so* /usr/local/nvidia/lib/libcuda.so* /usr/local/nvidia/lib64/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" time=2025-02-07T09:38:48.387Z level=DEBUG source=gpu.go:576 msg="discovered GPU libraries" paths="[/usr/lib/x86_64-linux-gnu/libcuda.so.1 /usr/lib/wsl/drivers/nv_dispi.inf_amd64_adf5a840df867035/libcuda.so.1.1]" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1 dlsym: cuInit - 0x7fad9dc20ec0 dlsym: cuDriverGetVersion - 0x7fad9dc20e80 dlsym: cuDeviceGetCount - 0x7fad9dc20efd dlsym: cuDeviceGet - 0x7fad9dc20ef7 dlsym: cuDeviceGetAttribute - 0x7fad9dc20de0 dlsym: cuDeviceGetUuid - 0x7fad9dc20f09 dlsym: cuDeviceGetName - 0x7fad9dc20f03 dlsym: cuCtxCreate_v3 - 0x7fad9dc20f7b dlsym: cuMemGetInfo_v2 - 0x7fad9dc2107d dlsym: cuCtxDestroy - 0x7fad9dc20f8d calling cuInit calling cuDriverGetVersion raw version 0x2f26 CUDA driver version: 12.7 calling cuDeviceGetCount device count 1 time=2025-02-07T09:38:48.475Z level=DEBUG source=gpu.go:134 msg="detected GPUs" count=1 library=/usr/lib/x86_64-linux-gnu/libcuda.so.1 [GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8] CUDA totalMem 24563 mb [GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8] CUDA freeMem 23042 mb [GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8] Compute Capability 8.9 time=2025-02-07T09:38:48.649Z level=DEBUG source=amd_linux.go:421 msg="amdgpu driver not detected /sys/module/amdgpu" releasing cuda driver library time=2025-02-07T09:38:48.650Z level=INFO source=types.go:131 msg="inference compute" id=GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8 library=cuda variant=v12 compute=8.9 driver=12.7 name="NVIDIA GeForce RTX 4090 D" total="24.0 GiB" available="22.5 GiB" time=2025-02-07T09:39:04.454Z level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="62.6 GiB" before.free="44.5 GiB" before.free_swap="16.0 GiB" now.total="62.6 GiB" now.free="44.5 GiB" now.free_swap="16.0 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1 dlsym: cuInit - 0x7fad9dc20ec0 dlsym: cuDriverGetVersion - 0x7fad9dc20e80 dlsym: cuDeviceGetCount - 0x7fad9dc20efd dlsym: cuDeviceGet - 0x7fad9dc20ef7 dlsym: cuDeviceGetAttribute - 0x7fad9dc20de0 dlsym: cuDeviceGetUuid - 0x7fad9dc20f09 dlsym: cuDeviceGetName - 0x7fad9dc20f03 dlsym: cuCtxCreate_v3 - 0x7fad9dc20f7b dlsym: cuMemGetInfo_v2 - 0x7fad9dc2107d dlsym: cuCtxDestroy - 0x7fad9dc20f8d calling cuInit calling cuDriverGetVersion [GIN] 2025/02/07 - 09:38:55 | 200 | 31.54µs | 127.0.0.1 | HEAD "/" [GIN] 2025/02/07 - 09:38:55 | 200 | 25.668935ms | 127.0.0.1 | GET "/api/tags" [GIN] 2025/02/07 - 09:39:04 | 200 | 32.526µs | 127.0.0.1 | HEAD "/" [GIN] 2025/02/07 - 09:39:04 | 200 | 103.398398ms | 127.0.0.1 | POST "/api/show" raw version 0x2f26 CUDA driver version: 12.7 calling cuDeviceGetCount device count 1 time=2025-02-07T09:39:04.586Z level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8 name="NVIDIA GeForce RTX 4090 D" overhead="0 B" before.total="24.0 GiB" before.free="22.5 GiB" now.total="24.0 GiB" now.free="22.5 GiB" now.used="1.5 GiB" releasing cuda driver library time=2025-02-07T09:39:04.586Z level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x5599decec4c0 gpu_count=1 time=2025-02-07T09:39:04.719Z level=DEBUG source=sched.go:224 msg="loading first model" model=/root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be time=2025-02-07T09:39:04.719Z level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[22.5 GiB]" time=2025-02-07T09:39:04.719Z level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="62.6 GiB" before.free="44.5 GiB" before.free_swap="16.0 GiB" now.total="62.6 GiB" now.free="44.5 GiB" now.free_swap="16.0 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1 dlsym: cuInit - 0x7fad9dc20ec0 dlsym: cuDriverGetVersion - 0x7fad9dc20e80 dlsym: cuDeviceGetCount - 0x7fad9dc20efd dlsym: cuDeviceGet - 0x7fad9dc20ef7 dlsym: cuDeviceGetAttribute - 0x7fad9dc20de0 dlsym: cuDeviceGetUuid - 0x7fad9dc20f09 dlsym: cuDeviceGetName - 0x7fad9dc20f03 dlsym: cuCtxCreate_v3 - 0x7fad9dc20f7b dlsym: cuMemGetInfo_v2 - 0x7fad9dc2107d dlsym: cuCtxDestroy - 0x7fad9dc20f8d calling cuInit calling cuDriverGetVersion raw version 0x2f26 CUDA driver version: 12.7 calling cuDeviceGetCount device count 1 time=2025-02-07T09:39:04.841Z level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8 name="NVIDIA GeForce RTX 4090 D" overhead="0 B" before.total="24.0 GiB" before.free="22.5 GiB" now.total="24.0 GiB" now.free="22.5 GiB" now.used="1.5 GiB" releasing cuda driver library time=2025-02-07T09:39:04.841Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be gpu=GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8 parallel=1 available=24161288192 required="17.3 GiB" time=2025-02-07T09:39:04.841Z level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="62.6 GiB" before.free="44.5 GiB" before.free_swap="16.0 GiB" now.total="62.6 GiB" now.free="44.5 GiB" now.free_swap="16.0 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1 dlsym: cuInit - 0x7fad9dc20ec0 dlsym: cuDriverGetVersion - 0x7fad9dc20e80 dlsym: cuDeviceGetCount - 0x7fad9dc20efd dlsym: cuDeviceGet - 0x7fad9dc20ef7 dlsym: cuDeviceGetAttribute - 0x7fad9dc20de0 dlsym: cuDeviceGetUuid - 0x7fad9dc20f09 dlsym: cuDeviceGetName - 0x7fad9dc20f03 dlsym: cuCtxCreate_v3 - 0x7fad9dc20f7b dlsym: cuMemGetInfo_v2 - 0x7fad9dc2107d dlsym: cuCtxDestroy - 0x7fad9dc20f8d calling cuInit calling cuDriverGetVersion raw version 0x2f26 CUDA driver version: 12.7 calling cuDeviceGetCount device count 1 time=2025-02-07T09:39:04.960Z level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8 name="NVIDIA GeForce RTX 4090 D" overhead="0 B" before.total="24.0 GiB" before.free="22.5 GiB" now.total="24.0 GiB" now.free="22.5 GiB" now.used="1.5 GiB" releasing cuda driver library time=2025-02-07T09:39:04.960Z level=INFO source=server.go:104 msg="system memory" total="62.6 GiB" free="44.5 GiB" free_swap="16.0 GiB" time=2025-02-07T09:39:04.960Z level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[22.5 GiB]" time=2025-02-07T09:39:04.961Z level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="62.6 GiB" before.free="44.5 GiB" before.free_swap="16.0 GiB" now.total="62.6 GiB" now.free="44.5 GiB" now.free_swap="16.0 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1 dlsym: cuInit - 0x7fad9dc20ec0 dlsym: cuDriverGetVersion - 0x7fad9dc20e80 dlsym: cuDeviceGetCount - 0x7fad9dc20efd dlsym: cuDeviceGet - 0x7fad9dc20ef7 dlsym: cuDeviceGetAttribute - 0x7fad9dc20de0 dlsym: cuDeviceGetUuid - 0x7fad9dc20f09 dlsym: cuDeviceGetName - 0x7fad9dc20f03 dlsym: cuCtxCreate_v3 - 0x7fad9dc20f7b dlsym: cuMemGetInfo_v2 - 0x7fad9dc2107d dlsym: cuCtxDestroy - 0x7fad9dc20f8d calling cuInit calling cuDriverGetVersion raw version 0x2f26 CUDA driver version: 12.7 calling cuDeviceGetCount device count 1 time=2025-02-07T09:39:05.074Z level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8 name="NVIDIA GeForce RTX 4090 D" overhead="0 B" before.total="24.0 GiB" before.free="22.5 GiB" now.total="24.0 GiB" now.free="22.5 GiB" now.used="1.5 GiB" releasing cuda driver library time=2025-02-07T09:39:05.074Z level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[22.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="17.3 GiB" memory.required.partial="17.3 GiB" memory.required.kv="8.0 GiB" memory.required.allocations="[17.3 GiB]" memory.weights.total="11.9 GiB" memory.weights.repeating="11.5 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="4.2 GiB" memory.graph.partial="4.4 GiB" time=2025-02-07T09:39:05.074Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server time=2025-02-07T09:39:05.074Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server time=2025-02-07T09:39:05.074Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v11_avx/ollama_llama_server time=2025-02-07T09:39:05.074Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v12_avx/ollama_llama_server time=2025-02-07T09:39:05.074Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server time=2025-02-07T09:39:05.075Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server time=2025-02-07T09:39:05.075Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v11_avx/ollama_llama_server time=2025-02-07T09:39:05.075Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v12_avx/ollama_llama_server time=2025-02-07T09:39:05.075Z level=INFO source=server.go:223 msg="enabling flash attention" time=2025-02-07T09:39:05.075Z level=WARN source=server.go:231 msg="kv cache type not supported by model" type="" time=2025-02-07T09:39:05.075Z level=INFO source=server.go:376 msg="starting llama server" cmd="/usr/lib/ollama/runners/cuda_v12_avx/ollama_llama_server runner --model /root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be --ctx-size 65536 --batch-size 512 --n-gpu-layers 33 --verbose --threads 64 --flash-attn --parallel 1 --port 37207" time=2025-02-07T09:39:05.075Z level=DEBUG source=server.go:393 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin CUDA_VISIBLE_DEVICES=GPU-2ef77503-2f4c-3683-b2c4-50548cd200d8 LD_LIBRARY_PATH=/usr/lib/ollama:/usr/lib/ollama:/usr/lib/ollama/runners/cuda_v12_avx:/usr/local/nvidia/lib:/usr/local/nvidia/lib64]" time=2025-02-07T09:39:05.075Z level=INFO source=sched.go:449 msg="loaded runners" count=1 time=2025-02-07T09:39:05.075Z level=INFO source=server.go:555 msg="waiting for llama runner to start responding" time=2025-02-07T09:39:05.076Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error" time=2025-02-07T09:39:05.107Z level=INFO source=runner.go:936 msg="starting go runner" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 4090 D, compute capability 8.9, VMM: yes time=2025-02-07T09:39:05.197Z level=INFO source=runner.go:937 msg=system info="CUDA : ARCHS = 600,610,620,700,720,750,800,860,870,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=64 time=2025-02-07T09:39:05.197Z level=INFO source=.:0 msg="Server listening on 127.0.0.1:37207" llama_load_model_from_file: using device CUDA0 (NVIDIA GeForce RTX 4090 D) - 23042 MiB free time=2025-02-07T09:39:05.327Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server loading model" llama_model_loader: loaded meta data with 28 key-value pairs and 292 tensors from /root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = DeepSeek R1 Distill Llama 8B llama_model_loader: - kv 3: general.basename str = DeepSeek-R1-Distill-Llama llama_model_loader: - kv 4: general.size_label str = 8B llama_model_loader: - kv 5: llama.block_count u32 = 32 llama_model_loader: - kv 6: llama.context_length u32 = 131072 llama_model_loader: - kv 7: llama.embedding_length u32 = 4096 llama_model_loader: - kv 8: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 9: llama.attention.head_count u32 = 32 llama_model_loader: - kv 10: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 11: llama.rope.freq_base f32 = 500000.000000 llama_model_loader: - kv 12: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 13: general.file_type u32 = 15 llama_model_loader: - kv 14: llama.vocab_size u32 = 128256 llama_model_loader: - kv 15: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 16: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 17: tokenizer.ggml.pre str = llama-bpe llama_model_loader: - kv 18: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 19: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 20: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... llama_model_loader: - kv 21: tokenizer.ggml.bos_token_id u32 = 128000 llama_model_loader: - kv 22: tokenizer.ggml.eos_token_id u32 = 128001 llama_model_loader: - kv 23: tokenizer.ggml.padding_token_id u32 = 128001 llama_model_loader: - kv 24: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 25: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 26: tokenizer.chat_template str = {% if not add_generation_prompt is de... llama_model_loader: - kv 27: general.quantization_version u32 = 2 llama_model_loader: - type f32: 66 tensors llama_model_loader: - type q4_K: 193 tensors llama_model_loader: - type q6_K: 33 tensors llm_load_vocab: control token: 128254 '<|reserved_special_token_246|>' is not marked as EOG llm_load_vocab: control token: 128249 '<|reserved_special_token_241|>' is not marked as EOG llm_load_vocab: control token: 128246 '<|reserved_special_token_238|>' is not marked as EOG llm_load_vocab: control token: 128243 '<|reserved_special_token_235|>' is not marked as EOG llm_load_vocab: control token: 128242 '<|reserved_special_token_234|>' is not marked as EOG llm_load_vocab: control token: 128241 '<|reserved_special_token_233|>' is not marked as EOG llm_load_vocab: control token: 128240 '<|reserved_special_token_232|>' is not marked as EOG llm_load_vocab: control token: 128235 '<|reserved_special_token_227|>' is not marked as EOG llm_load_vocab: control token: 128231 '<|reserved_special_token_223|>' is not marked as EOG llm_load_vocab: control token: 128230 '<|reserved_special_token_222|>' is not marked as EOG llm_load_vocab: control token: 128228 '<|reserved_special_token_220|>' is not marked as EOG llm_load_vocab: control token: 128225 '<|reserved_special_token_217|>' is not marked as EOG llm_load_vocab: control token: 128218 '<|reserved_special_token_210|>' is not marked as EOG llm_load_vocab: control token: 128214 '<|reserved_special_token_206|>' is not marked as EOG llm_load_vocab: control token: 128213 '<|reserved_special_token_205|>' is not marked as EOG llm_load_vocab: control token: 128207 '<|reserved_special_token_199|>' is not marked as EOG llm_load_vocab: control token: 128206 '<|reserved_special_token_198|>' is not marked as EOG llm_load_vocab: control token: 128204 '<|reserved_special_token_196|>' is not marked as EOG llm_load_vocab: control token: 128200 '<|reserved_special_token_192|>' is not marked as EOG llm_load_vocab: control token: 128199 '<|reserved_special_token_191|>' is not marked as EOG llm_load_vocab: control token: 128198 '<|reserved_special_token_190|>' is not marked as EOG llm_load_vocab: control token: 128196 '<|reserved_special_token_188|>' is not marked as EOG llm_load_vocab: control token: 128194 '<|reserved_special_token_186|>' is not marked as EOG llm_load_vocab: control token: 128193 '<|reserved_special_token_185|>' is not marked as EOG llm_load_vocab: control token: 128188 '<|reserved_special_token_180|>' is not marked as EOG llm_load_vocab: control token: 128187 '<|reserved_special_token_179|>' is not marked as EOG llm_load_vocab: control token: 128185 '<|reserved_special_token_177|>' is not marked as EOG llm_load_vocab: control token: 128184 '<|reserved_special_token_176|>' is not marked as EOG llm_load_vocab: control token: 128180 '<|reserved_special_token_172|>' is not marked as EOG llm_load_vocab: control token: 128179 '<|reserved_special_token_171|>' is not marked as EOG llm_load_vocab: control token: 128178 '<|reserved_special_token_170|>' is not marked as EOG llm_load_vocab: control token: 128177 '<|reserved_special_token_169|>' is not marked as EOG llm_load_vocab: control token: 128176 '<|reserved_special_token_168|>' is not marked as EOG llm_load_vocab: control token: 128175 '<|reserved_special_token_167|>' is not marked as EOG llm_load_vocab: control token: 128171 '<|reserved_special_token_163|>' is not marked as EOG llm_load_vocab: control token: 128170 '<|reserved_special_token_162|>' is not marked as EOG llm_load_vocab: control token: 128169 '<|reserved_special_token_161|>' is not marked as EOG llm_load_vocab: control token: 128168 '<|reserved_special_token_160|>' is not marked as EOG llm_load_vocab: control token: 128165 '<|reserved_special_token_157|>' is not marked as EOG llm_load_vocab: control token: 128162 '<|reserved_special_token_154|>' is not marked as EOG llm_load_vocab: control token: 128158 '<|reserved_special_token_150|>' is not marked as EOG llm_load_vocab: control token: 128156 '<|reserved_special_token_148|>' is not marked as EOG llm_load_vocab: control token: 128155 '<|reserved_special_token_147|>' is not marked as EOG llm_load_vocab: control token: 128154 '<|reserved_special_token_146|>' is not marked as EOG llm_load_vocab: control token: 128151 '<|reserved_special_token_143|>' is not marked as EOG llm_load_vocab: control token: 128149 '<|reserved_special_token_141|>' is not marked as EOG llm_load_vocab: control token: 128147 '<|reserved_special_token_139|>' is not marked as EOG llm_load_vocab: control token: 128146 '<|reserved_special_token_138|>' is not marked as EOG llm_load_vocab: control token: 128144 '<|reserved_special_token_136|>' is not marked as EOG llm_load_vocab: control token: 128142 '<|reserved_special_token_134|>' is not marked as EOG llm_load_vocab: control token: 128141 '<|reserved_special_token_133|>' is not marked as EOG llm_load_vocab: control token: 128138 '<|reserved_special_token_130|>' is not marked as EOG llm_load_vocab: control token: 128136 '<|reserved_special_token_128|>' is not marked as EOG llm_load_vocab: control token: 128135 '<|reserved_special_token_127|>' is not marked as EOG llm_load_vocab: control token: 128134 '<|reserved_special_token_126|>' is not marked as EOG llm_load_vocab: control token: 128133 '<|reserved_special_token_125|>' is not marked as EOG llm_load_vocab: control token: 128131 '<|reserved_special_token_123|>' is not marked as EOG llm_load_vocab: control token: 128128 '<|reserved_special_token_120|>' is not marked as EOG llm_load_vocab: control token: 128124 '<|reserved_special_token_116|>' is not marked as EOG llm_load_vocab: control token: 128123 '<|reserved_special_token_115|>' is not marked as EOG llm_load_vocab: control token: 128122 '<|reserved_special_token_114|>' is not marked as EOG llm_load_vocab: control token: 128119 '<|reserved_special_token_111|>' is not marked as EOG llm_load_vocab: control token: 128115 '<|reserved_special_token_107|>' is not marked as EOG llm_load_vocab: control token: 128112 '<|reserved_special_token_104|>' is not marked as EOG llm_load_vocab: control token: 128110 '<|reserved_special_token_102|>' is not marked as EOG llm_load_vocab: control token: 128109 '<|reserved_special_token_101|>' is not marked as EOG llm_load_vocab: control token: 128108 '<|reserved_special_token_100|>' is not marked as EOG llm_load_vocab: control token: 128106 '<|reserved_special_token_98|>' is not marked as EOG llm_load_vocab: control token: 128103 '<|reserved_special_token_95|>' is not marked as EOG llm_load_vocab: control token: 128102 '<|reserved_special_token_94|>' is not marked as EOG llm_load_vocab: control token: 128101 '<|reserved_special_token_93|>' is not marked as EOG llm_load_vocab: control token: 128097 '<|reserved_special_token_89|>' is not marked as EOG llm_load_vocab: control token: 128091 '<|reserved_special_token_83|>' is not marked as EOG llm_load_vocab: control token: 128090 '<|reserved_special_token_82|>' is not marked as EOG llm_load_vocab: control token: 128089 '<|reserved_special_token_81|>' is not marked as EOG llm_load_vocab: control token: 128087 '<|reserved_special_token_79|>' is not marked as EOG llm_load_vocab: control token: 128085 '<|reserved_special_token_77|>' is not marked as EOG llm_load_vocab: control token: 128081 '<|reserved_special_token_73|>' is not marked as EOG llm_load_vocab: control token: 128078 '<|reserved_special_token_70|>' is not marked as EOG llm_load_vocab: control token: 128076 '<|reserved_special_token_68|>' is not marked as EOG llm_load_vocab: control token: 128075 '<|reserved_special_token_67|>' is not marked as EOG llm_load_vocab: control token: 128073 '<|reserved_special_token_65|>' is not marked as EOG llm_load_vocab: control token: 128068 '<|reserved_special_token_60|>' is not marked as EOG llm_load_vocab: control token: 128067 '<|reserved_special_token_59|>' is not marked as EOG llm_load_vocab: control token: 128065 '<|reserved_special_token_57|>' is not marked as EOG llm_load_vocab: control token: 128063 '<|reserved_special_token_55|>' is not marked as EOG llm_load_vocab: control token: 128062 '<|reserved_special_token_54|>' is not marked as EOG llm_load_vocab: control token: 128060 '<|reserved_special_token_52|>' is not marked as EOG llm_load_vocab: control token: 128059 '<|reserved_special_token_51|>' is not marked as EOG llm_load_vocab: control token: 128057 '<|reserved_special_token_49|>' is not marked as EOG llm_load_vocab: control token: 128054 '<|reserved_special_token_46|>' is not marked as EOG llm_load_vocab: control token: 128046 '<|reserved_special_token_38|>' is not marked as EOG llm_load_vocab: control token: 128045 '<|reserved_special_token_37|>' is not marked as EOG llm_load_vocab: control token: 128044 '<|reserved_special_token_36|>' is not marked as EOG llm_load_vocab: control token: 128043 '<|reserved_special_token_35|>' is not marked as EOG llm_load_vocab: control token: 128038 '<|reserved_special_token_30|>' is not marked as EOG llm_load_vocab: control token: 128036 '<|reserved_special_token_28|>' is not marked as EOG llm_load_vocab: control token: 128035 '<|reserved_special_token_27|>' is not marked as EOG llm_load_vocab: control token: 128032 '<|reserved_special_token_24|>' is not marked as EOG llm_load_vocab: control token: 128028 '<|reserved_special_token_20|>' is not marked as EOG llm_load_vocab: control token: 128027 '<|reserved_special_token_19|>' is not marked as EOG llm_load_vocab: control token: 128024 '<|reserved_special_token_16|>' is not marked as EOG llm_load_vocab: control token: 128023 '<|reserved_special_token_15|>' is not marked as EOG llm_load_vocab: control token: 128022 '<|reserved_special_token_14|>' is not marked as EOG llm_load_vocab: control token: 128021 '<|reserved_special_token_13|>' is not marked as EOG llm_load_vocab: control token: 128018 '<|reserved_special_token_10|>' is not marked as EOG llm_load_vocab: control token: 128016 '<|reserved_special_token_8|>' is not marked as EOG llm_load_vocab: control token: 128012 '<｜Assistant｜>' is not marked as EOG llm_load_vocab: control token: 128011 '<｜User｜>' is not marked as EOG llm_load_vocab: control token: 128005 '<|reserved_special_token_2|>' is not marked as EOG llm_load_vocab: control token: 128004 '<|finetune_right_pad_id|>' is not marked as EOG llm_load_vocab: control token: 128002 '<|reserved_special_token_0|>' is not marked as EOG llm_load_vocab: control token: 128252 '<|reserved_special_token_244|>' is not marked as EOG llm_load_vocab: control token: 128190 '<|reserved_special_token_182|>' is not marked as EOG llm_load_vocab: control token: 128183 '<|reserved_special_token_175|>' is not marked as EOG llm_load_vocab: control token: 128137 '<|reserved_special_token_129|>' is not marked as EOG llm_load_vocab: control token: 128182 '<|reserved_special_token_174|>' is not marked as EOG llm_load_vocab: control token: 128040 '<|reserved_special_token_32|>' is not marked as EOG llm_load_vocab: control token: 128048 '<|reserved_special_token_40|>' is not marked as EOG llm_load_vocab: control token: 128092 '<|reserved_special_token_84|>' is not marked as EOG llm_load_vocab: control token: 128215 '<|reserved_special_token_207|>' is not marked as EOG llm_load_vocab: control token: 128107 '<|reserved_special_token_99|>' is not marked as EOG llm_load_vocab: control token: 128208 '<|reserved_special_token_200|>' is not marked as EOG llm_load_vocab: control token: 128145 '<|reserved_special_token_137|>' is not marked as EOG llm_load_vocab: control token: 128031 '<|reserved_special_token_23|>' is not marked as EOG llm_load_vocab: control token: 128129 '<|reserved_special_token_121|>' is not marked as EOG llm_load_vocab: control token: 128201 '<|reserved_special_token_193|>' is not marked as EOG llm_load_vocab: control token: 128074 '<|reserved_special_token_66|>' is not marked as EOG llm_load_vocab: control token: 128095 '<|reserved_special_token_87|>' is not marked as EOG llm_load_vocab: control token: 128186 '<|reserved_special_token_178|>' is not marked as EOG llm_load_vocab: control token: 128143 '<|reserved_special_token_135|>' is not marked as EOG llm_load_vocab: control token: 128229 '<|reserved_special_token_221|>' is not marked as EOG llm_load_vocab: control token: 128007 '<|end_header_id|>' is not marked as EOG llm_load_vocab: control token: 128055 '<|reserved_special_token_47|>' is not marked as EOG llm_load_vocab: control token: 128056 '<|reserved_special_token_48|>' is not marked as EOG llm_load_vocab: control token: 128061 '<|reserved_special_token_53|>' is not marked as EOG llm_load_vocab: control token: 128153 '<|reserved_special_token_145|>' is not marked as EOG llm_load_vocab: control token: 128152 '<|reserved_special_token_144|>' is not marked as EOG llm_load_vocab: control token: 128212 '<|reserved_special_token_204|>' is not marked as EOG llm_load_vocab: control token: 128172 '<|reserved_special_token_164|>' is not marked as EOG llm_load_vocab: control token: 128160 '<|reserved_special_token_152|>' is not marked as EOG llm_load_vocab: control token: 128041 '<|reserved_special_token_33|>' is not marked as EOG llm_load_vocab: control token: 128181 '<|reserved_special_token_173|>' is not marked as EOG llm_load_vocab: control token: 128094 '<|reserved_special_token_86|>' is not marked as EOG llm_load_vocab: control token: 128118 '<|reserved_special_token_110|>' is not marked as EOG llm_load_vocab: control token: 128236 '<|reserved_special_token_228|>' is not marked as EOG llm_load_vocab: control token: 128148 '<|reserved_special_token_140|>' is not marked as EOG llm_load_vocab: control token: 128042 '<|reserved_special_token_34|>' is not marked as EOG llm_load_vocab: control token: 128139 '<|reserved_special_token_131|>' is not marked as EOG llm_load_vocab: control token: 128173 '<|reserved_special_token_165|>' is not marked as EOG llm_load_vocab: control token: 128239 '<|reserved_special_token_231|>' is not marked as EOG llm_load_vocab: control token: 128157 '<|reserved_special_token_149|>' is not marked as EOG llm_load_vocab: control token: 128052 '<|reserved_special_token_44|>' is not marked as EOG llm_load_vocab: control token: 128026 '<|reserved_special_token_18|>' is not marked as EOG llm_load_vocab: control token: 128003 '<|reserved_special_token_1|>' is not marked as EOG llm_load_vocab: control token: 128019 '<|reserved_special_token_11|>' is not marked as EOG llm_load_vocab: control token: 128116 '<|reserved_special_token_108|>' is not marked as EOG llm_load_vocab: control token: 128161 '<|reserved_special_token_153|>' is not marked as EOG llm_load_vocab: control token: 128000 '<｜begin▁of▁sentence｜>' is not marked as EOG llm_load_vocab: control token: 128226 '<|reserved_special_token_218|>' is not marked as EOG llm_load_vocab: control token: 128159 '<|reserved_special_token_151|>' is not marked as EOG llm_load_vocab: control token: 128088 '<|reserved_special_token_80|>' is not marked as EOG llm_load_vocab: control token: 128163 '<|reserved_special_token_155|>' is not marked as EOG llm_load_vocab: control token: 128113 '<|reserved_special_token_105|>' is not marked as EOG llm_load_vocab: control token: 128250 '<|reserved_special_token_242|>' is not marked as EOG llm_load_vocab: control token: 128125 '<|reserved_special_token_117|>' is not marked as EOG llm_load_vocab: control token: 128053 '<|reserved_special_token_45|>' is not marked as EOG llm_load_vocab: control token: 128224 '<|reserved_special_token_216|>' is not marked as EOG llm_load_vocab: control token: 128247 '<|reserved_special_token_239|>' is not marked as EOG llm_load_vocab: control token: 128251 '<|reserved_special_token_243|>' is not marked as EOG llm_load_vocab: control token: 128216 '<|reserved_special_token_208|>' is not marked as EOG llm_load_vocab: control token: 128006 '<|start_header_id|>' is not marked as EOG llm_load_vocab: control token: 128211 '<|reserved_special_token_203|>' is not marked as EOG llm_load_vocab: control token: 128077 '<|reserved_special_token_69|>' is not marked as EOG llm_load_vocab: control token: 128237 '<|reserved_special_token_229|>' is not marked as EOG llm_load_vocab: control token: 128086 '<|reserved_special_token_78|>' is not marked as EOG llm_load_vocab: control token: 128001 '<｜end▁of▁sentence｜>' is not marked as EOG llm_load_vocab: control token: 128227 '<|reserved_special_token_219|>' is not marked as EOG llm_load_vocab: control token: 128058 '<|reserved_special_token_50|>' is not marked as EOG llm_load_vocab: control token: 128100 '<|reserved_special_token_92|>' is not marked as EOG llm_load_vocab: control token: 128209 '<|reserved_special_token_201|>' is not marked as EOG llm_load_vocab: control token: 128084 '<|reserved_special_token_76|>' is not marked as EOG llm_load_vocab: control token: 128071 '<|reserved_special_token_63|>' is not marked as EOG llm_load_vocab: control token: 128070 '<|reserved_special_token_62|>' is not marked as EOG llm_load_vocab: control token: 128049 '<|reserved_special_token_41|>' is not marked as EOG llm_load_vocab: control token: 128197 '<|reserved_special_token_189|>' is not marked as EOG llm_load_vocab: control token: 128072 '<|reserved_special_token_64|>' is not marked as EOG llm_load_vocab: control token: 128223 '<|reserved_special_token_215|>' is not marked as EOG llm_load_vocab: control token: 128217 '<|reserved_special_token_209|>' is not marked as EOG llm_load_vocab: control token: 128111 '<|reserved_special_token_103|>' is not marked as EOG llm_load_vocab: control token: 128203 '<|reserved_special_token_195|>' is not marked as EOG llm_load_vocab: control token: 128051 '<|reserved_special_token_43|>' is not marked as EOG llm_load_vocab: control token: 128030 '<|reserved_special_token_22|>' is not marked as EOG llm_load_vocab: control token: 128117 '<|reserved_special_token_109|>' is not marked as EOG llm_load_vocab: control token: 128010 '<|python_tag|>' is not marked as EOG llm_load_vocab: control token: 128238 '<|reserved_special_token_230|>' is not marked as EOG llm_load_vocab: control token: 128255 '<|reserved_special_token_247|>' is not marked as EOG llm_load_vocab: control token: 128202 '<|reserved_special_token_194|>' is not marked as EOG llm_load_vocab: control token: 128132 '<|reserved_special_token_124|>' is not marked as EOG llm_load_vocab: control token: 128248 '<|reserved_special_token_240|>' is not marked as EOG llm_load_vocab: control token: 128167 '<|reserved_special_token_159|>' is not marked as EOG llm_load_vocab: control token: 128127 '<|reserved_special_token_119|>' is not marked as EOG llm_load_vocab: control token: 128105 '<|reserved_special_token_97|>' is not marked as EOG llm_load_vocab: control token: 128039 '<|reserved_special_token_31|>' is not marked as EOG llm_load_vocab: control token: 128232 '<|reserved_special_token_224|>' is not marked as EOG llm_load_vocab: control token: 128166 '<|reserved_special_token_158|>' is not marked as EOG llm_load_vocab: control token: 128130 '<|reserved_special_token_122|>' is not marked as EOG llm_load_vocab: control token: 128114 '<|reserved_special_token_106|>' is not marked as EOG llm_load_vocab: control token: 128234 '<|reserved_special_token_226|>' is not marked as EOG llm_load_vocab: control token: 128191 '<|reserved_special_token_183|>' is not marked as EOG llm_load_vocab: control token: 128064 '<|reserved_special_token_56|>' is not marked as EOG llm_load_vocab: control token: 128140 '<|reserved_special_token_132|>' is not marked as EOG llm_load_vocab: control token: 128096 '<|reserved_special_token_88|>' is not marked as EOG llm_load_vocab: control token: 128098 '<|reserved_special_token_90|>' is not marked as EOG llm_load_vocab: control token: 128192 '<|reserved_special_token_184|>' is not marked as EOG llm_load_vocab: control token: 128093 '<|reserved_special_token_85|>' is not marked as EOG llm_load_vocab: control token: 128150 '<|reserved_special_token_142|>' is not marked as EOG llm_load_vocab: control token: 128222 '<|reserved_special_token_214|>' is not marked as EOG llm_load_vocab: control token: 128233 '<|reserved_special_token_225|>' is not marked as EOG llm_load_vocab: control token: 128220 '<|reserved_special_token_212|>' is not marked as EOG llm_load_vocab: control token: 128034 '<|reserved_special_token_26|>' is not marked as EOG llm_load_vocab: control token: 128033 '<|reserved_special_token_25|>' is not marked as EOG llm_load_vocab: control token: 128253 '<|reserved_special_token_245|>' is not marked as EOG llm_load_vocab: control token: 128195 '<|reserved_special_token_187|>' is not marked as EOG llm_load_vocab: control token: 128099 '<|reserved_special_token_91|>' is not marked as EOG llm_load_vocab: control token: 128189 '<|reserved_special_token_181|>' is not marked as EOG llm_load_vocab: control token: 128210 '<|reserved_special_token_202|>' is not marked as EOG llm_load_vocab: control token: 128174 '<|reserved_special_token_166|>' is not marked as EOG llm_load_vocab: control token: 128083 '<|reserved_special_token_75|>' is not marked as EOG llm_load_vocab: control token: 128080 '<|reserved_special_token_72|>' is not marked as EOG llm_load_vocab: control token: 128104 '<|reserved_special_token_96|>' is not marked as EOG llm_load_vocab: control token: 128082 '<|reserved_special_token_74|>' is not marked as EOG llm_load_vocab: control token: 128219 '<|reserved_special_token_211|>' is not marked as EOG llm_load_vocab: control token: 128017 '<|reserved_special_token_9|>' is not marked as EOG llm_load_vocab: control token: 128050 '<|reserved_special_token_42|>' is not marked as EOG llm_load_vocab: control token: 128205 '<|reserved_special_token_197|>' is not marked as EOG llm_load_vocab: control token: 128047 '<|reserved_special_token_39|>' is not marked as EOG llm_load_vocab: control token: 128164 '<|reserved_special_token_156|>' is not marked as EOG llm_load_vocab: control token: 128020 '<|reserved_special_token_12|>' is not marked as EOG llm_load_vocab: control token: 128069 '<|reserved_special_token_61|>' is not marked as EOG llm_load_vocab: control token: 128245 '<|reserved_special_token_237|>' is not marked as EOG llm_load_vocab: control token: 128121 '<|reserved_special_token_113|>' is not marked as EOG llm_load_vocab: control token: 128079 '<|reserved_special_token_71|>' is not marked as EOG llm_load_vocab: control token: 128037 '<|reserved_special_token_29|>' is not marked as EOG llm_load_vocab: control token: 128244 '<|reserved_special_token_236|>' is not marked as EOG llm_load_vocab: control token: 128029 '<|reserved_special_token_21|>' is not marked as EOG llm_load_vocab: control token: 128221 '<|reserved_special_token_213|>' is not marked as EOG llm_load_vocab: control token: 128066 '<|reserved_special_token_58|>' is not marked as EOG llm_load_vocab: control token: 128120 '<|reserved_special_token_112|>' is not marked as EOG llm_load_vocab: control token: 128015 '<｜▁pad▁｜>' is not marked as EOG llm_load_vocab: control token: 128025 '<|reserved_special_token_17|>' is not marked as EOG llm_load_vocab: control token: 128126 '<|reserved_special_token_118|>' is not marked as EOG llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect llm_load_vocab: special tokens cache size = 256 llm_load_vocab: token to piece cache size = 0.7999 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 128256 llm_load_print_meta: n_merges = 280147 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 131072 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 500000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 131072 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = 8B llm_load_print_meta: model ftype = Q4_K - Medium llm_load_print_meta: model params = 8.03 B llm_load_print_meta: model size = 4.58 GiB (4.89 BPW) llm_load_print_meta: general.name = DeepSeek R1 Distill Llama 8B llm_load_print_meta: BOS token = 128000 '<｜begin▁of▁sentence｜>' llm_load_print_meta: EOS token = 128001 '<｜end▁of▁sentence｜>' llm_load_print_meta: EOT token = 128009 '<|eot_id|>' llm_load_print_meta: EOM token = 128008 '<|eom_id|>' llm_load_print_meta: PAD token = 128001 '<｜end▁of▁sentence｜>' llm_load_print_meta: LF token = 128 'Ä' llm_load_print_meta: EOG token = 128001 '<｜end▁of▁sentence｜>' llm_load_print_meta: EOG token = 128008 '<|eom_id|>' llm_load_print_meta: EOG token = 128009 '<|eot_id|>' llm_load_print_meta: max token length = 256 llm_load_tensors: tensor 'token_embd.weight' (q4_K) (and 0 others) cannot be used with preferred buffer type CPU_AARCH64, using CPU instead

GiteaMirror closed this issue 2026-04-28 22:56:42 -05:00

GiteaMirror commented 2026-04-28 22:56:49 -05:00

Author

Owner

Copy Link

@jessegross commented on GitHub (Feb 7, 2025):

Are you seeing a problem other than this warning?

<!-- gh-comment-id:2643605650 --> @jessegross commented on GitHub (Feb 7, 2025): Are you seeing a problem other than this warning?

GiteaMirror commented 2026-04-28 22:56:52 -05:00

Author

Owner

Copy Link

@LwengGitHub commented on GitHub (Feb 8, 2025):

This is all loading log info. Looks normal, and it works fine for subsequent use. But This warning (using CPU instead).
I've observed that when the context length is adjusted to be larger, the portion using the CPU increases. Ex. num_ctx 131072.

----------------------------------Log-Info---------------------------------
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.

• using env: export GIN_MODE=release
• using code: gin.SetMode(gin.ReleaseMode)

[GIN-debug] POST /api/pull --> github.com/ollama/ollama/server.(Server).PullHandler-fm (5 handlers)
[GIN-debug] POST /api/generate --> github.com/ollama/ollama/server.(Server).GenerateHandler-fm (5 handlers)
[GIN-debug] POST /api/chat --> github.com/ollama/ollama/server.(Server).ChatHandler-fm (5 handlers)
[GIN-debug] POST /api/embed --> github.com/ollama/ollama/server.(Server).EmbedHandler-fm (5 handlers)
[GIN-debug] POST /api/embeddings --> github.com/ollama/ollama/server.(Server).EmbeddingsHandler-fm (5 handlers)
[GIN-debug] POST /api/create --> github.com/ollama/ollama/server.(Server).CreateHandler-fm (5 handlers)
[GIN-debug] POST /api/push --> github.com/ollama/ollama/server.(Server).PushHandler-fm (5 handlers)
[GIN-debug] POST /api/copy --> github.com/ollama/ollama/server.(Server).CopyHandler-fm (5 handlers)
[GIN-debug] DELETE /api/delete --> github.com/ollama/ollama/server.(Server).DeleteHandler-fm (5 handlers)
[GIN-debug] POST /api/show --> github.com/ollama/ollama/server.(Server).ShowHandler-fm (5 handlers)
[GIN-debug] POST /api/blobs/:digest --> github.com/ollama/ollama/server.(Server).CreateBlobHandler-fm (5 handlers)
[GIN-debug] HEAD /api/blobs/:digest --> github.com/ollama/ollama/server.(Server).HeadBlobHandler-fm (5 handlers)
[GIN-debug] GET /api/ps --> github.com/ollama/ollama/server.(Server).PsHandler-fm (5 handlers)
[GIN-debug] POST /v1/chat/completions --> github.com/ollama/ollama/server.(Server).ChatHandler-fm (6 handlers)
[GIN-debug] POST /v1/completions --> github.com/ollama/ollama/server.(Server).GenerateHandler-fm (6 handlers)
[GIN-debug] POST /v1/embeddings --> github.com/ollama/ollama/server.(Server).EmbedHandler-fm (6 handlers)
[GIN-debug] GET /v1/models --> github.com/ollama/ollama/server.(Server).ListHandler-fm (6 handlers)
[GIN-debug] GET /v1/models/:model --> github.com/ollama/ollama/server.(Server).ShowHandler-fm (6 handlers)
[GIN-debug] GET / --> github.com/ollama/ollama/server.(Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] GET /api/tags --> github.com/ollama/ollama/server.(Server).ListHandler-fm (5 handlers)
[GIN-debug] GET /api/version --> github.com/ollama/ollama/server.(Server).GenerateRoutes.func2 (5 handlers)
[GIN-debug] HEAD / --> github.com/ollama/ollama/server.(Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] HEAD /api/tags --> github.com/ollama/ollama/server.(Server).ListHandler-fm (5 handlers)
[GIN-debug] HEAD /api/version --> github.com/ollama/ollama/server.(Server).GenerateRoutes.func2 (5 handlers)
2025/02/08 01:09:11 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES:0 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:30m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost: https://localhost: http://127.0.0.1 https://127.0.0.1 http://127.0.0.1: https://127.0.0.1: http://0.0.0.0 https://0.0.0.0 http://0.0.0.0: https://0.0.0.0: app:// file:// tauri:// vscode-webview://] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2025-02-08T01:09:11.816Z level=INFO source=images.go:432 msg="total blobs: 5"
time=2025-02-08T01:09:11.834Z level=INFO source=images.go:439 msg="total unused blobs removed: 0"
time=2025-02-08T01:09:11.854Z level=INFO source=routes.go:1238 msg="Listening on [::]:11434 (version 0.5.7-0-ga420a45-dirty)"
time=2025-02-08T01:09:11.855Z level=DEBUG source=common.go:80 msg="runners located" dir=/usr/lib/ollama/runners
time=2025-02-08T01:09:11.856Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server
time=2025-02-08T01:09:11.856Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server
time=2025-02-08T01:09:11.856Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v11_avx/ollama_llama_server
time=2025-02-08T01:09:11.856Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v12_avx/ollama_llama_server
time=2025-02-08T01:09:11.856Z level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cpu_avx2 cuda_v11_avx cuda_v12_avx cpu cpu_avx]"
time=2025-02-08T01:09:11.856Z level=DEBUG source=routes.go:1268 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2025-02-08T01:09:11.856Z level=DEBUG source=sched.go:105 msg="starting llm scheduler"
time=2025-02-08T01:09:11.856Z level=INFO source=gpu.go:226 msg="looking for compatible GPUs"
time=2025-02-08T01:09:11.863Z level=DEBUG source=gpu.go:99 msg="searching for GPU discovery libraries for NVIDIA"
time=2025-02-08T01:09:11.863Z level=DEBUG source=gpu.go:517 msg="Searching for GPU library" name=libcuda.so
time=2025-02-08T01:09:11.863Z level=DEBUG source=gpu.go:543 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so /usr/lib/ollama/libcuda.so /usr/local/nvidia/lib/libcuda.so /usr/local/nvidia/lib64/libcuda.so /usr/local/cuda/targets//lib/libcuda.so /usr/lib/-linux-gnu/nvidia/current/libcuda.so /usr/lib/-linux-gnu/libcuda.so /usr/lib/wsl/lib/libcuda.so /usr/lib/wsl/drivers//libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2025-02-08T01:09:11.866Z level=DEBUG source=gpu.go:576 msg="discovered GPU libraries" paths="[/usr/lib/x86_64-linux-gnu/libcuda.so.1 /usr/lib/wsl/drivers/nv_dispi.inf_amd64_adf5a840df867035/libcuda.so.1.1]"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1
dlsym: cuInit - 0x7f844c420ec0
dlsym: cuDriverGetVersion - 0x7f844c420e80
dlsym: cuDeviceGetCount - 0x7f844c420efd
dlsym: cuDeviceGet - 0x7f844c420ef7
dlsym: cuDeviceGetAttribute - 0x7f844c420de0
dlsym: cuDeviceGetUuid - 0x7f844c420f09
dlsym: cuDeviceGetName - 0x7f844c420f03
dlsym: cuCtxCreate_v3 - 0x7f844c420f7b
dlsym: cuMemGetInfo_v2 - 0x7f844c42107d
dlsym: cuCtxDestroy - 0x7f844c420f8d
calling cuInit
calling cuDriverGetVersion
raw version 0x2f26
CUDA driver version: 12.7
calling cuDeviceGetCount
device count 1
time=2025-02-08T01:09:11.976Z level=DEBUG source=gpu.go:134 msg="detected GPUs" count=1 library=/usr/lib/x86_64-linux-gnu/libcuda.so.1
[GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57] CUDA totalMem 24563 mb
[GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57] CUDA freeMem 23042 mb
[GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57] Compute Capability 8.9
time=2025-02-08T01:09:12.136Z level=DEBUG source=amd_linux.go:421 msg="amdgpu driver not detected /sys/module/amdgpu"
releasing cuda driver library
time=2025-02-08T01:09:12.136Z level=INFO source=types.go:131 msg="inference compute" id=GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57 library=cuda variant=v12 compute=8.9 driver=12.7 name="NVIDIA GeForce RTX 4090 D" total="24.0 GiB" available="22.5 GiB"
time=2025-02-08T01:09:31.695Z level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="62.6 GiB" before.free="43.0 GiB" before.free_swap="16.0 GiB" now.total="62.6 GiB" now.free="43.0 GiB" now.free_swap="16.0 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1
dlsym: cuInit - 0x7f844c420ec0
dlsym: cuDriverGetVersion - 0x7f844c420e80
dlsym: cuDeviceGetCount - 0x7f844c420efd
dlsym: cuDeviceGet - 0x7f844c420ef7
dlsym: cuDeviceGetAttribute - 0x7f844c420de0
dlsym: cuDeviceGetUuid - 0x7f844c420f09
dlsym: cuDeviceGetName - 0x7f844c420f03
dlsym: cuCtxCreate_v3 - 0x7f844c420f7b
dlsym: cuMemGetInfo_v2 - 0x7f844c42107d
dlsym: cuCtxDestroy - 0x7f844c420f8d
calling cuInit
calling cuDriverGetVersion
raw version 0x2f26
CUDA driver version: 12.7
calling cuDeviceGetCount
device count 1
time=2025-02-08T01:09:31.819Z level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57 name="NVIDIA GeForce RTX 4090 D" overhead="0 B" before.total="24.0 GiB" before.free="22.5 GiB" now.total="24.0 GiB" now.free="22.5 GiB" now.used="1.5 GiB"
releasing cuda driver library
time=2025-02-08T01:09:31.820Z level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x556960a934c0 gpu_count=1
time=2025-02-08T01:09:31.942Z level=DEBUG source=sched.go:224 msg="loading first model" model=/root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be
time=2025-02-08T01:09:31.942Z level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[22.5 GiB]"
time=2025-02-08T01:09:31.943Z level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="62.6 GiB" before.free="43.0 GiB" before.free_swap="16.0 GiB" now.total="62.6 GiB" now.free="43.0 GiB" now.free_swap="16.0 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1
dlsym: cuInit - 0x7f844c420ec0
dlsym: cuDriverGetVersion - 0x7f844c420e80
[GIN] 2025/02/08 - 01:09:22 | 200 | 51.803µs | 127.0.0.1 | HEAD "/"
[GIN] 2025/02/08 - 01:09:22 | 200 | 22.569215ms | 127.0.0.1 | GET "/api/tags"
[GIN] 2025/02/08 - 01:09:31 | 200 | 27.397µs | 127.0.0.1 | HEAD "/"
[GIN] 2025/02/08 - 01:09:31 | 200 | 108.571714ms | 127.0.0.1 | POST "/api/show"
dlsym: cuDeviceGetCount - 0x7f844c420efd
dlsym: cuDeviceGet - 0x7f844c420ef7
dlsym: cuDeviceGetAttribute - 0x7f844c420de0
dlsym: cuDeviceGetUuid - 0x7f844c420f09
dlsym: cuDeviceGetName - 0x7f844c420f03
dlsym: cuCtxCreate_v3 - 0x7f844c420f7b
dlsym: cuMemGetInfo_v2 - 0x7f844c42107d
dlsym: cuCtxDestroy - 0x7f844c420f8d
calling cuInit
calling cuDriverGetVersion
raw version 0x2f26
CUDA driver version: 12.7
calling cuDeviceGetCount
device count 1
time=2025-02-08T01:09:32.086Z level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57 name="NVIDIA GeForce RTX 4090 D" overhead="0 B" before.total="24.0 GiB" before.free="22.5 GiB" now.total="24.0 GiB" now.free="22.5 GiB" now.used="1.5 GiB"
releasing cuda driver library
time=2025-02-08T01:09:32.086Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be gpu=GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57 parallel=1 available=24161288192 required="17.3 GiB"
time=2025-02-08T01:09:32.086Z level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="62.6 GiB" before.free="43.0 GiB" before.free_swap="16.0 GiB" now.total="62.6 GiB" now.free="43.0 GiB" now.free_swap="16.0 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1
dlsym: cuInit - 0x7f844c420ec0
dlsym: cuDriverGetVersion - 0x7f844c420e80
dlsym: cuDeviceGetCount - 0x7f844c420efd
dlsym: cuDeviceGet - 0x7f844c420ef7
dlsym: cuDeviceGetAttribute - 0x7f844c420de0
dlsym: cuDeviceGetUuid - 0x7f844c420f09
dlsym: cuDeviceGetName - 0x7f844c420f03
dlsym: cuCtxCreate_v3 - 0x7f844c420f7b
dlsym: cuMemGetInfo_v2 - 0x7f844c42107d
dlsym: cuCtxDestroy - 0x7f844c420f8d
calling cuInit
calling cuDriverGetVersion
raw version 0x2f26
CUDA driver version: 12.7
calling cuDeviceGetCount
device count 1
time=2025-02-08T01:09:32.202Z level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57 name="NVIDIA GeForce RTX 4090 D" overhead="0 B" before.total="24.0 GiB" before.free="22.5 GiB" now.total="24.0 GiB" now.free="22.5 GiB" now.used="1.5 GiB"
releasing cuda driver library
time=2025-02-08T01:09:32.202Z level=INFO source=server.go:104 msg="system memory" total="62.6 GiB" free="43.0 GiB" free_swap="16.0 GiB"
time=2025-02-08T01:09:32.202Z level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[22.5 GiB]"
time=2025-02-08T01:09:32.203Z level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="62.6 GiB" before.free="43.0 GiB" before.free_swap="16.0 GiB" now.total="62.6 GiB" now.free="43.0 GiB" now.free_swap="16.0 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1
dlsym: cuInit - 0x7f844c420ec0
dlsym: cuDriverGetVersion - 0x7f844c420e80
dlsym: cuDeviceGetCount - 0x7f844c420efd
dlsym: cuDeviceGet - 0x7f844c420ef7
dlsym: cuDeviceGetAttribute - 0x7f844c420de0
dlsym: cuDeviceGetUuid - 0x7f844c420f09
dlsym: cuDeviceGetName - 0x7f844c420f03
dlsym: cuCtxCreate_v3 - 0x7f844c420f7b
dlsym: cuMemGetInfo_v2 - 0x7f844c42107d
dlsym: cuCtxDestroy - 0x7f844c420f8d
calling cuInit
calling cuDriverGetVersion
raw version 0x2f26
CUDA driver version: 12.7
calling cuDeviceGetCount
device count 1
time=2025-02-08T01:09:32.318Z level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57 name="NVIDIA GeForce RTX 4090 D" overhead="0 B" before.total="24.0 GiB" before.free="22.5 GiB" now.total="24.0 GiB" now.free="22.5 GiB" now.used="1.5 GiB"
releasing cuda driver library
time=2025-02-08T01:09:32.320Z level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[22.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="17.3 GiB" memory.required.partial="17.3 GiB" memory.required.kv="8.0 GiB" memory.required.allocations="[17.3 GiB]" memory.weights.total="11.9 GiB" memory.weights.repeating="11.5 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="4.2 GiB" memory.graph.partial="4.4 GiB"
time=2025-02-08T01:09:32.320Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server
time=2025-02-08T01:09:32.320Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server
time=2025-02-08T01:09:32.320Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v11_avx/ollama_llama_server
time=2025-02-08T01:09:32.320Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v12_avx/ollama_llama_server
time=2025-02-08T01:09:32.320Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server
time=2025-02-08T01:09:32.320Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server
time=2025-02-08T01:09:32.320Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v11_avx/ollama_llama_server
time=2025-02-08T01:09:32.320Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v12_avx/ollama_llama_server
time=2025-02-08T01:09:32.320Z level=INFO source=server.go:223 msg="enabling flash attention"
time=2025-02-08T01:09:32.320Z level=WARN source=server.go:231 msg="kv cache type not supported by model" type=""
time=2025-02-08T01:09:32.321Z level=INFO source=server.go:376 msg="starting llama server" cmd="/usr/lib/ollama/runners/cuda_v12_avx/ollama_llama_server runner --model /root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be --ctx-size 65536 --batch-size 512 --n-gpu-layers 33 --verbose --threads 64 --flash-attn --parallel 1 --port 41863"
time=2025-02-08T01:09:32.321Z level=DEBUG source=server.go:393 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin CUDA_VISIBLE_DEVICES=GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57 LD_LIBRARY_PATH=/usr/lib/ollama:/usr/lib/ollama:/usr/lib/ollama/runners/cuda_v12_avx:/usr/local/nvidia/lib:/usr/local/nvidia/lib64]"
time=2025-02-08T01:09:32.321Z level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-02-08T01:09:32.321Z level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
time=2025-02-08T01:09:32.322Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
time=2025-02-08T01:09:32.392Z level=INFO source=runner.go:936 msg="starting go runner"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 4090 D, compute capability 8.9, VMM: yes
time=2025-02-08T01:09:32.479Z level=INFO source=runner.go:937 msg=system info="CUDA : ARCHS = 600,610,620,700,720,750,800,860,870,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=64
time=2025-02-08T01:09:32.479Z level=INFO source=.:0 msg="Server listening on 127.0.0.1:41863"
time=2025-02-08T01:09:32.573Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server loading model"
llama_load_model_from_file: using device CUDA0 (NVIDIA GeForce RTX 4090 D) - 23042 MiB free
llama_model_loader: loaded meta data with 28 key-value pairs and 292 tensors from /root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = DeepSeek R1 Distill Llama 8B
llama_model_loader: - kv 3: general.basename str = DeepSeek-R1-Distill-Llama
llama_model_loader: - kv 4: general.size_label str = 8B
llama_model_loader: - kv 5: llama.block_count u32 = 32
llama_model_loader: - kv 6: llama.context_length u32 = 131072
llama_model_loader: - kv 7: llama.embedding_length u32 = 4096
llama_model_loader: - kv 8: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 9: llama.attention.head_count u32 = 32
llama_model_loader: - kv 10: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 11: llama.rope.freq_base f32 = 500000.000000
llama_model_loader: - kv 12: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 13: general.file_type u32 = 15
llama_model_loader: - kv 14: llama.vocab_size u32 = 128256
llama_model_loader: - kv 15: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 16: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 17: tokenizer.ggml.pre str = llama-bpe
llama_model_loader: - kv 18: tokenizer.ggml.tokens arr[str,128256] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 19: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 20: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv 21: tokenizer.ggml.bos_token_id u32 = 128000
llama_model_loader: - kv 22: tokenizer.ggml.eos_token_id u32 = 128001
llama_model_loader: - kv 23: tokenizer.ggml.padding_token_id u32 = 128001
llama_model_loader: - kv 24: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 25: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 26: tokenizer.chat_template str = {% if not add_generation_prompt is de...
llama_model_loader: - kv 27: general.quantization_version u32 = 2
llama_model_loader: - type f32: 66 tensors
llama_model_loader: - type q4_K: 193 tensors
llama_model_loader: - type q6_K: 33 tensors
llm_load_vocab: control token: 128254 '<|reserved_special_token_246|>' is not marked as EOG
llm_load_vocab: control token: 128249 '<|reserved_special_token_241|>' is not marked as EOG
llm_load_vocab: control token: 128246 '<|reserved_special_token_238|>' is not marked as EOG
llm_load_vocab: control token: 128243 '<|reserved_special_token_235|>' is not marked as EOG
llm_load_vocab: control token: 128242 '<|reserved_special_token_234|>' is not marked as EOG
llm_load_vocab: control token: 128241 '<|reserved_special_token_233|>' is not marked as EOG
llm_load_vocab: control token: 128240 '<|reserved_special_token_232|>' is not marked as EOG
llm_load_vocab: control token: 128235 '<|reserved_special_token_227|>' is not marked as EOG
llm_load_vocab: control token: 128231 '<|reserved_special_token_223|>' is not marked as EOG
llm_load_vocab: control token: 128230 '<|reserved_special_token_222|>' is not marked as EOG
llm_load_vocab: control token: 128228 '<|reserved_special_token_220|>' is not marked as EOG
llm_load_vocab: control token: 128225 '<|reserved_special_token_217|>' is not marked as EOG
llm_load_vocab: control token: 128218 '<|reserved_special_token_210|>' is not marked as EOG
llm_load_vocab: control token: 128214 '<|reserved_special_token_206|>' is not marked as EOG
llm_load_vocab: control token: 128213 '<|reserved_special_token_205|>' is not marked as EOG
llm_load_vocab: control token: 128207 '<|reserved_special_token_199|>' is not marked as EOG
llm_load_vocab: control token: 128206 '<|reserved_special_token_198|>' is not marked as EOG
llm_load_vocab: control token: 128204 '<|reserved_special_token_196|>' is not marked as EOG
llm_load_vocab: control token: 128200 '<|reserved_special_token_192|>' is not marked as EOG
llm_load_vocab: control token: 128199 '<|reserved_special_token_191|>' is not marked as EOG
llm_load_vocab: control token: 128198 '<|reserved_special_token_190|>' is not marked as EOG
llm_load_vocab: control token: 128196 '<|reserved_special_token_188|>' is not marked as EOG
llm_load_vocab: control token: 128194 '<|reserved_special_token_186|>' is not marked as EOG
llm_load_vocab: control token: 128193 '<|reserved_special_token_185|>' is not marked as EOG
llm_load_vocab: control token: 128188 '<|reserved_special_token_180|>' is not marked as EOG
llm_load_vocab: control token: 128187 '<|reserved_special_token_179|>' is not marked as EOG
llm_load_vocab: control token: 128185 '<|reserved_special_token_177|>' is not marked as EOG
llm_load_vocab: control token: 128184 '<|reserved_special_token_176|>' is not marked as EOG
llm_load_vocab: control token: 128180 '<|reserved_special_token_172|>' is not marked as EOG
llm_load_vocab: control token: 128179 '<|reserved_special_token_171|>' is not marked as EOG
llm_load_vocab: control token: 128178 '<|reserved_special_token_170|>' is not marked as EOG
llm_load_vocab: control token: 128177 '<|reserved_special_token_169|>' is not marked as EOG
llm_load_vocab: control token: 128176 '<|reserved_special_token_168|>' is not marked as EOG
llm_load_vocab: control token: 128175 '<|reserved_special_token_167|>' is not marked as EOG
llm_load_vocab: control token: 128171 '<|reserved_special_token_163|>' is not marked as EOG
llm_load_vocab: control token: 128170 '<|reserved_special_token_162|>' is not marked as EOG
llm_load_vocab: control token: 128169 '<|reserved_special_token_161|>' is not marked as EOG
llm_load_vocab: control token: 128168 '<|reserved_special_token_160|>' is not marked as EOG
llm_load_vocab: control token: 128165 '<|reserved_special_token_157|>' is not marked as EOG
llm_load_vocab: control token: 128162 '<|reserved_special_token_154|>' is not marked as EOG
llm_load_vocab: control token: 128158 '<|reserved_special_token_150|>' is not marked as EOG
llm_load_vocab: control token: 128156 '<|reserved_special_token_148|>' is not marked as EOG
llm_load_vocab: control token: 128155 '<|reserved_special_token_147|>' is not marked as EOG
llm_load_vocab: control token: 128154 '<|reserved_special_token_146|>' is not marked as EOG
llm_load_vocab: control token: 128151 '<|reserved_special_token_143|>' is not marked as EOG
llm_load_vocab: control token: 128149 '<|reserved_special_token_141|>' is not marked as EOG
llm_load_vocab: control token: 128147 '<|reserved_special_token_139|>' is not marked as EOG
llm_load_vocab: control token: 128146 '<|reserved_special_token_138|>' is not marked as EOG
llm_load_vocab: control token: 128144 '<|reserved_special_token_136|>' is not marked as EOG
llm_load_vocab: control token: 128142 '<|reserved_special_token_134|>' is not marked as EOG
llm_load_vocab: control token: 128141 '<|reserved_special_token_133|>' is not marked as EOG
llm_load_vocab: control token: 128138 '<|reserved_special_token_130|>' is not marked as EOG
llm_load_vocab: control token: 128136 '<|reserved_special_token_128|>' is not marked as EOG
llm_load_vocab: control token: 128135 '<|reserved_special_token_127|>' is not marked as EOG
llm_load_vocab: control token: 128134 '<|reserved_special_token_126|>' is not marked as EOG
llm_load_vocab: control token: 128133 '<|reserved_special_token_125|>' is not marked as EOG
llm_load_vocab: control token: 128131 '<|reserved_special_token_123|>' is not marked as EOG
llm_load_vocab: control token: 128128 '<|reserved_special_token_120|>' is not marked as EOG
llm_load_vocab: control token: 128124 '<|reserved_special_token_116|>' is not marked as EOG
llm_load_vocab: control token: 128123 '<|reserved_special_token_115|>' is not marked as EOG
llm_load_vocab: control token: 128122 '<|reserved_special_token_114|>' is not marked as EOG
llm_load_vocab: control token: 128119 '<|reserved_special_token_111|>' is not marked as EOG
llm_load_vocab: control token: 128115 '<|reserved_special_token_107|>' is not marked as EOG
llm_load_vocab: control token: 128112 '<|reserved_special_token_104|>' is not marked as EOG
llm_load_vocab: control token: 128110 '<|reserved_special_token_102|>' is not marked as EOG
llm_load_vocab: control token: 128109 '<|reserved_special_token_101|>' is not marked as EOG
llm_load_vocab: control token: 128108 '<|reserved_special_token_100|>' is not marked as EOG
llm_load_vocab: control token: 128106 '<|reserved_special_token_98|>' is not marked as EOG
llm_load_vocab: control token: 128103 '<|reserved_special_token_95|>' is not marked as EOG
llm_load_vocab: control token: 128102 '<|reserved_special_token_94|>' is not marked as EOG
llm_load_vocab: control token: 128101 '<|reserved_special_token_93|>' is not marked as EOG
llm_load_vocab: control token: 128097 '<|reserved_special_token_89|>' is not marked as EOG
llm_load_vocab: control token: 128091 '<|reserved_special_token_83|>' is not marked as EOG
llm_load_vocab: control token: 128090 '<|reserved_special_token_82|>' is not marked as EOG
llm_load_vocab: control token: 128089 '<|reserved_special_token_81|>' is not marked as EOG
llm_load_vocab: control token: 128087 '<|reserved_special_token_79|>' is not marked as EOG
llm_load_vocab: control token: 128085 '<|reserved_special_token_77|>' is not marked as EOG
llm_load_vocab: control token: 128081 '<|reserved_special_token_73|>' is not marked as EOG
llm_load_vocab: control token: 128078 '<|reserved_special_token_70|>' is not marked as EOG
llm_load_vocab: control token: 128076 '<|reserved_special_token_68|>' is not marked as EOG
llm_load_vocab: control token: 128075 '<|reserved_special_token_67|>' is not marked as EOG
llm_load_vocab: control token: 128073 '<|reserved_special_token_65|>' is not marked as EOG
llm_load_vocab: control token: 128068 '<|reserved_special_token_60|>' is not marked as EOG
llm_load_vocab: control token: 128067 '<|reserved_special_token_59|>' is not marked as EOG
llm_load_vocab: control token: 128065 '<|reserved_special_token_57|>' is not marked as EOG
llm_load_vocab: control token: 128063 '<|reserved_special_token_55|>' is not marked as EOG
llm_load_vocab: control token: 128062 '<|reserved_special_token_54|>' is not marked as EOG
llm_load_vocab: control token: 128060 '<|reserved_special_token_52|>' is not marked as EOG
llm_load_vocab: control token: 128059 '<|reserved_special_token_51|>' is not marked as EOG
llm_load_vocab: control token: 128057 '<|reserved_special_token_49|>' is not marked as EOG
llm_load_vocab: control token: 128054 '<|reserved_special_token_46|>' is not marked as EOG
llm_load_vocab: control token: 128046 '<|reserved_special_token_38|>' is not marked as EOG
llm_load_vocab: control token: 128045 '<|reserved_special_token_37|>' is not marked as EOG
llm_load_vocab: control token: 128044 '<|reserved_special_token_36|>' is not marked as EOG
llm_load_vocab: control token: 128043 '<|reserved_special_token_35|>' is not marked as EOG
llm_load_vocab: control token: 128038 '<|reserved_special_token_30|>' is not marked as EOG
llm_load_vocab: control token: 128036 '<|reserved_special_token_28|>' is not marked as EOG
llm_load_vocab: control token: 128035 '<|reserved_special_token_27|>' is not marked as EOG
llm_load_vocab: control token: 128032 '<|reserved_special_token_24|>' is not marked as EOG
llm_load_vocab: control token: 128028 '<|reserved_special_token_20|>' is not marked as EOG
llm_load_vocab: control token: 128027 '<|reserved_special_token_19|>' is not marked as EOG
llm_load_vocab: control token: 128024 '<|reserved_special_token_16|>' is not marked as EOG
llm_load_vocab: control token: 128023 '<|reserved_special_token_15|>' is not marked as EOG
llm_load_vocab: control token: 128022 '<|reserved_special_token_14|>' is not marked as EOG
llm_load_vocab: control token: 128021 '<|reserved_special_token_13|>' is not marked as EOG
llm_load_vocab: control token: 128018 '<|reserved_special_token_10|>' is not marked as EOG
llm_load_vocab: control token: 128016 '<|reserved_special_token_8|>' is not marked as EOG
llm_load_vocab: control token: 128012 '<｜Assistant｜>' is not marked as EOG
llm_load_vocab: control token: 128011 '<｜User｜>' is not marked as EOG
llm_load_vocab: control token: 128005 '<|reserved_special_token_2|>' is not marked as EOG
llm_load_vocab: control token: 128004 '<|finetune_right_pad_id|>' is not marked as EOG
llm_load_vocab: control token: 128002 '<|reserved_special_token_0|>' is not marked as EOG
llm_load_vocab: control token: 128252 '<|reserved_special_token_244|>' is not marked as EOG
llm_load_vocab: control token: 128190 '<|reserved_special_token_182|>' is not marked as EOG
llm_load_vocab: control token: 128183 '<|reserved_special_token_175|>' is not marked as EOG
llm_load_vocab: control token: 128137 '<|reserved_special_token_129|>' is not marked as EOG
llm_load_vocab: control token: 128182 '<|reserved_special_token_174|>' is not marked as EOG
llm_load_vocab: control token: 128040 '<|reserved_special_token_32|>' is not marked as EOG
llm_load_vocab: control token: 128048 '<|reserved_special_token_40|>' is not marked as EOG
llm_load_vocab: control token: 128092 '<|reserved_special_token_84|>' is not marked as EOG
llm_load_vocab: control token: 128215 '<|reserved_special_token_207|>' is not marked as EOG
llm_load_vocab: control token: 128107 '<|reserved_special_token_99|>' is not marked as EOG
llm_load_vocab: control token: 128208 '<|reserved_special_token_200|>' is not marked as EOG
llm_load_vocab: control token: 128145 '<|reserved_special_token_137|>' is not marked as EOG
llm_load_vocab: control token: 128031 '<|reserved_special_token_23|>' is not marked as EOG
llm_load_vocab: control token: 128129 '<|reserved_special_token_121|>' is not marked as EOG
llm_load_vocab: control token: 128201 '<|reserved_special_token_193|>' is not marked as EOG
llm_load_vocab: control token: 128074 '<|reserved_special_token_66|>' is not marked as EOG
llm_load_vocab: control token: 128095 '<|reserved_special_token_87|>' is not marked as EOG
llm_load_vocab: control token: 128186 '<|reserved_special_token_178|>' is not marked as EOG
llm_load_vocab: control token: 128143 '<|reserved_special_token_135|>' is not marked as EOG
llm_load_vocab: control token: 128229 '<|reserved_special_token_221|>' is not marked as EOG
llm_load_vocab: control token: 128007 '<|end_header_id|>' is not marked as EOG
llm_load_vocab: control token: 128055 '<|reserved_special_token_47|>' is not marked as EOG
llm_load_vocab: control token: 128056 '<|reserved_special_token_48|>' is not marked as EOG
llm_load_vocab: control token: 128061 '<|reserved_special_token_53|>' is not marked as EOG
llm_load_vocab: control token: 128153 '<|reserved_special_token_145|>' is not marked as EOG
llm_load_vocab: control token: 128152 '<|reserved_special_token_144|>' is not marked as EOG
llm_load_vocab: control token: 128212 '<|reserved_special_token_204|>' is not marked as EOG
llm_load_vocab: control token: 128172 '<|reserved_special_token_164|>' is not marked as EOG
llm_load_vocab: control token: 128160 '<|reserved_special_token_152|>' is not marked as EOG
llm_load_vocab: control token: 128041 '<|reserved_special_token_33|>' is not marked as EOG
llm_load_vocab: control token: 128181 '<|reserved_special_token_173|>' is not marked as EOG
llm_load_vocab: control token: 128094 '<|reserved_special_token_86|>' is not marked as EOG
llm_load_vocab: control token: 128118 '<|reserved_special_token_110|>' is not marked as EOG
llm_load_vocab: control token: 128236 '<|reserved_special_token_228|>' is not marked as EOG
llm_load_vocab: control token: 128148 '<|reserved_special_token_140|>' is not marked as EOG
llm_load_vocab: control token: 128042 '<|reserved_special_token_34|>' is not marked as EOG
llm_load_vocab: control token: 128139 '<|reserved_special_token_131|>' is not marked as EOG
llm_load_vocab: control token: 128173 '<|reserved_special_token_165|>' is not marked as EOG
llm_load_vocab: control token: 128239 '<|reserved_special_token_231|>' is not marked as EOG
llm_load_vocab: control token: 128157 '<|reserved_special_token_149|>' is not marked as EOG
llm_load_vocab: control token: 128052 '<|reserved_special_token_44|>' is not marked as EOG
llm_load_vocab: control token: 128026 '<|reserved_special_token_18|>' is not marked as EOG
llm_load_vocab: control token: 128003 '<|reserved_special_token_1|>' is not marked as EOG
llm_load_vocab: control token: 128019 '<|reserved_special_token_11|>' is not marked as EOG
llm_load_vocab: control token: 128116 '<|reserved_special_token_108|>' is not marked as EOG
llm_load_vocab: control token: 128161 '<|reserved_special_token_153|>' is not marked as EOG
llm_load_vocab: control token: 128000 '<｜begin▁of▁sentence｜>' is not marked as EOG
llm_load_vocab: control token: 128226 '<|reserved_special_token_218|>' is not marked as EOG
llm_load_vocab: control token: 128159 '<|reserved_special_token_151|>' is not marked as EOG
llm_load_vocab: control token: 128088 '<|reserved_special_token_80|>' is not marked as EOG
llm_load_vocab: control token: 128163 '<|reserved_special_token_155|>' is not marked as EOG
llm_load_vocab: control token: 128113 '<|reserved_special_token_105|>' is not marked as EOG
llm_load_vocab: control token: 128250 '<|reserved_special_token_242|>' is not marked as EOG
llm_load_vocab: control token: 128125 '<|reserved_special_token_117|>' is not marked as EOG
llm_load_vocab: control token: 128053 '<|reserved_special_token_45|>' is not marked as EOG
llm_load_vocab: control token: 128224 '<|reserved_special_token_216|>' is not marked as EOG
llm_load_vocab: control token: 128247 '<|reserved_special_token_239|>' is not marked as EOG
llm_load_vocab: control token: 128251 '<|reserved_special_token_243|>' is not marked as EOG
llm_load_vocab: control token: 128216 '<|reserved_special_token_208|>' is not marked as EOG
llm_load_vocab: control token: 128006 '<|start_header_id|>' is not marked as EOG
llm_load_vocab: control token: 128211 '<|reserved_special_token_203|>' is not marked as EOG
llm_load_vocab: control token: 128077 '<|reserved_special_token_69|>' is not marked as EOG
llm_load_vocab: control token: 128237 '<|reserved_special_token_229|>' is not marked as EOG
llm_load_vocab: control token: 128086 '<|reserved_special_token_78|>' is not marked as EOG
llm_load_vocab: control token: 128001 '<｜end▁of▁sentence｜>' is not marked as EOG
llm_load_vocab: control token: 128227 '<|reserved_special_token_219|>' is not marked as EOG
llm_load_vocab: control token: 128058 '<|reserved_special_token_50|>' is not marked as EOG
llm_load_vocab: control token: 128100 '<|reserved_special_token_92|>' is not marked as EOG
llm_load_vocab: control token: 128209 '<|reserved_special_token_201|>' is not marked as EOG
llm_load_vocab: control token: 128084 '<|reserved_special_token_76|>' is not marked as EOG
llm_load_vocab: control token: 128071 '<|reserved_special_token_63|>' is not marked as EOG
llm_load_vocab: control token: 128070 '<|reserved_special_token_62|>' is not marked as EOG
llm_load_vocab: control token: 128049 '<|reserved_special_token_41|>' is not marked as EOG
llm_load_vocab: control token: 128197 '<|reserved_special_token_189|>' is not marked as EOG
llm_load_vocab: control token: 128072 '<|reserved_special_token_64|>' is not marked as EOG
llm_load_vocab: control token: 128223 '<|reserved_special_token_215|>' is not marked as EOG
llm_load_vocab: control token: 128217 '<|reserved_special_token_209|>' is not marked as EOG
llm_load_vocab: control token: 128111 '<|reserved_special_token_103|>' is not marked as EOG
llm_load_vocab: control token: 128203 '<|reserved_special_token_195|>' is not marked as EOG
llm_load_vocab: control token: 128051 '<|reserved_special_token_43|>' is not marked as EOG
llm_load_vocab: control token: 128030 '<|reserved_special_token_22|>' is not marked as EOG
llm_load_vocab: control token: 128117 '<|reserved_special_token_109|>' is not marked as EOG
llm_load_vocab: control token: 128010 '<|python_tag|>' is not marked as EOG
llm_load_vocab: control token: 128238 '<|reserved_special_token_230|>' is not marked as EOG
llm_load_vocab: control token: 128255 '<|reserved_special_token_247|>' is not marked as EOG
llm_load_vocab: control token: 128202 '<|reserved_special_token_194|>' is not marked as EOG
llm_load_vocab: control token: 128132 '<|reserved_special_token_124|>' is not marked as EOG
llm_load_vocab: control token: 128248 '<|reserved_special_token_240|>' is not marked as EOG
llm_load_vocab: control token: 128167 '<|reserved_special_token_159|>' is not marked as EOG
llm_load_vocab: control token: 128127 '<|reserved_special_token_119|>' is not marked as EOG
llm_load_vocab: control token: 128105 '<|reserved_special_token_97|>' is not marked as EOG
llm_load_vocab: control token: 128039 '<|reserved_special_token_31|>' is not marked as EOG
llm_load_vocab: control token: 128232 '<|reserved_special_token_224|>' is not marked as EOG
llm_load_vocab: control token: 128166 '<|reserved_special_token_158|>' is not marked as EOG
llm_load_vocab: control token: 128130 '<|reserved_special_token_122|>' is not marked as EOG
llm_load_vocab: control token: 128114 '<|reserved_special_token_106|>' is not marked as EOG
llm_load_vocab: control token: 128234 '<|reserved_special_token_226|>' is not marked as EOG
llm_load_vocab: control token: 128191 '<|reserved_special_token_183|>' is not marked as EOG
llm_load_vocab: control token: 128064 '<|reserved_special_token_56|>' is not marked as EOG
llm_load_vocab: control token: 128140 '<|reserved_special_token_132|>' is not marked as EOG
llm_load_vocab: control token: 128096 '<|reserved_special_token_88|>' is not marked as EOG
llm_load_vocab: control token: 128098 '<|reserved_special_token_90|>' is not marked as EOG
llm_load_vocab: control token: 128192 '<|reserved_special_token_184|>' is not marked as EOG
llm_load_vocab: control token: 128093 '<|reserved_special_token_85|>' is not marked as EOG
llm_load_vocab: control token: 128150 '<|reserved_special_token_142|>' is not marked as EOG
llm_load_vocab: control token: 128222 '<|reserved_special_token_214|>' is not marked as EOG
llm_load_vocab: control token: 128233 '<|reserved_special_token_225|>' is not marked as EOG
llm_load_vocab: control token: 128220 '<|reserved_special_token_212|>' is not marked as EOG
llm_load_vocab: control token: 128034 '<|reserved_special_token_26|>' is not marked as EOG
llm_load_vocab: control token: 128033 '<|reserved_special_token_25|>' is not marked as EOG
llm_load_vocab: control token: 128253 '<|reserved_special_token_245|>' is not marked as EOG
llm_load_vocab: control token: 128195 '<|reserved_special_token_187|>' is not marked as EOG
llm_load_vocab: control token: 128099 '<|reserved_special_token_91|>' is not marked as EOG
llm_load_vocab: control token: 128189 '<|reserved_special_token_181|>' is not marked as EOG
llm_load_vocab: control token: 128210 '<|reserved_special_token_202|>' is not marked as EOG
llm_load_vocab: control token: 128174 '<|reserved_special_token_166|>' is not marked as EOG
llm_load_vocab: control token: 128083 '<|reserved_special_token_75|>' is not marked as EOG
llm_load_vocab: control token: 128080 '<|reserved_special_token_72|>' is not marked as EOG
llm_load_vocab: control token: 128104 '<|reserved_special_token_96|>' is not marked as EOG
llm_load_vocab: control token: 128082 '<|reserved_special_token_74|>' is not marked as EOG
llm_load_vocab: control token: 128219 '<|reserved_special_token_211|>' is not marked as EOG
llm_load_vocab: control token: 128017 '<|reserved_special_token_9|>' is not marked as EOG
llm_load_vocab: control token: 128050 '<|reserved_special_token_42|>' is not marked as EOG
llm_load_vocab: control token: 128205 '<|reserved_special_token_197|>' is not marked as EOG
llm_load_vocab: control token: 128047 '<|reserved_special_token_39|>' is not marked as EOG
llm_load_vocab: control token: 128164 '<|reserved_special_token_156|>' is not marked as EOG
llm_load_vocab: control token: 128020 '<|reserved_special_token_12|>' is not marked as EOG
llm_load_vocab: control token: 128069 '<|reserved_special_token_61|>' is not marked as EOG
llm_load_vocab: control token: 128245 '<|reserved_special_token_237|>' is not marked as EOG
llm_load_vocab: control token: 128121 '<|reserved_special_token_113|>' is not marked as EOG
llm_load_vocab: control token: 128079 '<|reserved_special_token_71|>' is not marked as EOG
llm_load_vocab: control token: 128037 '<|reserved_special_token_29|>' is not marked as EOG
llm_load_vocab: control token: 128244 '<|reserved_special_token_236|>' is not marked as EOG
llm_load_vocab: control token: 128029 '<|reserved_special_token_21|>' is not marked as EOG
llm_load_vocab: control token: 128221 '<|reserved_special_token_213|>' is not marked as EOG
llm_load_vocab: control token: 128066 '<|reserved_special_token_58|>' is not marked as EOG
llm_load_vocab: control token: 128120 '<|reserved_special_token_112|>' is not marked as EOG
llm_load_vocab: control token: 128015 '<｜▁pad▁｜>' is not marked as EOG
llm_load_vocab: control token: 128025 '<|reserved_special_token_17|>' is not marked as EOG
llm_load_vocab: control token: 128126 '<|reserved_special_token_118|>' is not marked as EOG
llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 256
llm_load_vocab: token to piece cache size = 0.7999 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 128256
llm_load_print_meta: n_merges = 280147
llm_load_print_meta: vocab_only = 0
llm_load_print_meta: n_ctx_train = 131072
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 500000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn = 131072
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: ssm_dt_b_c_rms = 0
llm_load_print_meta: model type = 8B
llm_load_print_meta: model ftype = Q4_K - Medium
llm_load_print_meta: model params = 8.03 B
llm_load_print_meta: model size = 4.58 GiB (4.89 BPW)
llm_load_print_meta: general.name = DeepSeek R1 Distill Llama 8B
llm_load_print_meta: BOS token = 128000 '<｜begin▁of▁sentence｜>'
llm_load_print_meta: EOS token = 128001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: EOT token = 128009 '<|eot_id|>'
llm_load_print_meta: EOM token = 128008 '<|eom_id|>'
llm_load_print_meta: PAD token = 128001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: LF token = 128 'Ä'
llm_load_print_meta: EOG token = 128001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: EOG token = 128008 '<|eom_id|>'
llm_load_print_meta: EOG token = 128009 '<|eot_id|>'
llm_load_print_meta: max token length = 256
llm_load_tensors: tensor 'token_embd.weight' (q4_K) (and 0 others) cannot be used with preferred buffer type CPU_AARCH64, using CPU instead
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading output layer to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors: CPU_Mapped model buffer size = 281.81 MiB
llm_load_tensors: CUDA0 model buffer size = 4403.49 MiB
time=2025-02-08T01:11:55.536Z level=DEBUG source=server.go:600 msg="model load progress 0.06"
time=2025-02-08T01:11:55.787Z level=DEBUG source=server.go:600 msg="model load progress 0.35"
time=2025-02-08T01:11:56.039Z level=DEBUG source=server.go:600 msg="model load progress 0.67"
time=2025-02-08T01:11:56.290Z level=DEBUG source=server.go:600 msg="model load progress 0.99"
llama_new_context_with_model: n_seq_max = 1
llama_new_context_with_model: n_ctx = 65536
llama_new_context_with_model: n_ctx_per_seq = 65536
llama_new_context_with_model: n_batch = 512
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 1
llama_new_context_with_model: freq_base = 500000.0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: n_ctx_per_seq (65536) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_kv_cache_init: kv_size = 65536, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 32, can_shift = 1
llama_kv_cache_init: layer 0: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 1: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 2: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 3: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 4: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 5: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 6: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 7: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 8: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 9: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 10: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 11: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 12: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 13: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 14: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 15: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 16: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 17: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 18: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 19: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 20: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 21: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 22: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 23: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 24: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 25: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 26: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 27: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 28: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 29: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 30: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
llama_kv_cache_init: layer 31: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024
time=2025-02-08T01:11:56.541Z level=DEBUG source=server.go:600 msg="model load progress 1.00"
llama_kv_cache_init: CUDA0 KV buffer size = 8192.00 MiB
llama_new_context_with_model: KV self size = 8192.00 MiB, K (f16): 4096.00 MiB, V (f16): 4096.00 MiB
llama_new_context_with_model: CUDA_Host output buffer size = 0.50 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 258.50 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 136.01 MiB
llama_new_context_with_model: graph nodes = 903
llama_new_context_with_model: graph splits = 2
time=2025-02-08T01:11:56.793Z level=INFO source=server.go:594 msg="llama runner started in 144.60 seconds"
time=2025-02-08T01:11:56.793Z level=DEBUG source=sched.go:462 msg="finished setting up runner" model=/root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be
time=2025-02-08T01:11:56.793Z level=DEBUG source=sched.go:466 msg="context for request finished"
time=2025-02-08T01:11:56.793Z level=DEBUG source=sched.go:339 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be duration=2562047h47m16.854775807s
time=2025-02-08T01:11:56.793Z level=DEBUG source=sched.go:357 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be refCount=0
[GIN] 2025/02/08 - 01:11:56 | 200 | 2m25s | 127.0.0.1 | POST "/api/generate"

<!-- gh-comment-id:2644406314 --> @LwengGitHub commented on GitHub (Feb 8, 2025): This is all loading log info. Looks normal, and it works fine for subsequent use. But This warning (using CPU instead). I've observed that when the context length is adjusted to be larger, the portion using the CPU increases. Ex. num_ctx 131072. ----------------------------------Log-Info--------------------------------- [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached. [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production. - using env: export GIN_MODE=release - using code: gin.SetMode(gin.ReleaseMode) [GIN-debug] POST /api/pull --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers) [GIN-debug] POST /api/generate --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers) [GIN-debug] POST /api/chat --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers) [GIN-debug] POST /api/embed --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers) [GIN-debug] POST /api/embeddings --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers) [GIN-debug] POST /api/create --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers) [GIN-debug] POST /api/push --> github.com/ollama/ollama/server.(*Server).PushHandler-fm (5 handlers) [GIN-debug] POST /api/copy --> github.com/ollama/ollama/server.(*Server).CopyHandler-fm (5 handlers) [GIN-debug] DELETE /api/delete --> github.com/ollama/ollama/server.(*Server).DeleteHandler-fm (5 handlers) [GIN-debug] POST /api/show --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (5 handlers) [GIN-debug] POST /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers) [GIN-debug] HEAD /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers) [GIN-debug] GET /api/ps --> github.com/ollama/ollama/server.(*Server).PsHandler-fm (5 handlers) [GIN-debug] POST /v1/chat/completions --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers) [GIN-debug] POST /v1/completions --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (6 handlers) [GIN-debug] POST /v1/embeddings --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (6 handlers) [GIN-debug] GET /v1/models --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (6 handlers) [GIN-debug] GET /v1/models/:model --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (6 handlers) [GIN-debug] GET / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers) [GIN-debug] GET /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers) [GIN-debug] GET /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers) [GIN-debug] HEAD / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers) [GIN-debug] HEAD /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers) [GIN-debug] HEAD /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers) 2025/02/08 01:09:11 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES:0 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:30m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2025-02-08T01:09:11.816Z level=INFO source=images.go:432 msg="total blobs: 5" time=2025-02-08T01:09:11.834Z level=INFO source=images.go:439 msg="total unused blobs removed: 0" time=2025-02-08T01:09:11.854Z level=INFO source=routes.go:1238 msg="Listening on [::]:11434 (version 0.5.7-0-ga420a45-dirty)" time=2025-02-08T01:09:11.855Z level=DEBUG source=common.go:80 msg="runners located" dir=/usr/lib/ollama/runners time=2025-02-08T01:09:11.856Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server time=2025-02-08T01:09:11.856Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server time=2025-02-08T01:09:11.856Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v11_avx/ollama_llama_server time=2025-02-08T01:09:11.856Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v12_avx/ollama_llama_server time=2025-02-08T01:09:11.856Z level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cpu_avx2 cuda_v11_avx cuda_v12_avx cpu cpu_avx]" time=2025-02-08T01:09:11.856Z level=DEBUG source=routes.go:1268 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY" time=2025-02-08T01:09:11.856Z level=DEBUG source=sched.go:105 msg="starting llm scheduler" time=2025-02-08T01:09:11.856Z level=INFO source=gpu.go:226 msg="looking for compatible GPUs" time=2025-02-08T01:09:11.863Z level=DEBUG source=gpu.go:99 msg="searching for GPU discovery libraries for NVIDIA" time=2025-02-08T01:09:11.863Z level=DEBUG source=gpu.go:517 msg="Searching for GPU library" name=libcuda.so* time=2025-02-08T01:09:11.863Z level=DEBUG source=gpu.go:543 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /usr/lib/ollama/libcuda.so* /usr/local/nvidia/lib/libcuda.so* /usr/local/nvidia/lib64/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" time=2025-02-08T01:09:11.866Z level=DEBUG source=gpu.go:576 msg="discovered GPU libraries" paths="[/usr/lib/x86_64-linux-gnu/libcuda.so.1 /usr/lib/wsl/drivers/nv_dispi.inf_amd64_adf5a840df867035/libcuda.so.1.1]" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1 dlsym: cuInit - 0x7f844c420ec0 dlsym: cuDriverGetVersion - 0x7f844c420e80 dlsym: cuDeviceGetCount - 0x7f844c420efd dlsym: cuDeviceGet - 0x7f844c420ef7 dlsym: cuDeviceGetAttribute - 0x7f844c420de0 dlsym: cuDeviceGetUuid - 0x7f844c420f09 dlsym: cuDeviceGetName - 0x7f844c420f03 dlsym: cuCtxCreate_v3 - 0x7f844c420f7b dlsym: cuMemGetInfo_v2 - 0x7f844c42107d dlsym: cuCtxDestroy - 0x7f844c420f8d calling cuInit calling cuDriverGetVersion raw version 0x2f26 CUDA driver version: 12.7 calling cuDeviceGetCount device count 1 time=2025-02-08T01:09:11.976Z level=DEBUG source=gpu.go:134 msg="detected GPUs" count=1 library=/usr/lib/x86_64-linux-gnu/libcuda.so.1 [GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57] CUDA totalMem 24563 mb [GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57] CUDA freeMem 23042 mb [GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57] Compute Capability 8.9 time=2025-02-08T01:09:12.136Z level=DEBUG source=amd_linux.go:421 msg="amdgpu driver not detected /sys/module/amdgpu" releasing cuda driver library time=2025-02-08T01:09:12.136Z level=INFO source=types.go:131 msg="inference compute" id=GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57 library=cuda variant=v12 compute=8.9 driver=12.7 name="NVIDIA GeForce RTX 4090 D" total="24.0 GiB" available="22.5 GiB" time=2025-02-08T01:09:31.695Z level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="62.6 GiB" before.free="43.0 GiB" before.free_swap="16.0 GiB" now.total="62.6 GiB" now.free="43.0 GiB" now.free_swap="16.0 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1 dlsym: cuInit - 0x7f844c420ec0 dlsym: cuDriverGetVersion - 0x7f844c420e80 dlsym: cuDeviceGetCount - 0x7f844c420efd dlsym: cuDeviceGet - 0x7f844c420ef7 dlsym: cuDeviceGetAttribute - 0x7f844c420de0 dlsym: cuDeviceGetUuid - 0x7f844c420f09 dlsym: cuDeviceGetName - 0x7f844c420f03 dlsym: cuCtxCreate_v3 - 0x7f844c420f7b dlsym: cuMemGetInfo_v2 - 0x7f844c42107d dlsym: cuCtxDestroy - 0x7f844c420f8d calling cuInit calling cuDriverGetVersion raw version 0x2f26 CUDA driver version: 12.7 calling cuDeviceGetCount device count 1 time=2025-02-08T01:09:31.819Z level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57 name="NVIDIA GeForce RTX 4090 D" overhead="0 B" before.total="24.0 GiB" before.free="22.5 GiB" now.total="24.0 GiB" now.free="22.5 GiB" now.used="1.5 GiB" releasing cuda driver library time=2025-02-08T01:09:31.820Z level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x556960a934c0 gpu_count=1 time=2025-02-08T01:09:31.942Z level=DEBUG source=sched.go:224 msg="loading first model" model=/root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be time=2025-02-08T01:09:31.942Z level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[22.5 GiB]" time=2025-02-08T01:09:31.943Z level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="62.6 GiB" before.free="43.0 GiB" before.free_swap="16.0 GiB" now.total="62.6 GiB" now.free="43.0 GiB" now.free_swap="16.0 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1 dlsym: cuInit - 0x7f844c420ec0 dlsym: cuDriverGetVersion - 0x7f844c420e80 [GIN] 2025/02/08 - 01:09:22 | 200 | 51.803µs | 127.0.0.1 | HEAD "/" [GIN] 2025/02/08 - 01:09:22 | 200 | 22.569215ms | 127.0.0.1 | GET "/api/tags" [GIN] 2025/02/08 - 01:09:31 | 200 | 27.397µs | 127.0.0.1 | HEAD "/" [GIN] 2025/02/08 - 01:09:31 | 200 | 108.571714ms | 127.0.0.1 | POST "/api/show" dlsym: cuDeviceGetCount - 0x7f844c420efd dlsym: cuDeviceGet - 0x7f844c420ef7 dlsym: cuDeviceGetAttribute - 0x7f844c420de0 dlsym: cuDeviceGetUuid - 0x7f844c420f09 dlsym: cuDeviceGetName - 0x7f844c420f03 dlsym: cuCtxCreate_v3 - 0x7f844c420f7b dlsym: cuMemGetInfo_v2 - 0x7f844c42107d dlsym: cuCtxDestroy - 0x7f844c420f8d calling cuInit calling cuDriverGetVersion raw version 0x2f26 CUDA driver version: 12.7 calling cuDeviceGetCount device count 1 time=2025-02-08T01:09:32.086Z level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57 name="NVIDIA GeForce RTX 4090 D" overhead="0 B" before.total="24.0 GiB" before.free="22.5 GiB" now.total="24.0 GiB" now.free="22.5 GiB" now.used="1.5 GiB" releasing cuda driver library time=2025-02-08T01:09:32.086Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be gpu=GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57 parallel=1 available=24161288192 required="17.3 GiB" time=2025-02-08T01:09:32.086Z level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="62.6 GiB" before.free="43.0 GiB" before.free_swap="16.0 GiB" now.total="62.6 GiB" now.free="43.0 GiB" now.free_swap="16.0 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1 dlsym: cuInit - 0x7f844c420ec0 dlsym: cuDriverGetVersion - 0x7f844c420e80 dlsym: cuDeviceGetCount - 0x7f844c420efd dlsym: cuDeviceGet - 0x7f844c420ef7 dlsym: cuDeviceGetAttribute - 0x7f844c420de0 dlsym: cuDeviceGetUuid - 0x7f844c420f09 dlsym: cuDeviceGetName - 0x7f844c420f03 dlsym: cuCtxCreate_v3 - 0x7f844c420f7b dlsym: cuMemGetInfo_v2 - 0x7f844c42107d dlsym: cuCtxDestroy - 0x7f844c420f8d calling cuInit calling cuDriverGetVersion raw version 0x2f26 CUDA driver version: 12.7 calling cuDeviceGetCount device count 1 time=2025-02-08T01:09:32.202Z level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57 name="NVIDIA GeForce RTX 4090 D" overhead="0 B" before.total="24.0 GiB" before.free="22.5 GiB" now.total="24.0 GiB" now.free="22.5 GiB" now.used="1.5 GiB" releasing cuda driver library time=2025-02-08T01:09:32.202Z level=INFO source=server.go:104 msg="system memory" total="62.6 GiB" free="43.0 GiB" free_swap="16.0 GiB" time=2025-02-08T01:09:32.202Z level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[22.5 GiB]" time=2025-02-08T01:09:32.203Z level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="62.6 GiB" before.free="43.0 GiB" before.free_swap="16.0 GiB" now.total="62.6 GiB" now.free="43.0 GiB" now.free_swap="16.0 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.1 dlsym: cuInit - 0x7f844c420ec0 dlsym: cuDriverGetVersion - 0x7f844c420e80 dlsym: cuDeviceGetCount - 0x7f844c420efd dlsym: cuDeviceGet - 0x7f844c420ef7 dlsym: cuDeviceGetAttribute - 0x7f844c420de0 dlsym: cuDeviceGetUuid - 0x7f844c420f09 dlsym: cuDeviceGetName - 0x7f844c420f03 dlsym: cuCtxCreate_v3 - 0x7f844c420f7b dlsym: cuMemGetInfo_v2 - 0x7f844c42107d dlsym: cuCtxDestroy - 0x7f844c420f8d calling cuInit calling cuDriverGetVersion raw version 0x2f26 CUDA driver version: 12.7 calling cuDeviceGetCount device count 1 time=2025-02-08T01:09:32.318Z level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57 name="NVIDIA GeForce RTX 4090 D" overhead="0 B" before.total="24.0 GiB" before.free="22.5 GiB" now.total="24.0 GiB" now.free="22.5 GiB" now.used="1.5 GiB" releasing cuda driver library time=2025-02-08T01:09:32.320Z level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[22.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="17.3 GiB" memory.required.partial="17.3 GiB" memory.required.kv="8.0 GiB" memory.required.allocations="[17.3 GiB]" memory.weights.total="11.9 GiB" memory.weights.repeating="11.5 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="4.2 GiB" memory.graph.partial="4.4 GiB" time=2025-02-08T01:09:32.320Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server time=2025-02-08T01:09:32.320Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server time=2025-02-08T01:09:32.320Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v11_avx/ollama_llama_server time=2025-02-08T01:09:32.320Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v12_avx/ollama_llama_server time=2025-02-08T01:09:32.320Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx/ollama_llama_server time=2025-02-08T01:09:32.320Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cpu_avx2/ollama_llama_server time=2025-02-08T01:09:32.320Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v11_avx/ollama_llama_server time=2025-02-08T01:09:32.320Z level=DEBUG source=common.go:124 msg="availableServers : found" file=/usr/lib/ollama/runners/cuda_v12_avx/ollama_llama_server time=2025-02-08T01:09:32.320Z level=INFO source=server.go:223 msg="enabling flash attention" time=2025-02-08T01:09:32.320Z level=WARN source=server.go:231 msg="kv cache type not supported by model" type="" time=2025-02-08T01:09:32.321Z level=INFO source=server.go:376 msg="starting llama server" cmd="/usr/lib/ollama/runners/cuda_v12_avx/ollama_llama_server runner --model /root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be --ctx-size 65536 --batch-size 512 --n-gpu-layers 33 --verbose --threads 64 --flash-attn --parallel 1 --port 41863" time=2025-02-08T01:09:32.321Z level=DEBUG source=server.go:393 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin CUDA_VISIBLE_DEVICES=GPU-8597191f-7bd2-656c-5b45-1ef6b65f1d57 LD_LIBRARY_PATH=/usr/lib/ollama:/usr/lib/ollama:/usr/lib/ollama/runners/cuda_v12_avx:/usr/local/nvidia/lib:/usr/local/nvidia/lib64]" time=2025-02-08T01:09:32.321Z level=INFO source=sched.go:449 msg="loaded runners" count=1 time=2025-02-08T01:09:32.321Z level=INFO source=server.go:555 msg="waiting for llama runner to start responding" time=2025-02-08T01:09:32.322Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error" time=2025-02-08T01:09:32.392Z level=INFO source=runner.go:936 msg="starting go runner" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 4090 D, compute capability 8.9, VMM: yes time=2025-02-08T01:09:32.479Z level=INFO source=runner.go:937 msg=system info="CUDA : ARCHS = 600,610,620,700,720,750,800,860,870,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=64 time=2025-02-08T01:09:32.479Z level=INFO source=.:0 msg="Server listening on 127.0.0.1:41863" time=2025-02-08T01:09:32.573Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server loading model" llama_load_model_from_file: using device CUDA0 (NVIDIA GeForce RTX 4090 D) - 23042 MiB free llama_model_loader: loaded meta data with 28 key-value pairs and 292 tensors from /root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = DeepSeek R1 Distill Llama 8B llama_model_loader: - kv 3: general.basename str = DeepSeek-R1-Distill-Llama llama_model_loader: - kv 4: general.size_label str = 8B llama_model_loader: - kv 5: llama.block_count u32 = 32 llama_model_loader: - kv 6: llama.context_length u32 = 131072 llama_model_loader: - kv 7: llama.embedding_length u32 = 4096 llama_model_loader: - kv 8: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 9: llama.attention.head_count u32 = 32 llama_model_loader: - kv 10: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 11: llama.rope.freq_base f32 = 500000.000000 llama_model_loader: - kv 12: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 13: general.file_type u32 = 15 llama_model_loader: - kv 14: llama.vocab_size u32 = 128256 llama_model_loader: - kv 15: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 16: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 17: tokenizer.ggml.pre str = llama-bpe llama_model_loader: - kv 18: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 19: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 20: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... llama_model_loader: - kv 21: tokenizer.ggml.bos_token_id u32 = 128000 llama_model_loader: - kv 22: tokenizer.ggml.eos_token_id u32 = 128001 llama_model_loader: - kv 23: tokenizer.ggml.padding_token_id u32 = 128001 llama_model_loader: - kv 24: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 25: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 26: tokenizer.chat_template str = {% if not add_generation_prompt is de... llama_model_loader: - kv 27: general.quantization_version u32 = 2 llama_model_loader: - type f32: 66 tensors llama_model_loader: - type q4_K: 193 tensors llama_model_loader: - type q6_K: 33 tensors llm_load_vocab: control token: 128254 '<|reserved_special_token_246|>' is not marked as EOG llm_load_vocab: control token: 128249 '<|reserved_special_token_241|>' is not marked as EOG llm_load_vocab: control token: 128246 '<|reserved_special_token_238|>' is not marked as EOG llm_load_vocab: control token: 128243 '<|reserved_special_token_235|>' is not marked as EOG llm_load_vocab: control token: 128242 '<|reserved_special_token_234|>' is not marked as EOG llm_load_vocab: control token: 128241 '<|reserved_special_token_233|>' is not marked as EOG llm_load_vocab: control token: 128240 '<|reserved_special_token_232|>' is not marked as EOG llm_load_vocab: control token: 128235 '<|reserved_special_token_227|>' is not marked as EOG llm_load_vocab: control token: 128231 '<|reserved_special_token_223|>' is not marked as EOG llm_load_vocab: control token: 128230 '<|reserved_special_token_222|>' is not marked as EOG llm_load_vocab: control token: 128228 '<|reserved_special_token_220|>' is not marked as EOG llm_load_vocab: control token: 128225 '<|reserved_special_token_217|>' is not marked as EOG llm_load_vocab: control token: 128218 '<|reserved_special_token_210|>' is not marked as EOG llm_load_vocab: control token: 128214 '<|reserved_special_token_206|>' is not marked as EOG llm_load_vocab: control token: 128213 '<|reserved_special_token_205|>' is not marked as EOG llm_load_vocab: control token: 128207 '<|reserved_special_token_199|>' is not marked as EOG llm_load_vocab: control token: 128206 '<|reserved_special_token_198|>' is not marked as EOG llm_load_vocab: control token: 128204 '<|reserved_special_token_196|>' is not marked as EOG llm_load_vocab: control token: 128200 '<|reserved_special_token_192|>' is not marked as EOG llm_load_vocab: control token: 128199 '<|reserved_special_token_191|>' is not marked as EOG llm_load_vocab: control token: 128198 '<|reserved_special_token_190|>' is not marked as EOG llm_load_vocab: control token: 128196 '<|reserved_special_token_188|>' is not marked as EOG llm_load_vocab: control token: 128194 '<|reserved_special_token_186|>' is not marked as EOG llm_load_vocab: control token: 128193 '<|reserved_special_token_185|>' is not marked as EOG llm_load_vocab: control token: 128188 '<|reserved_special_token_180|>' is not marked as EOG llm_load_vocab: control token: 128187 '<|reserved_special_token_179|>' is not marked as EOG llm_load_vocab: control token: 128185 '<|reserved_special_token_177|>' is not marked as EOG llm_load_vocab: control token: 128184 '<|reserved_special_token_176|>' is not marked as EOG llm_load_vocab: control token: 128180 '<|reserved_special_token_172|>' is not marked as EOG llm_load_vocab: control token: 128179 '<|reserved_special_token_171|>' is not marked as EOG llm_load_vocab: control token: 128178 '<|reserved_special_token_170|>' is not marked as EOG llm_load_vocab: control token: 128177 '<|reserved_special_token_169|>' is not marked as EOG llm_load_vocab: control token: 128176 '<|reserved_special_token_168|>' is not marked as EOG llm_load_vocab: control token: 128175 '<|reserved_special_token_167|>' is not marked as EOG llm_load_vocab: control token: 128171 '<|reserved_special_token_163|>' is not marked as EOG llm_load_vocab: control token: 128170 '<|reserved_special_token_162|>' is not marked as EOG llm_load_vocab: control token: 128169 '<|reserved_special_token_161|>' is not marked as EOG llm_load_vocab: control token: 128168 '<|reserved_special_token_160|>' is not marked as EOG llm_load_vocab: control token: 128165 '<|reserved_special_token_157|>' is not marked as EOG llm_load_vocab: control token: 128162 '<|reserved_special_token_154|>' is not marked as EOG llm_load_vocab: control token: 128158 '<|reserved_special_token_150|>' is not marked as EOG llm_load_vocab: control token: 128156 '<|reserved_special_token_148|>' is not marked as EOG llm_load_vocab: control token: 128155 '<|reserved_special_token_147|>' is not marked as EOG llm_load_vocab: control token: 128154 '<|reserved_special_token_146|>' is not marked as EOG llm_load_vocab: control token: 128151 '<|reserved_special_token_143|>' is not marked as EOG llm_load_vocab: control token: 128149 '<|reserved_special_token_141|>' is not marked as EOG llm_load_vocab: control token: 128147 '<|reserved_special_token_139|>' is not marked as EOG llm_load_vocab: control token: 128146 '<|reserved_special_token_138|>' is not marked as EOG llm_load_vocab: control token: 128144 '<|reserved_special_token_136|>' is not marked as EOG llm_load_vocab: control token: 128142 '<|reserved_special_token_134|>' is not marked as EOG llm_load_vocab: control token: 128141 '<|reserved_special_token_133|>' is not marked as EOG llm_load_vocab: control token: 128138 '<|reserved_special_token_130|>' is not marked as EOG llm_load_vocab: control token: 128136 '<|reserved_special_token_128|>' is not marked as EOG llm_load_vocab: control token: 128135 '<|reserved_special_token_127|>' is not marked as EOG llm_load_vocab: control token: 128134 '<|reserved_special_token_126|>' is not marked as EOG llm_load_vocab: control token: 128133 '<|reserved_special_token_125|>' is not marked as EOG llm_load_vocab: control token: 128131 '<|reserved_special_token_123|>' is not marked as EOG llm_load_vocab: control token: 128128 '<|reserved_special_token_120|>' is not marked as EOG llm_load_vocab: control token: 128124 '<|reserved_special_token_116|>' is not marked as EOG llm_load_vocab: control token: 128123 '<|reserved_special_token_115|>' is not marked as EOG llm_load_vocab: control token: 128122 '<|reserved_special_token_114|>' is not marked as EOG llm_load_vocab: control token: 128119 '<|reserved_special_token_111|>' is not marked as EOG llm_load_vocab: control token: 128115 '<|reserved_special_token_107|>' is not marked as EOG llm_load_vocab: control token: 128112 '<|reserved_special_token_104|>' is not marked as EOG llm_load_vocab: control token: 128110 '<|reserved_special_token_102|>' is not marked as EOG llm_load_vocab: control token: 128109 '<|reserved_special_token_101|>' is not marked as EOG llm_load_vocab: control token: 128108 '<|reserved_special_token_100|>' is not marked as EOG llm_load_vocab: control token: 128106 '<|reserved_special_token_98|>' is not marked as EOG llm_load_vocab: control token: 128103 '<|reserved_special_token_95|>' is not marked as EOG llm_load_vocab: control token: 128102 '<|reserved_special_token_94|>' is not marked as EOG llm_load_vocab: control token: 128101 '<|reserved_special_token_93|>' is not marked as EOG llm_load_vocab: control token: 128097 '<|reserved_special_token_89|>' is not marked as EOG llm_load_vocab: control token: 128091 '<|reserved_special_token_83|>' is not marked as EOG llm_load_vocab: control token: 128090 '<|reserved_special_token_82|>' is not marked as EOG llm_load_vocab: control token: 128089 '<|reserved_special_token_81|>' is not marked as EOG llm_load_vocab: control token: 128087 '<|reserved_special_token_79|>' is not marked as EOG llm_load_vocab: control token: 128085 '<|reserved_special_token_77|>' is not marked as EOG llm_load_vocab: control token: 128081 '<|reserved_special_token_73|>' is not marked as EOG llm_load_vocab: control token: 128078 '<|reserved_special_token_70|>' is not marked as EOG llm_load_vocab: control token: 128076 '<|reserved_special_token_68|>' is not marked as EOG llm_load_vocab: control token: 128075 '<|reserved_special_token_67|>' is not marked as EOG llm_load_vocab: control token: 128073 '<|reserved_special_token_65|>' is not marked as EOG llm_load_vocab: control token: 128068 '<|reserved_special_token_60|>' is not marked as EOG llm_load_vocab: control token: 128067 '<|reserved_special_token_59|>' is not marked as EOG llm_load_vocab: control token: 128065 '<|reserved_special_token_57|>' is not marked as EOG llm_load_vocab: control token: 128063 '<|reserved_special_token_55|>' is not marked as EOG llm_load_vocab: control token: 128062 '<|reserved_special_token_54|>' is not marked as EOG llm_load_vocab: control token: 128060 '<|reserved_special_token_52|>' is not marked as EOG llm_load_vocab: control token: 128059 '<|reserved_special_token_51|>' is not marked as EOG llm_load_vocab: control token: 128057 '<|reserved_special_token_49|>' is not marked as EOG llm_load_vocab: control token: 128054 '<|reserved_special_token_46|>' is not marked as EOG llm_load_vocab: control token: 128046 '<|reserved_special_token_38|>' is not marked as EOG llm_load_vocab: control token: 128045 '<|reserved_special_token_37|>' is not marked as EOG llm_load_vocab: control token: 128044 '<|reserved_special_token_36|>' is not marked as EOG llm_load_vocab: control token: 128043 '<|reserved_special_token_35|>' is not marked as EOG llm_load_vocab: control token: 128038 '<|reserved_special_token_30|>' is not marked as EOG llm_load_vocab: control token: 128036 '<|reserved_special_token_28|>' is not marked as EOG llm_load_vocab: control token: 128035 '<|reserved_special_token_27|>' is not marked as EOG llm_load_vocab: control token: 128032 '<|reserved_special_token_24|>' is not marked as EOG llm_load_vocab: control token: 128028 '<|reserved_special_token_20|>' is not marked as EOG llm_load_vocab: control token: 128027 '<|reserved_special_token_19|>' is not marked as EOG llm_load_vocab: control token: 128024 '<|reserved_special_token_16|>' is not marked as EOG llm_load_vocab: control token: 128023 '<|reserved_special_token_15|>' is not marked as EOG llm_load_vocab: control token: 128022 '<|reserved_special_token_14|>' is not marked as EOG llm_load_vocab: control token: 128021 '<|reserved_special_token_13|>' is not marked as EOG llm_load_vocab: control token: 128018 '<|reserved_special_token_10|>' is not marked as EOG llm_load_vocab: control token: 128016 '<|reserved_special_token_8|>' is not marked as EOG llm_load_vocab: control token: 128012 '<｜Assistant｜>' is not marked as EOG llm_load_vocab: control token: 128011 '<｜User｜>' is not marked as EOG llm_load_vocab: control token: 128005 '<|reserved_special_token_2|>' is not marked as EOG llm_load_vocab: control token: 128004 '<|finetune_right_pad_id|>' is not marked as EOG llm_load_vocab: control token: 128002 '<|reserved_special_token_0|>' is not marked as EOG llm_load_vocab: control token: 128252 '<|reserved_special_token_244|>' is not marked as EOG llm_load_vocab: control token: 128190 '<|reserved_special_token_182|>' is not marked as EOG llm_load_vocab: control token: 128183 '<|reserved_special_token_175|>' is not marked as EOG llm_load_vocab: control token: 128137 '<|reserved_special_token_129|>' is not marked as EOG llm_load_vocab: control token: 128182 '<|reserved_special_token_174|>' is not marked as EOG llm_load_vocab: control token: 128040 '<|reserved_special_token_32|>' is not marked as EOG llm_load_vocab: control token: 128048 '<|reserved_special_token_40|>' is not marked as EOG llm_load_vocab: control token: 128092 '<|reserved_special_token_84|>' is not marked as EOG llm_load_vocab: control token: 128215 '<|reserved_special_token_207|>' is not marked as EOG llm_load_vocab: control token: 128107 '<|reserved_special_token_99|>' is not marked as EOG llm_load_vocab: control token: 128208 '<|reserved_special_token_200|>' is not marked as EOG llm_load_vocab: control token: 128145 '<|reserved_special_token_137|>' is not marked as EOG llm_load_vocab: control token: 128031 '<|reserved_special_token_23|>' is not marked as EOG llm_load_vocab: control token: 128129 '<|reserved_special_token_121|>' is not marked as EOG llm_load_vocab: control token: 128201 '<|reserved_special_token_193|>' is not marked as EOG llm_load_vocab: control token: 128074 '<|reserved_special_token_66|>' is not marked as EOG llm_load_vocab: control token: 128095 '<|reserved_special_token_87|>' is not marked as EOG llm_load_vocab: control token: 128186 '<|reserved_special_token_178|>' is not marked as EOG llm_load_vocab: control token: 128143 '<|reserved_special_token_135|>' is not marked as EOG llm_load_vocab: control token: 128229 '<|reserved_special_token_221|>' is not marked as EOG llm_load_vocab: control token: 128007 '<|end_header_id|>' is not marked as EOG llm_load_vocab: control token: 128055 '<|reserved_special_token_47|>' is not marked as EOG llm_load_vocab: control token: 128056 '<|reserved_special_token_48|>' is not marked as EOG llm_load_vocab: control token: 128061 '<|reserved_special_token_53|>' is not marked as EOG llm_load_vocab: control token: 128153 '<|reserved_special_token_145|>' is not marked as EOG llm_load_vocab: control token: 128152 '<|reserved_special_token_144|>' is not marked as EOG llm_load_vocab: control token: 128212 '<|reserved_special_token_204|>' is not marked as EOG llm_load_vocab: control token: 128172 '<|reserved_special_token_164|>' is not marked as EOG llm_load_vocab: control token: 128160 '<|reserved_special_token_152|>' is not marked as EOG llm_load_vocab: control token: 128041 '<|reserved_special_token_33|>' is not marked as EOG llm_load_vocab: control token: 128181 '<|reserved_special_token_173|>' is not marked as EOG llm_load_vocab: control token: 128094 '<|reserved_special_token_86|>' is not marked as EOG llm_load_vocab: control token: 128118 '<|reserved_special_token_110|>' is not marked as EOG llm_load_vocab: control token: 128236 '<|reserved_special_token_228|>' is not marked as EOG llm_load_vocab: control token: 128148 '<|reserved_special_token_140|>' is not marked as EOG llm_load_vocab: control token: 128042 '<|reserved_special_token_34|>' is not marked as EOG llm_load_vocab: control token: 128139 '<|reserved_special_token_131|>' is not marked as EOG llm_load_vocab: control token: 128173 '<|reserved_special_token_165|>' is not marked as EOG llm_load_vocab: control token: 128239 '<|reserved_special_token_231|>' is not marked as EOG llm_load_vocab: control token: 128157 '<|reserved_special_token_149|>' is not marked as EOG llm_load_vocab: control token: 128052 '<|reserved_special_token_44|>' is not marked as EOG llm_load_vocab: control token: 128026 '<|reserved_special_token_18|>' is not marked as EOG llm_load_vocab: control token: 128003 '<|reserved_special_token_1|>' is not marked as EOG llm_load_vocab: control token: 128019 '<|reserved_special_token_11|>' is not marked as EOG llm_load_vocab: control token: 128116 '<|reserved_special_token_108|>' is not marked as EOG llm_load_vocab: control token: 128161 '<|reserved_special_token_153|>' is not marked as EOG llm_load_vocab: control token: 128000 '<｜begin▁of▁sentence｜>' is not marked as EOG llm_load_vocab: control token: 128226 '<|reserved_special_token_218|>' is not marked as EOG llm_load_vocab: control token: 128159 '<|reserved_special_token_151|>' is not marked as EOG llm_load_vocab: control token: 128088 '<|reserved_special_token_80|>' is not marked as EOG llm_load_vocab: control token: 128163 '<|reserved_special_token_155|>' is not marked as EOG llm_load_vocab: control token: 128113 '<|reserved_special_token_105|>' is not marked as EOG llm_load_vocab: control token: 128250 '<|reserved_special_token_242|>' is not marked as EOG llm_load_vocab: control token: 128125 '<|reserved_special_token_117|>' is not marked as EOG llm_load_vocab: control token: 128053 '<|reserved_special_token_45|>' is not marked as EOG llm_load_vocab: control token: 128224 '<|reserved_special_token_216|>' is not marked as EOG llm_load_vocab: control token: 128247 '<|reserved_special_token_239|>' is not marked as EOG llm_load_vocab: control token: 128251 '<|reserved_special_token_243|>' is not marked as EOG llm_load_vocab: control token: 128216 '<|reserved_special_token_208|>' is not marked as EOG llm_load_vocab: control token: 128006 '<|start_header_id|>' is not marked as EOG llm_load_vocab: control token: 128211 '<|reserved_special_token_203|>' is not marked as EOG llm_load_vocab: control token: 128077 '<|reserved_special_token_69|>' is not marked as EOG llm_load_vocab: control token: 128237 '<|reserved_special_token_229|>' is not marked as EOG llm_load_vocab: control token: 128086 '<|reserved_special_token_78|>' is not marked as EOG llm_load_vocab: control token: 128001 '<｜end▁of▁sentence｜>' is not marked as EOG llm_load_vocab: control token: 128227 '<|reserved_special_token_219|>' is not marked as EOG llm_load_vocab: control token: 128058 '<|reserved_special_token_50|>' is not marked as EOG llm_load_vocab: control token: 128100 '<|reserved_special_token_92|>' is not marked as EOG llm_load_vocab: control token: 128209 '<|reserved_special_token_201|>' is not marked as EOG llm_load_vocab: control token: 128084 '<|reserved_special_token_76|>' is not marked as EOG llm_load_vocab: control token: 128071 '<|reserved_special_token_63|>' is not marked as EOG llm_load_vocab: control token: 128070 '<|reserved_special_token_62|>' is not marked as EOG llm_load_vocab: control token: 128049 '<|reserved_special_token_41|>' is not marked as EOG llm_load_vocab: control token: 128197 '<|reserved_special_token_189|>' is not marked as EOG llm_load_vocab: control token: 128072 '<|reserved_special_token_64|>' is not marked as EOG llm_load_vocab: control token: 128223 '<|reserved_special_token_215|>' is not marked as EOG llm_load_vocab: control token: 128217 '<|reserved_special_token_209|>' is not marked as EOG llm_load_vocab: control token: 128111 '<|reserved_special_token_103|>' is not marked as EOG llm_load_vocab: control token: 128203 '<|reserved_special_token_195|>' is not marked as EOG llm_load_vocab: control token: 128051 '<|reserved_special_token_43|>' is not marked as EOG llm_load_vocab: control token: 128030 '<|reserved_special_token_22|>' is not marked as EOG llm_load_vocab: control token: 128117 '<|reserved_special_token_109|>' is not marked as EOG llm_load_vocab: control token: 128010 '<|python_tag|>' is not marked as EOG llm_load_vocab: control token: 128238 '<|reserved_special_token_230|>' is not marked as EOG llm_load_vocab: control token: 128255 '<|reserved_special_token_247|>' is not marked as EOG llm_load_vocab: control token: 128202 '<|reserved_special_token_194|>' is not marked as EOG llm_load_vocab: control token: 128132 '<|reserved_special_token_124|>' is not marked as EOG llm_load_vocab: control token: 128248 '<|reserved_special_token_240|>' is not marked as EOG llm_load_vocab: control token: 128167 '<|reserved_special_token_159|>' is not marked as EOG llm_load_vocab: control token: 128127 '<|reserved_special_token_119|>' is not marked as EOG llm_load_vocab: control token: 128105 '<|reserved_special_token_97|>' is not marked as EOG llm_load_vocab: control token: 128039 '<|reserved_special_token_31|>' is not marked as EOG llm_load_vocab: control token: 128232 '<|reserved_special_token_224|>' is not marked as EOG llm_load_vocab: control token: 128166 '<|reserved_special_token_158|>' is not marked as EOG llm_load_vocab: control token: 128130 '<|reserved_special_token_122|>' is not marked as EOG llm_load_vocab: control token: 128114 '<|reserved_special_token_106|>' is not marked as EOG llm_load_vocab: control token: 128234 '<|reserved_special_token_226|>' is not marked as EOG llm_load_vocab: control token: 128191 '<|reserved_special_token_183|>' is not marked as EOG llm_load_vocab: control token: 128064 '<|reserved_special_token_56|>' is not marked as EOG llm_load_vocab: control token: 128140 '<|reserved_special_token_132|>' is not marked as EOG llm_load_vocab: control token: 128096 '<|reserved_special_token_88|>' is not marked as EOG llm_load_vocab: control token: 128098 '<|reserved_special_token_90|>' is not marked as EOG llm_load_vocab: control token: 128192 '<|reserved_special_token_184|>' is not marked as EOG llm_load_vocab: control token: 128093 '<|reserved_special_token_85|>' is not marked as EOG llm_load_vocab: control token: 128150 '<|reserved_special_token_142|>' is not marked as EOG llm_load_vocab: control token: 128222 '<|reserved_special_token_214|>' is not marked as EOG llm_load_vocab: control token: 128233 '<|reserved_special_token_225|>' is not marked as EOG llm_load_vocab: control token: 128220 '<|reserved_special_token_212|>' is not marked as EOG llm_load_vocab: control token: 128034 '<|reserved_special_token_26|>' is not marked as EOG llm_load_vocab: control token: 128033 '<|reserved_special_token_25|>' is not marked as EOG llm_load_vocab: control token: 128253 '<|reserved_special_token_245|>' is not marked as EOG llm_load_vocab: control token: 128195 '<|reserved_special_token_187|>' is not marked as EOG llm_load_vocab: control token: 128099 '<|reserved_special_token_91|>' is not marked as EOG llm_load_vocab: control token: 128189 '<|reserved_special_token_181|>' is not marked as EOG llm_load_vocab: control token: 128210 '<|reserved_special_token_202|>' is not marked as EOG llm_load_vocab: control token: 128174 '<|reserved_special_token_166|>' is not marked as EOG llm_load_vocab: control token: 128083 '<|reserved_special_token_75|>' is not marked as EOG llm_load_vocab: control token: 128080 '<|reserved_special_token_72|>' is not marked as EOG llm_load_vocab: control token: 128104 '<|reserved_special_token_96|>' is not marked as EOG llm_load_vocab: control token: 128082 '<|reserved_special_token_74|>' is not marked as EOG llm_load_vocab: control token: 128219 '<|reserved_special_token_211|>' is not marked as EOG llm_load_vocab: control token: 128017 '<|reserved_special_token_9|>' is not marked as EOG llm_load_vocab: control token: 128050 '<|reserved_special_token_42|>' is not marked as EOG llm_load_vocab: control token: 128205 '<|reserved_special_token_197|>' is not marked as EOG llm_load_vocab: control token: 128047 '<|reserved_special_token_39|>' is not marked as EOG llm_load_vocab: control token: 128164 '<|reserved_special_token_156|>' is not marked as EOG llm_load_vocab: control token: 128020 '<|reserved_special_token_12|>' is not marked as EOG llm_load_vocab: control token: 128069 '<|reserved_special_token_61|>' is not marked as EOG llm_load_vocab: control token: 128245 '<|reserved_special_token_237|>' is not marked as EOG llm_load_vocab: control token: 128121 '<|reserved_special_token_113|>' is not marked as EOG llm_load_vocab: control token: 128079 '<|reserved_special_token_71|>' is not marked as EOG llm_load_vocab: control token: 128037 '<|reserved_special_token_29|>' is not marked as EOG llm_load_vocab: control token: 128244 '<|reserved_special_token_236|>' is not marked as EOG llm_load_vocab: control token: 128029 '<|reserved_special_token_21|>' is not marked as EOG llm_load_vocab: control token: 128221 '<|reserved_special_token_213|>' is not marked as EOG llm_load_vocab: control token: 128066 '<|reserved_special_token_58|>' is not marked as EOG llm_load_vocab: control token: 128120 '<|reserved_special_token_112|>' is not marked as EOG llm_load_vocab: control token: 128015 '<｜▁pad▁｜>' is not marked as EOG llm_load_vocab: control token: 128025 '<|reserved_special_token_17|>' is not marked as EOG llm_load_vocab: control token: 128126 '<|reserved_special_token_118|>' is not marked as EOG llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect llm_load_vocab: special tokens cache size = 256 llm_load_vocab: token to piece cache size = 0.7999 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 128256 llm_load_print_meta: n_merges = 280147 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 131072 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 500000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 131072 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = 8B llm_load_print_meta: model ftype = Q4_K - Medium llm_load_print_meta: model params = 8.03 B llm_load_print_meta: model size = 4.58 GiB (4.89 BPW) llm_load_print_meta: general.name = DeepSeek R1 Distill Llama 8B llm_load_print_meta: BOS token = 128000 '<｜begin▁of▁sentence｜>' llm_load_print_meta: EOS token = 128001 '<｜end▁of▁sentence｜>' llm_load_print_meta: EOT token = 128009 '<|eot_id|>' llm_load_print_meta: EOM token = 128008 '<|eom_id|>' llm_load_print_meta: PAD token = 128001 '<｜end▁of▁sentence｜>' llm_load_print_meta: LF token = 128 'Ä' llm_load_print_meta: EOG token = 128001 '<｜end▁of▁sentence｜>' llm_load_print_meta: EOG token = 128008 '<|eom_id|>' llm_load_print_meta: EOG token = 128009 '<|eot_id|>' llm_load_print_meta: max token length = 256 llm_load_tensors: tensor 'token_embd.weight' (q4_K) (and 0 others) cannot be used with preferred buffer type CPU_AARCH64, using CPU instead llm_load_tensors: offloading 32 repeating layers to GPU llm_load_tensors: offloading output layer to GPU llm_load_tensors: offloaded 33/33 layers to GPU llm_load_tensors: CPU_Mapped model buffer size = 281.81 MiB llm_load_tensors: CUDA0 model buffer size = 4403.49 MiB time=2025-02-08T01:11:55.536Z level=DEBUG source=server.go:600 msg="model load progress 0.06" time=2025-02-08T01:11:55.787Z level=DEBUG source=server.go:600 msg="model load progress 0.35" time=2025-02-08T01:11:56.039Z level=DEBUG source=server.go:600 msg="model load progress 0.67" time=2025-02-08T01:11:56.290Z level=DEBUG source=server.go:600 msg="model load progress 0.99" llama_new_context_with_model: n_seq_max = 1 llama_new_context_with_model: n_ctx = 65536 llama_new_context_with_model: n_ctx_per_seq = 65536 llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 1 llama_new_context_with_model: freq_base = 500000.0 llama_new_context_with_model: freq_scale = 1 llama_new_context_with_model: n_ctx_per_seq (65536) < n_ctx_train (131072) -- the full capacity of the model will not be utilized llama_kv_cache_init: kv_size = 65536, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 32, can_shift = 1 llama_kv_cache_init: layer 0: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 1: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 2: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 3: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 4: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 5: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 6: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 7: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 8: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 9: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 10: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 11: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 12: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 13: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 14: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 15: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 16: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 17: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 18: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 19: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 20: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 21: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 22: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 23: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 24: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 25: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 26: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 27: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 28: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 29: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 30: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 llama_kv_cache_init: layer 31: n_embd_k_gqa = 1024, n_embd_v_gqa = 1024 time=2025-02-08T01:11:56.541Z level=DEBUG source=server.go:600 msg="model load progress 1.00" llama_kv_cache_init: CUDA0 KV buffer size = 8192.00 MiB llama_new_context_with_model: KV self size = 8192.00 MiB, K (f16): 4096.00 MiB, V (f16): 4096.00 MiB llama_new_context_with_model: CUDA_Host output buffer size = 0.50 MiB llama_new_context_with_model: CUDA0 compute buffer size = 258.50 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 136.01 MiB llama_new_context_with_model: graph nodes = 903 llama_new_context_with_model: graph splits = 2 time=2025-02-08T01:11:56.793Z level=INFO source=server.go:594 msg="llama runner started in 144.60 seconds" time=2025-02-08T01:11:56.793Z level=DEBUG source=sched.go:462 msg="finished setting up runner" model=/root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be time=2025-02-08T01:11:56.793Z level=DEBUG source=sched.go:466 msg="context for request finished" time=2025-02-08T01:11:56.793Z level=DEBUG source=sched.go:339 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be duration=2562047h47m16.854775807s time=2025-02-08T01:11:56.793Z level=DEBUG source=sched.go:357 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be refCount=0 [GIN] 2025/02/08 - 01:11:56 | 200 | 2m25s | 127.0.0.1 | POST "/api/generate"

GiteaMirror commented 2026-04-28 22:56:53 -05:00

Author

Owner

Copy Link

@jessegross commented on GitHub (Feb 8, 2025):

It's just a warning, you can ignore it.

Increasing num_ctx likely causes you to exceed the size of your VRAM, causing the model to spill over into system RAM and run partially on the CPU.

<!-- gh-comment-id:2644416900 --> @jessegross commented on GitHub (Feb 8, 2025): It's just a warning, you can ignore it. Increasing num_ctx likely causes you to exceed the size of your VRAM, causing the model to spill over into system RAM and run partially on the CPU.

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

parth-agent-tui-slash-selector

parth-remove-ollama-agent-command

parth-agent-harness-skills-synthetic-tool

hoyyeva/fix-anthropic-text-before-thinking

parth-agent-cli-markdown-rendering

mxyng/docs-cloud

parth-update-hermes-launch

hoyyeva/vscode-extension-docs-update

parth-gemma4-chat-template-renderer

parth-api-status-context-length

hoyyeva/wire-up-context-length

hoyyeva/claude-code-context-doc

jmorganca/investigate-issue-17046

hoyyeva/hermes-docs

jmorganca/agent-loop-style

hoyyeva/openclaw

parth-agent-loop

hoyyeva/ollama-vscode-extension

brucemacd/cache-metrics

brucemacd/hermes-desktop

hoyyeva/docs-vscode

parth-input-style-experiment

brucemacd/docs-glm52

hoyyeva/poc-docs

Parth/mlx-launch-recommendations

parth-first-time-app-cli-experience

test/darwin-xcode-pin

improve-cloud-model-recommendations

hoyyeva/goose-docs

jmorganca/context-limit-fixes

hoyyeva/qwen-doc

hoyyeva/vscode-docs

jmorganca/remove-mlx-imagegen-code

parth-copilot-token-length-defaults

hoyyeva/poolside-windows

laguna-support

jmorganca/harden-markdown-rendering

laguna-renderer-parser

laguna-llamacpp

codex/make-integration-hidden-and-lunchable

brucemacd/omp-docs

pdevine/gguf-mtp-oldstyle

hoyyeva/migrate-pi

hoyyeva/anthropic-local-image-path

parth-launch-codex-app

hoyyeva/anthropic-reference-images-path

parth-anthropic-reference-images-path

brucemacd/download-before-remove

hoyyeva/editor-config-repair

parth-mlx-decode-checkpoints

parth/hide-claude-desktop-till-release

parth-add-claude-code-autoinstall

release_v0.22.0

pdevine/manifest-list

codex/fix-codex-model-metadata-warning

pdevine/addressable-manifest

brucemacd/launch-fetch-reccomended

jmorganca/llama-compat

launch-copilot-cli

release_v0.20.7

parth-auto-save-backup

parth-test

jmorganca/gemma4-audio-replacements

fix-manifest-digest-on-pull

hoyyeva/vscode-improve

brucemacd/install-server-wait

parth/update-claude-docs

brucemac/start-ap-install

pdevine/mlx-update

pdevine/qwen35_vision

drifkin/api-show-fallback

mintlify/image-generation-1773352582

hoyyeva/server-context-length-local-config

jmorganca/faster-reptition-penalties

jmorganca/convert-nemotron

parth-pi-thinking

pdevine/sampling-penalties

jmorganca/fix-create-quantization-memory

dongchen/resumable_transfer_fix

pdevine/sampling-cache-error

jessegross/mlx-usage

hoyyeva/openclaw-config

hoyyeva/app-html

pdevine/qwen3next

brucemacd/sign-sh-install

brucemacd/tui-update

brucemacd/usage-api

jmorganca/launch-empty

fix-app-dist-embed

mxyng/mlx-compile

mxyng/mlx-quant

mxyng/mlx-glm4.7

mxyng/mlx

brucemacd/simplify-model-picker

jmorganca/qwen3-concurrent

fix-glm-4.7-flash-mla-config

drifkin/qwen3-coder-opening-tag

brucemacd/usage-cli

fix-cuda12-fattn-shmem

ollama-imagegen-docs

parth/fix-multiline-inputs

brucemacd/config-docs

mxyng/model-files

mxyng/simple-execute

fix-imagegen-ollama-models

mxyng/async-upload

jmorganca/lazy-no-dtype-changes

imagegen-auto-detect-create

parth/decrease-concurrent-download-hf

fix-mlx-quantize-init

jmorganca/x-cleanup

usage

imagegen-readme

jmorganca/glm-image

mlx-gpu-cd

jmorganca/imagegen-modelfile

parth/agent-skills

parth/agent-allowlist

parth/signed-in-offline

parth/agents

parth/fix-context-chopping

improve-cloud-flow

parth/add-models-websearch

parth/prompt-renderer-mcp

jmorganca/native-settings

jmorganca/download-stream-hash

jmorganca/client2-rebased

brucemacd/oai-chat-req-multipart

jessegross/multi_chunk_reserve

grace/additional-omit-empty

grace/mistral-3-large

mxyng/tokenizer2

mxyng/tokenizer

jessegross/flash

hoyyeva/windows-nacked-app

mxyng/cleanup-attention

grace/deepseek-parser

hoyyeva/remember-unsent-prompt

parth/add-lfs-pointer-error-conversion

parth/olmo2-test2

hoyyeva/ollama-launchagent-plist

nicole/olmo-model

parth/olmo-test

mxyng/remove-embedded

parth/render-template

jmorganca/intellect-3

parth/remove-prealloc-linter

jmorganca/cmd-eval

nicole/nomic-embed-text-fix

mxyng/lint-2

hoyyeva/add-gemini-3-pro-preview

hoyyeva/load-model-list

mxyng/expand-path

mxyng/environ-2

hoyyeva/deeplink-json-encoding

parth/improve-tool-calling-tests

hoyyeva/conversation

hoyyeva/assistant-edit-response

hoyyeva/thinking

origin/brucemacd/invalid-char-i-err

parth/improve-tool-calling

jmorganca/required-omitempty

grace/qwen3-vl-tests

mxyng/iter-client

parth/docs-readme

nicole/embed-test

pdevine/integration-benchstat

parth/remove-generate-cmd

parth/add-toolcall-id

mxyng/server-tests

jmorganca/glm-4.6

jmorganca/gin-h-compat

drifkin/stable-tool-args

pdevine/qwen3-more-thinking

parth/add-websearch-client

nicole/websearch_local

jmorganca/qwen3-coder-updates

grace/deepseek-v3-migration-tests

mxyng/fix-create

jmorganca/cloud-errors

pdevine/parser-tidy

revert-12233-parth/simplify-entrypoints-runner

parth/enable-so-gpt-oss

brucemacd/qwen3vl

jmorganca/readme-simplify

parth/gpt-oss-structured-outputs

revert-12039-jmorganca/tools-braces

mxyng/embeddings

mxyng/gguf

mxyng/benchmark

mxyng/types-null

parth/move-parsing

mxyng/gemma2

jmorganca/docs

mxyng/16-bit

mxyng/create-stdin

pdevine/authorizedkeys

mxyng/quant

parth/opt-in-error-context-window

brucemacd/cache-models

brucemacd/runner-completion

jmorganca/llama-update-6

brucemacd/benchmark-list

brucemacd/partial-read-caps

parth/deepseek-r1-tools

mxyng/omit-array

parth/tool-prefix-temp

brucemacd/runner-test

jmorganca/qwen25vl

brucemacd/model-forward-test-ext

parth/python-function-parsing

jmorganca/cuda-compression-none

drifkin/num-parallel

drifkin/chat-truncation-fix

jmorganca/sync

parth/python-tools-calling

drifkin/array-head-count

brucemacd/create-no-loop

parth/server-enable-content-stream-with-tools

qwen25omni

mxyng/v3

brucemacd/ropeconfig

jmorganca/silence-tokenizer

parth/sample-so-test

parth/sampling-structured-outputs

brucemacd/doc-go-engine

parth/constrained-sampling-json

jmorganca/mistral-wip

brucemacd/mistral-small-convert

parth/sample-unmarshal-json-for-params

brucemacd/jomorganca/mistral

pdevine/bfloat16

jmorganca/mistral

brucemacd/mistral

pdevine/logging

parth/sample-correctness-fix

parth/sample-fix-sorting

jmorgan/sample-fix-sorting-extras

jmorganca/temp-0-images

brucemacd/parallel-embed-models

brucemacd/shim-grammar

jmorganca/fix-gguf-error

bmizerany/nameswork

jmorganca/faster-releases

bmizerany/validatenames

brucemacd/err-no-vocab

brucemacd/rope-config

brucemacd/err-hint

brucemacd/qwen2_5

brucemacd/logprobs

brucemacd/new_runner_graph_bench

progress-flicker

brucemacd/forward-test

brucemacd/go_qwen2

pdevine/gemma2

jmorganca/add-missing-symlink-eval

mxyng/next-debug

parth/set-context-size-openai

brucemacd/next-bpe-bench

brucemacd/next-bpe-test

brucemacd/new_runner_e2e

brucemacd/new_runner_qwen2

pdevine/convert-cohere2

brucemacd/convert-cli

parth/log-probs

mxyng/next-mlx

mxyng/cmd-history

parth/templating

parth/tokenize-detokenize

brucemacd/check-key-register

bmizerany/grammar

jmorganca/vendor-081b29bd

mxyng/func-checks

jmorganca/fix-null-format

parth/fix-default-to-warn-json

jmorganca/qwen2vl

jmorganca/no-concat

parth/cmd-cleanup-SO

brucemacd/check-key-register-structured-err

parth/openai-stream-usage

parth/fix-referencing-so

stream-tools-stop

jmorganca/degin-1

brucemacd/install-path-clean

brucemacd/push-name-validation

brucemacd/browser-key-register

jmorganca/openai-fix-first-message

jmorganca/fix-proxy

jessegross/sample

parth/disallow-streaming-tools

dhiltgen/remove_submodule

jmorganca/ga

jmorganca/mllama

pdevine/newlines

pdevine/geems-2b

jmorganca/llama-bump

mxyng/modelname-7

mxyng/gin-slog

mxyng/modelname-6

jyan/convert-prog

jyan/quant5

paligemma-support

pdevine/import-docs

jmorganca/openai-context

jyan/paligemma

jyan/p2

jyan/palitest

bmizerany/embedspeedup

jmorganca/llama-vit

brucemacd/allow-ollama

royh/ep-methods

royh/whisper

mxyng/api-models

mxyng/fix-memory

jyan/q4_4/8

jyan/ollama-v

royh/stream-tools

roy-embed-parallel

bmizerany/hrm

revert-5963-revert-5924-mxyng/llama3.1-rope

royh/embed-viz

jyan/local2

jyan/auth

jyan/local

jyan/parse-temp

jmorganca/template-mistral

jyan/reord-g

royh-openai-suffixdocs

royh-imgembed

royh-embed-parallel

jyan/quant4

royh-precision

jyan/progress

pdevine/fix-template

jyan/quant3

pdevine/ggla

mxyng/update-registry-domain

jmorganca/ggml-static

mxyng/create-context

jyan/v0.146

mxyng/layers-from-files

build_dist

bmizerany/noseek

royh-ls

royh-name

timeout

mxyng/server-timestamp

bmizerany/nosillyggufslurps

royh-params

jmorganca/llama-cpp-7c26775

royh-openai-delete

royh-show-rigid

jmorganca/enable-fa

jmorganca/no-error-template

jyan/format

royh-testdelete

bmizerany/fastverify

language_support

pdevine/ps-glitches

brucemacd/tokenize

bruce/iq-quants

bmizerany/filepathwithcoloninhost

mxyng/split-bin

bmizerany/client-registry

jmorganca/if-none-match

native

jmorganca/native

jmorganca/batch-embeddings

jmorganca/initcmake

jmorganca/mm

pdevine/showggmlinfo

modenameenforcealphanum

bmizerany/modenameenforcealphanum

jmorganca/done-reason

jmorganca/llama-cpp-8960fe8

ollama.com

bmizerany/filepathnobuild

bmizerany/types/model/defaultfix

rmdisplaylong

nogogen

bmizerany/x

modelfile-readme

bmizerany/replacecolon

jmorganca/limit

jmorganca/execstack

jmorganca/replace-assets

mxyng/tune-concurrency

jmorganca/testing

whitespace-detection

jmorganca/options

upgrade-all

scratch

cuda-search

mattw/airenamer

mattw/allmodelsonhuggingface

mattw/quantcontext

mattw/whatneedstorun

brucemacd/llama-mem-calc

mattw/faq-context

mattw/communitylinks

mattw/noprune

mattw/python-functioncalling

rename

mxyng/install

pulse

remove-first

editor

mattw/selfqueryingretrieval

cgo

mattw/howtoquant

api

matt/streamingapi

format-config

mxyng/extra-args

shell

update-nous-hermes

cp-model

upload-progress

fix-unknown-model

fix-model-names

delete-fix

insecure-registry

ls

deletemodels

progressbar

readme-updates

license-layers

skip-list

list-models

modelpath

matt/examplemodelfiles

distribution

go-opts

v0.32.1

v0.32.1-rc0

v0.32.0

v0.32.0-rc0

v0.31.2

v0.31.2-rc2

v0.31.2-rc1

v0.31.2-rc0

v0.31.1

v0.31.0

v0.30.12-rc0

v0.30.11

v0.30.11-rc1

v0.30.11-rc0

v0.30.10

v0.30.10-rc1

v0.30.10-rc0

v0.30.9

v0.30.9-rc2

v0.30.9-rc1

v0.30.9-rc0

v0.30.8

v0.30.8-rc0

v0.30.7

v0.30.7-rc1

v0.30.7-rc0

v0.30.6

v0.30.6-rc0

v0.30.5

v0.30.5-rc0

v0.30.4

v0.30.4-rc1

v0.30.4-rc0

v0.30.3

v0.30.2

v0.30.2-rc0

v0.30.1

v0.30.1-rc0

v0.30.0

v0.30.0-rc32

v0.30.0-rc31

v0.30.0-rc30

v0.30.0-rc29

v0.30.0-rc28

v0.30.0-rc27

v0.30.0-rc26

v0.30.0-rc25

v0.30.0-rc24

v0.30.0-rc23

v0.30.0-rc22

v0.30.0-rc21

v0.30.0-rc20

v0.30.0-rc19

v0.30.0-rc18

v0.25.0-rc0

v0.30.0-rc17

v0.30.0-rc16

v0.24.0-rc1

v0.24.0

v0.24.0-rc0

v0.23.4

v0.23.4-rc0

v0.30.0-rc15

v0.23.3

v0.23.3-rc1

v0.30.0-rc14

v0.23.3-rc0

v0.30.0-rc13

v0.30.0-rc12

v0.30.0-rc11

v0.30.0-rc10

v0.30.0-rc9

v0.30.0-rc8

v0.30.0-rc7

v0.30.0-rc6

v0.30.0-rc5

v0.23.2

v0.23.2-rc0

v0.30.0-rc4

v0.30.0-rc3

v0.30.0-rc2

v0.30.0-rc1

v0.30.0-rc0

v0.23.1

v0.23.1-rc0

v0.23.0

v0.23.0-rc0

v0.22.1

v0.22.1-rc1

v0.22.1-rc0

v0.22.0

v0.22.0-rc1

v0.21.3-rc0

v0.21.2-rc1

v0.21.2

v0.21.2-rc0

v0.21.1

v0.21.1-rc1

v0.21.1-rc0

v0.21.0

v0.21.0-rc1

v0.21.0-rc0

v0.20.8-rc0

v0.20.7

v0.20.7-rc1

v0.20.7-rc0

v0.20.6

v0.20.6-rc1

v0.20.6-rc0

v0.20.5

v0.20.5-rc2

v0.20.5-rc1

v0.20.5-rc0

v0.20.4

v0.20.4-rc2

v0.20.4-rc1

v0.20.4-rc0

v0.20.3

v0.20.3-rc0

v0.20.2

v0.20.1

v0.20.1-rc2

v0.20.1-rc1

v0.20.1-rc0

v0.20.0

v0.20.0-rc1

v0.20.0-rc0

v0.19.0

v0.19.0-rc2

v0.19.0-rc1

v0.19.0-rc0

v0.18.4-rc1

v0.18.4-rc0

v0.18.3

v0.18.3-rc2

v0.18.3-rc1

v0.18.3-rc0

v0.18.2

v0.18.2-rc1

v0.18.2-rc0

v0.18.1

v0.18.1-rc1

v0.18.1-rc0

v0.18.0

v0.18.0-rc2

v0.18.0-rc1

v0.18.0-rc0

v0.17.8-rc4

v0.17.8-rc3

v0.17.8-rc2

v0.17.8-rc1

v0.17.8-rc0

v0.17.7

v0.17.7-rc2

v0.17.7-rc1

v0.17.7-rc0

v0.17.6

v0.17.5

v0.17.4

v0.17.3

v0.17.2

v0.17.1

v0.17.1-rc2

v0.17.1-rc1

v0.17.1-rc0

v0.17.0

v0.17.0-rc2

v0.17.0-rc1

v0.17.0-rc0

v0.16.3

v0.16.3-rc2

v0.16.3-rc1

v0.16.3-rc0

v0.16.2

v0.16.2-rc0

v0.16.1

v0.16.0

v0.16.0-rc2

v0.16.0-rc0

v0.16.0-rc1

v0.15.6

v0.15.5

v0.15.5-rc5

v0.15.5-rc4

v0.15.5-rc3

v0.15.5-rc2

v0.15.5-rc1

v0.15.5-rc0

v0.15.4

v0.15.3

v0.15.2

v0.15.1

v0.15.1-rc1

v0.15.1-rc0

v0.15.0-rc6

v0.15.0

v0.15.0-rc5

v0.15.0-rc4

v0.15.0-rc3

v0.15.0-rc2

v0.15.0-rc1

v0.15.0-rc0

v0.14.3

v0.14.3-rc3

v0.14.3-rc2

v0.14.3-rc1

v0.14.3-rc0

v0.14.2

v0.14.2-rc1

v0.14.2-rc0

v0.14.1

v0.14.0-rc11

v0.14.0

v0.14.0-rc10

v0.14.0-rc9

v0.14.0-rc8

v0.14.0-rc7

v0.14.0-rc6

v0.14.0-rc5

v0.14.0-rc4

v0.14.0-rc3

v0.14.0-rc2

v0.14.0-rc1

v0.14.0-rc0

v0.13.5

v0.13.5-rc1

v0.13.5-rc0

v0.13.4-rc2

v0.13.4

v0.13.4-rc1

v0.13.4-rc0

v0.13.3

v0.13.3-rc1

v0.13.3-rc0

v0.13.2

v0.13.2-rc2

v0.13.2-rc1

v0.13.2-rc0

v0.13.1

v0.13.1-rc2

v0.13.1-rc1

v0.13.1-rc0

v0.13.0

v0.13.0-rc0

v0.12.11

v0.12.11-rc1

v0.12.11-rc0

v0.12.10

v0.12.10-rc1

v0.12.10-rc0

v0.12.9-rc0

v0.12.9

v0.12.8

v0.12.8-rc0

v0.12.7

v0.12.7-rc1

v0.12.7-rc0

v0.12.7-citest0

v0.12.6

v0.12.6-rc1

v0.12.6-rc0

v0.12.5

v0.12.5-rc0

v0.12.4

v0.12.4-rc7

v0.12.4-rc6

v0.12.4-rc5

v0.12.4-rc4

v0.12.4-rc3

v0.12.4-rc2

v0.12.4-rc1

v0.12.4-rc0

v0.12.3

v0.12.2

v0.12.2-rc0

v0.12.1

v0.12.1-rc1

v0.12.1-rc2

v0.12.1-rc0

v0.12.0

v0.12.0-rc1

v0.12.0-rc0

v0.11.11

v0.11.11-rc3

v0.11.11-rc2

v0.11.11-rc1

v0.11.11-rc0

v0.11.10

v0.11.9

v0.11.9-rc0

v0.11.8

v0.11.8-rc0

v0.11.7-rc1

v0.11.7-rc0

v0.11.7

v0.11.6

v0.11.6-rc0

v0.11.5-rc4

v0.11.5-rc3

v0.11.5

v0.11.5-rc5

v0.11.5-rc2

v0.11.5-rc1

v0.11.5-rc0

v0.11.4

v0.11.4-rc0

v0.11.3

v0.11.3-rc0

v0.11.2

v0.11.1

v0.11.0-rc0

v0.11.0-rc1

v0.11.0-rc2

v0.11.0

v0.10.2-int1

v0.10.1

v0.10.0

v0.10.0-rc4

v0.10.0-rc3

v0.10.0-rc2

v0.10.0-rc1

v0.10.0-rc0

v0.9.7-rc1

v0.9.7-rc0

v0.9.6

v0.9.6-rc0

v0.9.6-ci0

v0.9.5

v0.9.4-rc5

v0.9.4-rc6

v0.9.4

v0.9.4-rc3

v0.9.4-rc4

v0.9.4-rc1

v0.9.4-rc2

v0.9.4-rc0

v0.9.3

v0.9.3-rc5

v0.9.4-citest0

v0.9.3-rc4

v0.9.3-rc3

v0.9.3-rc2

v0.9.3-rc1

v0.9.3-rc0

v0.9.2

v0.9.1

v0.9.1-rc1

v0.9.1-rc0

v0.9.1-ci1

v0.9.1-ci0

v0.9.0

v0.9.0-rc0

v0.8.0

v0.8.0-rc0

v0.7.1-rc2

v0.7.1

v0.7.1-rc1

v0.7.1-rc0

v0.7.0

v0.7.0-rc1

v0.7.0-rc0

v0.6.9-rc0

v0.6.8

v0.6.8-rc0

v0.6.7

v0.6.7-rc2

v0.6.7-rc1

v0.6.7-rc0

v0.6.6

v0.6.6-rc2

v0.6.6-rc1

v0.6.6-rc0

v0.6.5-rc1

v0.6.5

v0.6.5-rc0

v0.6.4-rc0

v0.6.4

v0.6.3-rc1

v0.6.3

v0.6.3-rc0

v0.6.2

v0.6.2-rc0

v0.6.1

v0.6.1-rc0

v0.6.0-rc0

v0.6.0

v0.5.14-rc0

v0.5.13

v0.5.13-rc6

v0.5.13-rc5

v0.5.13-rc4

v0.5.13-rc3

v0.5.13-rc2

v0.5.13-rc1

v0.5.13-rc0

v0.5.12

v0.5.12-rc1

v0.5.12-rc0

v0.5.11

v0.5.10

v0.5.9

v0.5.9-rc0

v0.5.8-rc13

v0.5.8

v0.5.8-rc12

v0.5.8-rc11

v0.5.8-rc10

v0.5.8-rc9

v0.5.8-rc8

v0.5.8-rc7

v0.5.8-rc6

v0.5.8-rc5

v0.5.8-rc4

v0.5.8-rc3

v0.5.8-rc2

v0.5.8-rc1

v0.5.8-rc0

v0.5.7

v0.5.6

v0.5.5

v0.5.5-rc0

v0.5.4

v0.5.3

v0.5.3-rc0

v0.5.2

v0.5.2-rc3

v0.5.2-rc2

v0.5.2-rc1

v0.5.2-rc0

v0.5.1

v0.5.0

v0.5.0-rc1

v0.4.8-rc0

v0.4.7

v0.4.6

v0.4.5

v0.4.4

v0.4.3

v0.4.3-rc0

v0.4.2

v0.4.2-rc1

v0.4.2-rc0

v0.4.1

v0.4.1-rc0

v0.4.0

v0.4.0-rc8

v0.4.0-rc7

v0.4.0-rc6

v0.4.0-rc5

v0.4.0-rc4

v0.4.0-rc3

v0.4.0-rc2

v0.4.0-rc1

v0.4.0-rc0

v0.4.0-ci3

v0.3.14

v0.3.14-rc0

v0.3.13

v0.3.12

v0.3.12-rc5

v0.3.12-rc4

v0.3.12-rc3

v0.3.12-rc2

v0.3.12-rc1

v0.3.11

v0.3.11-rc4

v0.3.11-rc3

v0.3.11-rc2

v0.3.11-rc1

v0.3.10

v0.3.10-rc1

v0.3.9

v0.3.8

v0.3.7

v0.3.7-rc6

v0.3.7-rc5

v0.3.7-rc4

v0.3.7-rc3

v0.3.7-rc2

v0.3.7-rc1

v0.3.6

v0.3.5

v0.3.4

v0.3.3

v0.3.2

v0.3.1

v0.3.0

v0.2.8

v0.2.8-rc2

v0.2.8-rc1

v0.2.7

v0.2.6

v0.2.5

v0.2.4

v0.2.3

v0.2.2

v0.2.2-rc2

v0.2.2-rc1

v0.2.1

v0.2.0

v0.1.49-rc14

v0.1.49-rc13

v0.1.49-rc12

v0.1.49-rc11

v0.1.49-rc10

v0.1.49-rc9

v0.1.49-rc8

v0.1.49-rc7

v0.1.49-rc6

v0.1.49-rc4

v0.1.49-rc5

v0.1.49-rc3

v0.1.49-rc2

v0.1.49-rc1

v0.1.48

v0.1.47

v0.1.46

v0.1.45-rc5

v0.1.45

v0.1.45-rc4

v0.1.45-rc3

v0.1.45-rc2

v0.1.45-rc1

v0.1.44

v0.1.43

v0.1.42

v0.1.41

v0.1.40

v0.1.40-rc1

v0.1.39

v0.1.39-rc2

v0.1.39-rc1

v0.1.38

v0.1.37

v0.1.36

v0.1.35

v0.1.35-rc1

v0.1.34

v0.1.34-rc1

v0.1.33

v0.1.33-rc7

v0.1.33-rc6

v0.1.33-rc5

v0.1.33-rc4

v0.1.33-rc3

v0.1.33-rc2

v0.1.33-rc1

v0.1.32

v0.1.32-rc2

v0.1.32-rc1

v0.1.31

v0.1.30

v0.1.29

v0.1.28

v0.1.27

v0.1.26

v0.1.25

v0.1.24

v0.1.23

v0.1.22

v0.1.21

v0.1.20

v0.1.19

v0.1.18

v0.1.17

v0.1.16

v0.1.15

v0.1.14

v0.1.13

v0.1.12

v0.1.11

v0.1.10

v0.1.9

v0.1.8

v0.1.7

v0.1.6

v0.1.5

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

v0.0.21

v0.0.20

v0.0.19

v0.0.18

v0.0.17

v0.0.16

v0.0.15

v0.0.14

v0.0.13

v0.0.12

v0.0.11

v0.0.10

v0.0.9

v0.0.8

v0.0.7

v0.0.6

v0.0.5

v0.0.4

v0.0.3

v0.0.2

v0.0.1

Labels

Clear labels

amd

api

app

bug

build

cli

cloud

compatibility

context-length

create

docker

documentation

embeddings

feature request

feedback wanted

good first issue

gpt-oss

gpu

harmony

help wanted

image

install

intel

js

launch

linux

macos

memory

mlx

model

needs more info

networking

nvidia

ollama.com

performance

pull-request

Mirrored from GitHub Pull Request

python

question

registry

rendering

thinking

tools

top

vulkan

windows

wsl

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

GiteaMirror ninjasurge

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/ollama#52299