github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 12:48:05 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #8348] API Hangs #5353

New Issue
Closed
opened 2026-04-12 16:33:31 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-04-12 16:33:31 -05:00
Owner
Copy Link

Originally created by @ergosumdre on GitHub (Jan 8, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/8348

What is the issue?

When running augmentoolkit to build a synthetic dataset, Ollama seems to hang, more specifically when CONCURRENCY_LIMIT is greater than 2.

In the server logs, I notice these lines before the GPU memory dumps:

llama_decode: failed to decode, ret = 1
panic: failed to decode batch: could not find a kv cache slot

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.5.4

Originally created by @ergosumdre on GitHub (Jan 8, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/8348 ### What is the issue? When running [augmentoolkit](https://github.com/e-p-armstrong/augmentoolkit) to build a synthetic dataset, Ollama seems to hang, more specifically when CONCURRENCY_LIMIT is greater than 2. In the server logs, I notice these lines before the GPU memory dumps: > llama_decode: failed to decode, ret = 1 > panic: failed to decode batch: could not find a kv cache slot ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.5.4
GiteaMirror added the bug label 2026-04-12 16:33:31 -05:00
GiteaMirror commented 2026-04-12 16:33:33 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 8, 2025):

Post full log.

<!-- gh-comment-id:2578854809 --> @rick-github commented on GitHub (Jan 8, 2025): Post full log.
GiteaMirror commented 2026-04-12 16:33:33 -05:00
Author
Owner
Copy Link

@ergosumdre commented on GitHub (Jan 8, 2025):

Sure thing,

Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: format = GGUF V3 (latest)
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: arch = granitemoe
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: vocab type = BPE
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: n_vocab = 49155
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: n_merges = 48891
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: vocab_only = 1
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: model type = ?B
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: model ftype = all F32
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: model params = 3.30 B
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: model size = 3.27 GiB (8.51 BPW)
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: general.name = Granite 3.1 3b A800M Instruct
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: BOS token = 0 '<|end_of_text|>'
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: EOS token = 0 '<|end_of_text|>'
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: UNK token = 0 '<|end_of_text|>'
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: PAD token = 0 '<|end_of_text|>'
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: LF token = 145 'Ä'
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: EOG token = 0 '<|end_of_text|>'
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: max token length = 512
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: f_embedding_scale = 0.000000
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: f_residual_scale = 0.000000
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: f_attention_scale = 0.000000
Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llama_model_load: vocab only - skipping tensors
Jan 08 11:28:40 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:40 | 200 | 20.453120629s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:42 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:42 | 200 | 22.55548243s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:45 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:45 | 200 | 25.461083956s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: llama_decode: failed to decode, ret = 1
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: llama_decode: failed to decode, ret = 1
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: panic: failed to decode batch: could not find a kv cache slot
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: goroutine 4 [running]:
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: github.com/ollama/ollama/llama/runner.(Server).run(0xc00018e000, {0x6515318b3de0, 0xc00018c050})
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: github.com/ollama/ollama/llama/runner/runner.go:344 +0x22e
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: github.com/ollama/ollama/llama/runner/runner.go:984 +0xde5
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.375752276s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.375146669s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.376178023s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.376436215s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.375838624s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.376042941s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.370867586s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.359274203s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.33982487s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.32894248s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.324757094s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.312808787s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.30587018s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.304609804s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 5.825517133s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 3.738662321s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 837.348699ms | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.426482449s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:02 dre-slimserver-3t systemd[1]: Stopping Ollama Service...
Jan 08 11:29:03 dre-slimserver-3t systemd[1]: ollama.service: Deactivated successfully.
Jan 08 11:29:03 dre-slimserver-3t systemd[1]: Stopped Ollama Service.
Jan 08 11:29:03 dre-slimserver-3t systemd[1]: ollama.service: Consumed 30.121s CPU time.
Jan 08 11:29:03 dre-slimserver-3t systemd[1]: Started Ollama Service.
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: 2025/01/08 11:29:03 routes.go:1259: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://100.100.70.109:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:10m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost: https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:03.583-06:00 level=INFO source=images.go:757 msg="total blobs: 194"
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:03.589-06:00 level=INFO source=images.go:764 msg="total unused blobs removed: 0"
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: - using env: export GIN_MODE=release
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: - using code: gin.SetMode(gin.ReleaseMode)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/pull --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/generate --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/chat --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/embed --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/embeddings --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/create --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/push --> github.com/ollama/ollama/server.(*Server).PushHandler-fm (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/copy --> github.com/ollama/ollama/server.(*Server).CopyHandler-fm (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] DELETE /api/delete --> github.com/ollama/ollama/server.(*Server).DeleteHandler-fm (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/show --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] HEAD /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] GET /api/ps --> github.com/ollama/ollama/server.(*Server).PsHandler-fm (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /v1/chat/completions --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /v1/completions --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (6 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /v1/embeddings --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (6 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] GET /v1/models --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (6 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] GET /v1/models/:model --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (6 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] GET / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] GET /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] GET /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] HEAD / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] HEAD /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] HEAD /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:03.590-06:00 level=INFO source=routes.go:1310 msg="Listening on 100.100.70.109:11434 (version 0.5.4)"
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:03.591-06:00 level=INFO source=routes.go:1339 msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2 cuda_v11_avx cuda_v12_avx rocm_avx]"
Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:03.591-06:00 level=INFO source=gpu.go:226 msg="looking for compatible GPUs"
Jan 08 11:29:04 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:04.273-06:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-3b28212c-2887-4f2a-6f4b-65a53d0f55ad library=cuda variant=v12 compute=7.0 driver=12.2 name="Tesla V100-SXM2-16GB" total="15.8 GiB" available="15.1 GiB"
Jan 08 11:29:04 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:04.273-06:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-a9fe4c2f-a87c-c645-82ef-2da178797d63 library=cuda variant=v12 compute=7.0 driver=12.2 name="Tesla V100-SXM2-16GB" total="15.8 GiB" available="15.5 GiB"
Jan 08 11:29:04 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:04.273-06:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-8e52c368-ae0c-66f4-0f9f-d9b1b5682d23 library=cuda variant=v12 compute=7.0 driver=12.2 name="Tesla V100-SXM2-16GB" total="15.8 GiB" available="15.5 GiB"
Jan 08 11:29:04 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:04.273-06:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-bb673042-8563-32d8-a436-fddbe6b0de03 library=cuda variant=v12 compute=7.0 driver=12.2 name="Tesla V100-SXM2-16GB" total="15.8 GiB" available="15.5 GiB"
Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.091-06:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0 gpu=GPU-a9fe4c2f-a87c-c645-82ef-2da178797d63 parallel=4 available=16606167040 required="4.6 GiB"
Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.858-06:00 level=INFO source=server.go:104 msg="system memory" total="125.7 GiB" free="121.1 GiB" free_swap="2.0 GiB"
Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.858-06:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[15.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="4.6 GiB" memory.required.partial="4.6 GiB" memory.required.kv="512.0 MiB" memory.required.allocations="[4.6 GiB]" memory.weights.total="3.7 GiB" memory.weights.repeating="3.6 GiB" memory.weights.nonrepeating="76.5 MiB" memory.graph.full="256.0 MiB" memory.graph.partial="256.0 MiB"
Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.859-06:00 level=INFO source=server.go:376 msg="starting llama server" cmd="/usr/local/lib/ollama/runners/cuda_v12_avx/ollama_llama_server runner --model /usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0 --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --threads 10 --parallel 4 --port 34001"
Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.859-06:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.859-06:00 level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.860-06:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.908-06:00 level=INFO source=runner.go:945 msg="starting go runner"
Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: ggml_cuda_init: found 1 CUDA devices:
Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: Device 0: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes
Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.926-06:00 level=INFO source=runner.go:946 msg=system info="CUDA : ARCHS = 600,610,620,700,720,750,800,860,870,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=10
Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.926-06:00 level=INFO source=.:0 msg="Server listening on 127.0.0.1:34001"
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_load_model_from_file: using device CUDA0 (Tesla V100-SXM2-16GB) - 15836 MiB free
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: loaded meta data with 42 key-value pairs and 322 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0 (version GGUF V3 (latest))
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 0: general.architecture str = granitemoe
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 1: general.type str = model
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 2: general.name str = Granite 3.1 3b A800M Instruct
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 3: general.finetune str = instruct
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 4: general.basename str = granite-3.1
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 5: general.size_label str = 3B-a800M
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 6: general.license str = apache-2.0
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 7: general.base_model.count u32 = 1
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 8: general.base_model.0.name str = Granite 3.1 3b A800M Base
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 9: general.base_model.0.organization str = Ibm Granite
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 10: general.base_model.0.repo_url str = https://huggingface.co/ibm-granite/gr...
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 11: general.tags arr[str,3] = ["language", "granite-3.1", "text-gen...
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 12: granitemoe.block_count u32 = 32
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 13: granitemoe.context_length u32 = 131072
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 14: granitemoe.embedding_length u32 = 1536
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 15: granitemoe.feed_forward_length u32 = 512
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 16: granitemoe.attention.head_count u32 = 24
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 17: granitemoe.attention.head_count_kv u32 = 8
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 18: granitemoe.rope.freq_base f32 = 10000000.000000
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 19: granitemoe.attention.layer_norm_rms_epsilon f32 = 0.000001
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 20: granitemoe.expert_count u32 = 40
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 21: granitemoe.expert_used_count u32 = 8
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 22: general.file_type u32 = 7
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 23: granitemoe.vocab_size u32 = 49155
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 24: granitemoe.rope.dimension_count u32 = 64
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 25: granitemoe.attention.scale f32 = 0.015625
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 26: granitemoe.embedding_scale f32 = 12.000000
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 27: granitemoe.residual_scale f32 = 0.220000
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 28: granitemoe.logit_scale f32 = 6.000000
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 29: tokenizer.ggml.model str = gpt2
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 30: tokenizer.ggml.pre str = refact
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 31: tokenizer.ggml.tokens arr[str,49155] = ["<|end_of_text|>", "<fim_prefix>", "...
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 32: tokenizer.ggml.token_type arr[i32,49155] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:06.111-06:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server loading model"
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 33: tokenizer.ggml.merges arr[str,48891] = ["Ġ Ġ", "ĠĠ ĠĠ", "ĠĠĠĠ ĠĠ...
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 34: tokenizer.ggml.bos_token_id u32 = 0
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 35: tokenizer.ggml.eos_token_id u32 = 0
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 36: tokenizer.ggml.unknown_token_id u32 = 0
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 37: tokenizer.ggml.padding_token_id u32 = 0
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 38: tokenizer.ggml.add_bos_token bool = false
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 39: tokenizer.chat_template str = {%- if messages[0]['role'] == 'system...
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 40: tokenizer.ggml.add_space_prefix bool = false
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 41: general.quantization_version u32 = 2
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - type f32: 97 tensors
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - type q8_0: 225 tensors
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_vocab: special tokens cache size = 22
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_vocab: token to piece cache size = 0.2826 MB
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: format = GGUF V3 (latest)
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: arch = granitemoe
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: vocab type = BPE
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_vocab = 49155
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_merges = 48891
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: vocab_only = 0
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_ctx_train = 131072
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_embd = 1536
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_layer = 32
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_head = 24
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_head_kv = 8
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_rot = 64
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_swa = 0
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_embd_head_k = 64
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_embd_head_v = 64
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_gqa = 3
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_embd_k_gqa = 512
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_embd_v_gqa = 512
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_norm_eps = 0.0e+00
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_norm_rms_eps = 1.0e-06
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_clamp_kqv = 0.0e+00
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_logit_scale = 6.0e+00
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_ff = 512
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_expert = 40
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_expert_used = 8
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: causal attn = 1
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: pooling type = 0
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: rope type = 0
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: rope scaling = linear
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: freq_base_train = 10000000.0
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: freq_scale_train = 1
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_ctx_orig_yarn = 131072
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: rope_finetuned = unknown
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: ssm_d_conv = 0
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: ssm_d_inner = 0
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: ssm_d_state = 0
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: ssm_dt_rank = 0
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: ssm_dt_b_c_rms = 0
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: model type = 3B
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: model ftype = Q8_0
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: model params = 3.30 B
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: model size = 3.27 GiB (8.51 BPW)
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: general.name = Granite 3.1 3b A800M Instruct
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: BOS token = 0 '<|end_of_text|>'
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: EOS token = 0 '<|end_of_text|>'
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: UNK token = 0 '<|end_of_text|>'
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: PAD token = 0 '<|end_of_text|>'
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: LF token = 145 'Ä'
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: EOG token = 0 '<|end_of_text|>'
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: max token length = 512
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_embedding_scale = 12.000000
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_residual_scale = 0.220000
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_attention_scale = 0.015625
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_tensors: offloading 32 repeating layers to GPU
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_tensors: offloading output layer to GPU
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_tensors: offloaded 33/33 layers to GPU
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_tensors: CPU_Mapped model buffer size = 76.50 MiB
Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_tensors: CUDA0 model buffer size = 3348.39 MiB
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: n_seq_max = 4
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: n_ctx = 8192
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: n_ctx_per_seq = 2048
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: n_batch = 2048
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: n_ubatch = 512
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: flash_attn = 0
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: freq_base = 10000000.0
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: freq_scale = 1
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: n_ctx_per_seq (2048) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_kv_cache_init: CUDA0 KV buffer size = 512.00 MiB
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: KV self size = 512.00 MiB, K (f16): 256.00 MiB, V (f16): 256.00 MiB
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: CUDA_Host output buffer size = 0.77 MiB
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: CUDA0 compute buffer size = 412.00 MiB
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: CUDA_Host compute buffer size = 19.01 MiB
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: graph nodes = 1960
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: graph splits = 2
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:07.367-06:00 level=INFO source=server.go:594 msg="llama runner started in 1.51 seconds"
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: loaded meta data with 42 key-value pairs and 322 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0 (version GGUF V3 (latest))
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 0: general.architecture str = granitemoe
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 1: general.type str = model
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 2: general.name str = Granite 3.1 3b A800M Instruct
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 3: general.finetune str = instruct
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 4: general.basename str = granite-3.1
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 5: general.size_label str = 3B-a800M
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 6: general.license str = apache-2.0
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 7: general.base_model.count u32 = 1
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 8: general.base_model.0.name str = Granite 3.1 3b A800M Base
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 9: general.base_model.0.organization str = Ibm Granite
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 10: general.base_model.0.repo_url str = https://huggingface.co/ibm-granite/gr...
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 11: general.tags arr[str,3] = ["language", "granite-3.1", "text-gen...
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 12: granitemoe.block_count u32 = 32
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 13: granitemoe.context_length u32 = 131072
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 14: granitemoe.embedding_length u32 = 1536
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 15: granitemoe.feed_forward_length u32 = 512
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 16: granitemoe.attention.head_count u32 = 24
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 17: granitemoe.attention.head_count_kv u32 = 8
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 18: granitemoe.rope.freq_base f32 = 10000000.000000
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 19: granitemoe.attention.layer_norm_rms_epsilon f32 = 0.000001
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 20: granitemoe.expert_count u32 = 40
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 21: granitemoe.expert_used_count u32 = 8
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 22: general.file_type u32 = 7
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 23: granitemoe.vocab_size u32 = 49155
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 24: granitemoe.rope.dimension_count u32 = 64
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 25: granitemoe.attention.scale f32 = 0.015625
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 26: granitemoe.embedding_scale f32 = 12.000000
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 27: granitemoe.residual_scale f32 = 0.220000
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 28: granitemoe.logit_scale f32 = 6.000000
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 29: tokenizer.ggml.model str = gpt2
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 30: tokenizer.ggml.pre str = refact
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 31: tokenizer.ggml.tokens arr[str,49155] = ["<|end_of_text|>", "<fim_prefix>", "...
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 32: tokenizer.ggml.token_type arr[i32,49155] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 33: tokenizer.ggml.merges arr[str,48891] = ["Ġ Ġ", "ĠĠ ĠĠ", "ĠĠĠĠ ĠĠ...
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 34: tokenizer.ggml.bos_token_id u32 = 0
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 35: tokenizer.ggml.eos_token_id u32 = 0
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 36: tokenizer.ggml.unknown_token_id u32 = 0
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 37: tokenizer.ggml.padding_token_id u32 = 0
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 38: tokenizer.ggml.add_bos_token bool = false
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 39: tokenizer.chat_template str = {%- if messages[0]['role'] == 'system...
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 40: tokenizer.ggml.add_space_prefix bool = false
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 41: general.quantization_version u32 = 2
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - type f32: 97 tensors
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - type q8_0: 225 tensors
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_vocab: special tokens cache size = 22
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_vocab: token to piece cache size = 0.2826 MB
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: format = GGUF V3 (latest)
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: arch = granitemoe
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: vocab type = BPE
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_vocab = 49155
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_merges = 48891
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: vocab_only = 1
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: model type = ?B
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: model ftype = all F32
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: model params = 3.30 B
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: model size = 3.27 GiB (8.51 BPW)
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: general.name = Granite 3.1 3b A800M Instruct
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: BOS token = 0 '<|end_of_text|>'
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: EOS token = 0 '<|end_of_text|>'
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: UNK token = 0 '<|end_of_text|>'
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: PAD token = 0 '<|end_of_text|>'
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: LF token = 145 'Ä'
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: EOG token = 0 '<|end_of_text|>'
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: max token length = 512
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_embedding_scale = 0.000000
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_residual_scale = 0.000000
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_attention_scale = 0.000000
Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_load: vocab only - skipping tensors
Jan 08 11:29:26 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:26 | 200 | 22.168833156s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:27 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:27 | 200 | 23.007174082s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:29 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:29 | 200 | 25.488162066s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: llama_decode: failed to decode, ret = 1
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: llama_decode: failed to decode, ret = 1
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: panic: failed to decode batch: could not find a kv cache slot
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: goroutine 4 [running]:
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: github.com/ollama/ollama/llama/runner.(*Server).run(0xc000190000, {0x5cf0c0a40de0, 0xc00018e050})
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: github.com/ollama/ollama/llama/runner/runner.go:344 +0x22e
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: github.com/ollama/ollama/llama/runner/runner.go:984 +0xde5
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.388184547s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.388918653s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.388476131s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.390067661s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.389589364s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.389713531s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.389941662s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.388112951s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.390710952s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.389109081s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.391067377s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.390580576s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.39114927s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.373548744s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 4.14116083s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 3.259212673s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 822.460608ms | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.428474109s | 100.121.189.48 | POST "/v1/chat/completions"
Jan 08 11:37:25 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:25.999-06:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.325324089 model=/usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0
Jan 08 11:37:26 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:26.774-06:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=6.099783887 model=/usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0
Jan 08 11:37:27 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:27.412-06:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=6.7380075569999995 model=/usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0
Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:28.201-06:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0 gpu=GPU-a9fe4c2f-a87c-c645-82ef-2da178797d63 parallel=4 available=16606167040 required="4.6 GiB"
Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:28.833-06:00 level=INFO source=server.go:104 msg="system memory" total="125.7 GiB" free="121.0 GiB" free_swap="2.0 GiB"
Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:28.834-06:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[15.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="4.6 GiB" memory.required.partial="4.6 GiB" memory.required.kv="512.0 MiB" memory.required.allocations="[4.6 GiB]" memory.weights.total="3.7 GiB" memory.weights.repeating="3.6 GiB" memory.weights.nonrepeating="76.5 MiB" memory.graph.full="256.0 MiB" memory.graph.partial="256.0 MiB"
Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:28.834-06:00 level=INFO source=server.go:376 msg="starting llama server" cmd="/usr/local/lib/ollama/runners/cuda_v12_avx/ollama_llama_server runner --model /usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0 --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --threads 10 --parallel 4 --port 32861"
Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:28.835-06:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:28.835-06:00 level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:28.835-06:00 level=WARN source=server.go:562 msg="client connection closed before server finished loading, aborting load"
Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:28.835-06:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="timed out waiting for llama runner to start: context canceled"
Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:37:28 | 499 | 7m57s | 100.121.189.48 | POST "/v1/chat/completions"

<!-- gh-comment-id:2578860955 --> @ergosumdre commented on GitHub (Jan 8, 2025): Sure thing, Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: format = GGUF V3 (latest) Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: arch = granitemoe Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: vocab type = BPE Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: n_vocab = 49155 Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: n_merges = 48891 Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: vocab_only = 1 Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: model type = ?B Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: model ftype = all F32 Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: model params = 3.30 B Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: model size = 3.27 GiB (8.51 BPW) Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: general.name = Granite 3.1 3b A800M Instruct Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: BOS token = 0 '<|end_of_text|>' Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: EOS token = 0 '<|end_of_text|>' Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: UNK token = 0 '<|end_of_text|>' Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: PAD token = 0 '<|end_of_text|>' Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: LF token = 145 'Ä' Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: EOG token = 0 '<|end_of_text|>' Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: max token length = 512 Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: f_embedding_scale = 0.000000 Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: f_residual_scale = 0.000000 Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llm_load_print_meta: f_attention_scale = 0.000000 Jan 08 11:28:23 dre-slimserver-3t ollama[270115]: llama_model_load: vocab only - skipping tensors Jan 08 11:28:40 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:40 | 200 | 20.453120629s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:42 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:42 | 200 | 22.55548243s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:45 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:45 | 200 | 25.461083956s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: llama_decode: failed to decode, ret = 1 Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: llama_decode: failed to decode, ret = 1 Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: panic: failed to decode batch: could not find a kv cache slot Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: goroutine 4 [running]: Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: github.com/ollama/ollama/llama/runner.(*Server).run(0xc00018e000, {0x6515318b3de0, 0xc00018c050}) Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: github.com/ollama/ollama/llama/runner/runner.go:344 +0x22e Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1 Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: github.com/ollama/ollama/llama/runner/runner.go:984 +0xde5 Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.375752276s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.375146669s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.376178023s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.376436215s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.375838624s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.376042941s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.370867586s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.359274203s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.33982487s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.32894248s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.324757094s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.312808787s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.30587018s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.304609804s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 5.825517133s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 3.738662321s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 837.348699ms | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:28:46 dre-slimserver-3t ollama[270115]: [GIN] 2025/01/08 - 11:28:46 | 200 | 26.426482449s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:02 dre-slimserver-3t systemd[1]: Stopping Ollama Service... Jan 08 11:29:03 dre-slimserver-3t systemd[1]: ollama.service: Deactivated successfully. Jan 08 11:29:03 dre-slimserver-3t systemd[1]: Stopped Ollama Service. Jan 08 11:29:03 dre-slimserver-3t systemd[1]: ollama.service: Consumed 30.121s CPU time. Jan 08 11:29:03 dre-slimserver-3t systemd[1]: Started Ollama Service. Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: 2025/01/08 11:29:03 routes.go:1259: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://100.100.70.109:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:10m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:03.583-06:00 level=INFO source=images.go:757 msg="total blobs: 194" Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:03.589-06:00 level=INFO source=images.go:764 msg="total unused blobs removed: 0" Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached. Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production. Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: - using env: export GIN_MODE=release Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: - using code: gin.SetMode(gin.ReleaseMode) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/pull --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/generate --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/chat --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/embed --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/embeddings --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/create --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/push --> github.com/ollama/ollama/server.(*Server).PushHandler-fm (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/copy --> github.com/ollama/ollama/server.(*Server).CopyHandler-fm (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] DELETE /api/delete --> github.com/ollama/ollama/server.(*Server).DeleteHandler-fm (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/show --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] HEAD /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] GET /api/ps --> github.com/ollama/ollama/server.(*Server).PsHandler-fm (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /v1/chat/completions --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /v1/completions --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (6 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] POST /v1/embeddings --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (6 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] GET /v1/models --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (6 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] GET /v1/models/:model --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (6 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] GET / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] GET /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] GET /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] HEAD / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] HEAD /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: [GIN-debug] HEAD /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers) Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:03.590-06:00 level=INFO source=routes.go:1310 msg="Listening on 100.100.70.109:11434 (version 0.5.4)" Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:03.591-06:00 level=INFO source=routes.go:1339 msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2 cuda_v11_avx cuda_v12_avx rocm_avx]" Jan 08 11:29:03 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:03.591-06:00 level=INFO source=gpu.go:226 msg="looking for compatible GPUs" Jan 08 11:29:04 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:04.273-06:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-3b28212c-2887-4f2a-6f4b-65a53d0f55ad library=cuda variant=v12 compute=7.0 driver=12.2 name="Tesla V100-SXM2-16GB" total="15.8 GiB" available="15.1 GiB" Jan 08 11:29:04 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:04.273-06:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-a9fe4c2f-a87c-c645-82ef-2da178797d63 library=cuda variant=v12 compute=7.0 driver=12.2 name="Tesla V100-SXM2-16GB" total="15.8 GiB" available="15.5 GiB" Jan 08 11:29:04 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:04.273-06:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-8e52c368-ae0c-66f4-0f9f-d9b1b5682d23 library=cuda variant=v12 compute=7.0 driver=12.2 name="Tesla V100-SXM2-16GB" total="15.8 GiB" available="15.5 GiB" Jan 08 11:29:04 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:04.273-06:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-bb673042-8563-32d8-a436-fddbe6b0de03 library=cuda variant=v12 compute=7.0 driver=12.2 name="Tesla V100-SXM2-16GB" total="15.8 GiB" available="15.5 GiB" Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.091-06:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0 gpu=GPU-a9fe4c2f-a87c-c645-82ef-2da178797d63 parallel=4 available=16606167040 required="4.6 GiB" Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.858-06:00 level=INFO source=server.go:104 msg="system memory" total="125.7 GiB" free="121.1 GiB" free_swap="2.0 GiB" Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.858-06:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[15.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="4.6 GiB" memory.required.partial="4.6 GiB" memory.required.kv="512.0 MiB" memory.required.allocations="[4.6 GiB]" memory.weights.total="3.7 GiB" memory.weights.repeating="3.6 GiB" memory.weights.nonrepeating="76.5 MiB" memory.graph.full="256.0 MiB" memory.graph.partial="256.0 MiB" Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.859-06:00 level=INFO source=server.go:376 msg="starting llama server" cmd="/usr/local/lib/ollama/runners/cuda_v12_avx/ollama_llama_server runner --model /usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0 --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --threads 10 --parallel 4 --port 34001" Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.859-06:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.859-06:00 level=INFO source=server.go:555 msg="waiting for llama runner to start responding" Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.860-06:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error" Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.908-06:00 level=INFO source=runner.go:945 msg="starting go runner" Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: ggml_cuda_init: found 1 CUDA devices: Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: Device 0: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.926-06:00 level=INFO source=runner.go:946 msg=system info="CUDA : ARCHS = 600,610,620,700,720,750,800,860,870,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=10 Jan 08 11:29:05 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:05.926-06:00 level=INFO source=.:0 msg="Server listening on 127.0.0.1:34001" Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_load_model_from_file: using device CUDA0 (Tesla V100-SXM2-16GB) - 15836 MiB free Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: loaded meta data with 42 key-value pairs and 322 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0 (version GGUF V3 (latest)) Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 0: general.architecture str = granitemoe Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 1: general.type str = model Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 2: general.name str = Granite 3.1 3b A800M Instruct Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 3: general.finetune str = instruct Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 4: general.basename str = granite-3.1 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 5: general.size_label str = 3B-a800M Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 6: general.license str = apache-2.0 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 7: general.base_model.count u32 = 1 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 8: general.base_model.0.name str = Granite 3.1 3b A800M Base Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 9: general.base_model.0.organization str = Ibm Granite Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 10: general.base_model.0.repo_url str = https://huggingface.co/ibm-granite/gr... Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 11: general.tags arr[str,3] = ["language", "granite-3.1", "text-gen... Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 12: granitemoe.block_count u32 = 32 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 13: granitemoe.context_length u32 = 131072 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 14: granitemoe.embedding_length u32 = 1536 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 15: granitemoe.feed_forward_length u32 = 512 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 16: granitemoe.attention.head_count u32 = 24 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 17: granitemoe.attention.head_count_kv u32 = 8 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 18: granitemoe.rope.freq_base f32 = 10000000.000000 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 19: granitemoe.attention.layer_norm_rms_epsilon f32 = 0.000001 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 20: granitemoe.expert_count u32 = 40 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 21: granitemoe.expert_used_count u32 = 8 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 22: general.file_type u32 = 7 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 23: granitemoe.vocab_size u32 = 49155 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 24: granitemoe.rope.dimension_count u32 = 64 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 25: granitemoe.attention.scale f32 = 0.015625 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 26: granitemoe.embedding_scale f32 = 12.000000 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 27: granitemoe.residual_scale f32 = 0.220000 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 28: granitemoe.logit_scale f32 = 6.000000 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 29: tokenizer.ggml.model str = gpt2 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 30: tokenizer.ggml.pre str = refact Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 31: tokenizer.ggml.tokens arr[str,49155] = ["<|end_of_text|>", "<fim_prefix>", "... Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 32: tokenizer.ggml.token_type arr[i32,49155] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ... Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:06.111-06:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server loading model" Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 33: tokenizer.ggml.merges arr[str,48891] = ["Ġ Ġ", "ĠĠ ĠĠ", "ĠĠĠĠ ĠĠ... Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 34: tokenizer.ggml.bos_token_id u32 = 0 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 35: tokenizer.ggml.eos_token_id u32 = 0 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 36: tokenizer.ggml.unknown_token_id u32 = 0 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 37: tokenizer.ggml.padding_token_id u32 = 0 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 38: tokenizer.ggml.add_bos_token bool = false Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 39: tokenizer.chat_template str = {%- if messages[0]['role'] == 'system... Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 40: tokenizer.ggml.add_space_prefix bool = false Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 41: general.quantization_version u32 = 2 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - type f32: 97 tensors Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llama_model_loader: - type q8_0: 225 tensors Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_vocab: special tokens cache size = 22 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_vocab: token to piece cache size = 0.2826 MB Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: format = GGUF V3 (latest) Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: arch = granitemoe Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: vocab type = BPE Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_vocab = 49155 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_merges = 48891 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: vocab_only = 0 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_ctx_train = 131072 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_embd = 1536 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_layer = 32 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_head = 24 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_head_kv = 8 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_rot = 64 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_swa = 0 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_embd_head_k = 64 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_embd_head_v = 64 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_gqa = 3 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_embd_k_gqa = 512 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_embd_v_gqa = 512 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_norm_eps = 0.0e+00 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_norm_rms_eps = 1.0e-06 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_clamp_kqv = 0.0e+00 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_logit_scale = 6.0e+00 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_ff = 512 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_expert = 40 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_expert_used = 8 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: causal attn = 1 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: pooling type = 0 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: rope type = 0 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: rope scaling = linear Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: freq_base_train = 10000000.0 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: freq_scale_train = 1 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_ctx_orig_yarn = 131072 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: rope_finetuned = unknown Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: ssm_d_conv = 0 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: ssm_d_inner = 0 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: ssm_d_state = 0 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: ssm_dt_rank = 0 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: ssm_dt_b_c_rms = 0 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: model type = 3B Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: model ftype = Q8_0 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: model params = 3.30 B Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: model size = 3.27 GiB (8.51 BPW) Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: general.name = Granite 3.1 3b A800M Instruct Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: BOS token = 0 '<|end_of_text|>' Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: EOS token = 0 '<|end_of_text|>' Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: UNK token = 0 '<|end_of_text|>' Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: PAD token = 0 '<|end_of_text|>' Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: LF token = 145 'Ä' Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: EOG token = 0 '<|end_of_text|>' Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: max token length = 512 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_embedding_scale = 12.000000 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_residual_scale = 0.220000 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_attention_scale = 0.015625 Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_tensors: offloading 32 repeating layers to GPU Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_tensors: offloading output layer to GPU Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_tensors: offloaded 33/33 layers to GPU Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_tensors: CPU_Mapped model buffer size = 76.50 MiB Jan 08 11:29:06 dre-slimserver-3t ollama[270648]: llm_load_tensors: CUDA0 model buffer size = 3348.39 MiB Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: n_seq_max = 4 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: n_ctx = 8192 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: n_ctx_per_seq = 2048 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: n_batch = 2048 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: n_ubatch = 512 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: flash_attn = 0 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: freq_base = 10000000.0 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: freq_scale = 1 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: n_ctx_per_seq (2048) < n_ctx_train (131072) -- the full capacity of the model will not be utilized Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_kv_cache_init: CUDA0 KV buffer size = 512.00 MiB Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: KV self size = 512.00 MiB, K (f16): 256.00 MiB, V (f16): 256.00 MiB Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: CUDA_Host output buffer size = 0.77 MiB Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: CUDA0 compute buffer size = 412.00 MiB Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: CUDA_Host compute buffer size = 19.01 MiB Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: graph nodes = 1960 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_new_context_with_model: graph splits = 2 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:29:07.367-06:00 level=INFO source=server.go:594 msg="llama runner started in 1.51 seconds" Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: loaded meta data with 42 key-value pairs and 322 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0 (version GGUF V3 (latest)) Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 0: general.architecture str = granitemoe Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 1: general.type str = model Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 2: general.name str = Granite 3.1 3b A800M Instruct Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 3: general.finetune str = instruct Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 4: general.basename str = granite-3.1 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 5: general.size_label str = 3B-a800M Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 6: general.license str = apache-2.0 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 7: general.base_model.count u32 = 1 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 8: general.base_model.0.name str = Granite 3.1 3b A800M Base Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 9: general.base_model.0.organization str = Ibm Granite Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 10: general.base_model.0.repo_url str = https://huggingface.co/ibm-granite/gr... Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 11: general.tags arr[str,3] = ["language", "granite-3.1", "text-gen... Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 12: granitemoe.block_count u32 = 32 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 13: granitemoe.context_length u32 = 131072 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 14: granitemoe.embedding_length u32 = 1536 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 15: granitemoe.feed_forward_length u32 = 512 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 16: granitemoe.attention.head_count u32 = 24 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 17: granitemoe.attention.head_count_kv u32 = 8 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 18: granitemoe.rope.freq_base f32 = 10000000.000000 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 19: granitemoe.attention.layer_norm_rms_epsilon f32 = 0.000001 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 20: granitemoe.expert_count u32 = 40 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 21: granitemoe.expert_used_count u32 = 8 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 22: general.file_type u32 = 7 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 23: granitemoe.vocab_size u32 = 49155 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 24: granitemoe.rope.dimension_count u32 = 64 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 25: granitemoe.attention.scale f32 = 0.015625 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 26: granitemoe.embedding_scale f32 = 12.000000 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 27: granitemoe.residual_scale f32 = 0.220000 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 28: granitemoe.logit_scale f32 = 6.000000 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 29: tokenizer.ggml.model str = gpt2 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 30: tokenizer.ggml.pre str = refact Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 31: tokenizer.ggml.tokens arr[str,49155] = ["<|end_of_text|>", "<fim_prefix>", "... Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 32: tokenizer.ggml.token_type arr[i32,49155] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ... Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 33: tokenizer.ggml.merges arr[str,48891] = ["Ġ Ġ", "ĠĠ ĠĠ", "ĠĠĠĠ ĠĠ... Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 34: tokenizer.ggml.bos_token_id u32 = 0 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 35: tokenizer.ggml.eos_token_id u32 = 0 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 36: tokenizer.ggml.unknown_token_id u32 = 0 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 37: tokenizer.ggml.padding_token_id u32 = 0 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 38: tokenizer.ggml.add_bos_token bool = false Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 39: tokenizer.chat_template str = {%- if messages[0]['role'] == 'system... Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 40: tokenizer.ggml.add_space_prefix bool = false Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - kv 41: general.quantization_version u32 = 2 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - type f32: 97 tensors Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_loader: - type q8_0: 225 tensors Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_vocab: special tokens cache size = 22 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_vocab: token to piece cache size = 0.2826 MB Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: format = GGUF V3 (latest) Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: arch = granitemoe Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: vocab type = BPE Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_vocab = 49155 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: n_merges = 48891 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: vocab_only = 1 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: model type = ?B Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: model ftype = all F32 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: model params = 3.30 B Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: model size = 3.27 GiB (8.51 BPW) Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: general.name = Granite 3.1 3b A800M Instruct Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: BOS token = 0 '<|end_of_text|>' Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: EOS token = 0 '<|end_of_text|>' Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: UNK token = 0 '<|end_of_text|>' Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: PAD token = 0 '<|end_of_text|>' Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: LF token = 145 'Ä' Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: EOG token = 0 '<|end_of_text|>' Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: max token length = 512 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_embedding_scale = 0.000000 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_residual_scale = 0.000000 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llm_load_print_meta: f_attention_scale = 0.000000 Jan 08 11:29:07 dre-slimserver-3t ollama[270648]: llama_model_load: vocab only - skipping tensors Jan 08 11:29:26 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:26 | 200 | 22.168833156s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:27 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:27 | 200 | 23.007174082s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:29 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:29 | 200 | 25.488162066s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: llama_decode: failed to decode, ret = 1 Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: llama_decode: failed to decode, ret = 1 Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: panic: failed to decode batch: could not find a kv cache slot Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: goroutine 4 [running]: Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: github.com/ollama/ollama/llama/runner.(*Server).run(0xc000190000, {0x5cf0c0a40de0, 0xc00018e050}) Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: github.com/ollama/ollama/llama/runner/runner.go:344 +0x22e Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1 Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: github.com/ollama/ollama/llama/runner/runner.go:984 +0xde5 Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.388184547s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.388918653s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.388476131s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.390067661s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.389589364s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.389713531s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.389941662s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.388112951s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.390710952s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.389109081s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.391067377s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.390580576s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.39114927s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.373548744s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 4.14116083s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 3.259212673s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 822.460608ms | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:29:30 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:29:30 | 200 | 26.428474109s | 100.121.189.48 | POST "/v1/chat/completions" Jan 08 11:37:25 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:25.999-06:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.325324089 model=/usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0 Jan 08 11:37:26 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:26.774-06:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=6.099783887 model=/usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0 Jan 08 11:37:27 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:27.412-06:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=6.7380075569999995 model=/usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0 Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:28.201-06:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0 gpu=GPU-a9fe4c2f-a87c-c645-82ef-2da178797d63 parallel=4 available=16606167040 required="4.6 GiB" Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:28.833-06:00 level=INFO source=server.go:104 msg="system memory" total="125.7 GiB" free="121.0 GiB" free_swap="2.0 GiB" Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:28.834-06:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[15.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="4.6 GiB" memory.required.partial="4.6 GiB" memory.required.kv="512.0 MiB" memory.required.allocations="[4.6 GiB]" memory.weights.total="3.7 GiB" memory.weights.repeating="3.6 GiB" memory.weights.nonrepeating="76.5 MiB" memory.graph.full="256.0 MiB" memory.graph.partial="256.0 MiB" Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:28.834-06:00 level=INFO source=server.go:376 msg="starting llama server" cmd="/usr/local/lib/ollama/runners/cuda_v12_avx/ollama_llama_server runner --model /usr/share/ollama/.ollama/models/blobs/sha256-c0cf6e02ad4e0f21b02593bb572a68faaca77307e5e16ec4cc13e5921235daa0 --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --threads 10 --parallel 4 --port 32861" Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:28.835-06:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:28.835-06:00 level=INFO source=server.go:555 msg="waiting for llama runner to start responding" Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:28.835-06:00 level=WARN source=server.go:562 msg="client connection closed before server finished loading, aborting load" Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: time=2025-01-08T11:37:28.835-06:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="timed out waiting for llama runner to start: context canceled" Jan 08 11:37:28 dre-slimserver-3t ollama[270648]: [GIN] 2025/01/08 - 11:37:28 | 499 | 7m57s | 100.121.189.48 | POST "/v1/chat/completions"
GiteaMirror commented 2026-04-12 16:33:34 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 8, 2025):

From the timestamps and call durations, the client is sending a lot of large simultaneous requests. Try increasing OLLAMA_NUM_PARALLEL, or increase the size of the context buffer, either by setting num_ctx in the API call, or setting a default for the model.

<!-- gh-comment-id:2578876531 --> @rick-github commented on GitHub (Jan 8, 2025): From the timestamps and call durations, the client is sending a lot of large simultaneous requests. Try increasing [`OLLAMA_NUM_PARALLEL`](https://github.com/ollama/ollama/blob/6ae2adc1af9199dd348c30070c9b2e26699a5169/envconfig/config.go#L249), or increase the size of the context buffer, either by setting [`num_ctx`](https://github.com/ollama/ollama/blob/6ae2adc1af9199dd348c30070c9b2e26699a5169/docs/modelfile.md#valid-parameters-and-values:~:text=mirostat_tau%205.0-,num_ctx,-Sets%20the%20size) in the API call, or [setting](https://github.com/ollama/ollama/issues/5965#issuecomment-2252354726) a default for the model.
GiteaMirror commented 2026-04-12 16:33:35 -05:00
Author
Owner
Copy Link

@ergosumdre commented on GitHub (Jan 9, 2025):

Thank you! I followed these steps and it seems to work!

<!-- gh-comment-id:2578942326 --> @ergosumdre commented on GitHub (Jan 9, 2025): Thank you! I followed these [steps](https://github.com/ollama/ollama/issues/5965#issuecomment-2252354726) and it seems to work!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#5353
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?