github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 04:34:16 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #12027] model switch not performed. #7986

New Issue
Open
opened 2026-04-12 20:10:47 -05:00 by GiteaMirror · 9 comments
GiteaMirror commented 2026-04-12 20:10:47 -05:00
Owner
Copy Link

Originally created by @mcr-ksh on GitHub (Aug 22, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/12027

What is the issue?

What is the issue?

a second model is not loading after a first model was used. This used to work on 11.5 but is defunct in 11.6.
There is no additional log, only that the first model was successfully loaded and API requests are shown for the second model, but an attempt to switch to the second model is not made.

When having a model loaded with env:

OLLAMA_MODELS=O:\models
OLLAMA_MAX_LOADED_MODELS=1
OLLAMA_HOST=0.0.0.0:11434
OLLAMA_NOHISTORY=1
OLLAMA_NEW_ESTIMATES=1
OLLAMA_LOAD_TIMEOUT=30m
OLLAMA_DEBUG=0
OLLAMA_NEW_ENGINE=1
OLLAMA_KEEP_ALIVE=24h
OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q4_0
GGML_CUDA_FORCE_MMQ=1
GGML_CUDA_FORCE_CUBLAS=1
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1

Relevant log output

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

0.11.6

Relevant log output

OS

No response

GPU

No response

CPU

No response

Ollama version

No response

Originally created by @mcr-ksh on GitHub (Aug 22, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/12027 ### What is the issue? ### What is the issue? a second model is not loading after a first model was used. This used to work on 11.5 but is defunct in 11.6. There is no additional log, only that the first model was successfully loaded and API requests are shown for the second model, but an attempt to switch to the second model is not made. When having a model loaded with env: ``` OLLAMA_MODELS=O:\models OLLAMA_MAX_LOADED_MODELS=1 OLLAMA_HOST=0.0.0.0:11434 OLLAMA_NOHISTORY=1 OLLAMA_NEW_ESTIMATES=1 OLLAMA_LOAD_TIMEOUT=30m OLLAMA_DEBUG=0 OLLAMA_NEW_ENGINE=1 OLLAMA_KEEP_ALIVE=24h OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q4_0 GGML_CUDA_FORCE_MMQ=1 GGML_CUDA_FORCE_CUBLAS=1 GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 ``` ### Relevant log output ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version 0.11.6 ### Relevant log output ```shell ``` ### OS _No response_ ### GPU _No response_ ### CPU _No response_ ### Ollama version _No response_
GiteaMirror added the bug label 2026-04-12 20:10:47 -05:00
GiteaMirror commented 2026-04-12 20:10:48 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 22, 2025):

$ ollama -v
ollama version is 0.11.6
$ ollama run qwen3:4b-instruct hello
Hello! How can I assist you today? 😊

$ ollama ps
NAME                 ID              SIZE      PROCESSOR    CONTEXT    UNTIL   
qwen3:4b-instruct    088c6bc07f1d    3.0 GB    100% GPU     4096       Forever  
$ ollama run gemma3 hello
Hello there! How's your day going so far? Is there anything I can help you 
with today? 😊 

$ ollama ps
NAME             ID              SIZE      PROCESSOR    CONTEXT    UNTIL   
gemma3:latest    a2af6cc3eb7f    5.1 GB    100% GPU     4096       Forever

Server logs may aid in debugging.

<!-- gh-comment-id:3214283447 --> @rick-github commented on GitHub (Aug 22, 2025): ```console $ ollama -v ollama version is 0.11.6 $ ollama run qwen3:4b-instruct hello Hello! How can I assist you today? 😊 $ ollama ps NAME ID SIZE PROCESSOR CONTEXT UNTIL qwen3:4b-instruct 088c6bc07f1d 3.0 GB 100% GPU 4096 Forever $ ollama run gemma3 hello Hello there! How's your day going so far? Is there anything I can help you with today? 😊 $ ollama ps NAME ID SIZE PROCESSOR CONTEXT UNTIL gemma3:latest a2af6cc3eb7f 5.1 GB 100% GPU 4096 Forever ``` [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) may aid in debugging.
GiteaMirror commented 2026-04-12 20:10:49 -05:00
Author
Owner
Copy Link

@mcr-ksh commented on GitHub (Aug 22, 2025):

I know. The server logs basically just show one model loading and the API responding. @rick-github did you have the same env variables set incl. new engine and estimates?

<!-- gh-comment-id:3214962775 --> @mcr-ksh commented on GitHub (Aug 22, 2025): I know. The server logs basically just show one model loading and the API responding. @rick-github did you have the same env variables set incl. new engine and estimates?
GiteaMirror commented 2026-04-12 20:10:49 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 22, 2025):

@rick-github did you have the same env variables set incl. new engine and estimates?

Yes.

Perform this test and show the results. Replace gemma3 and qwen3 with the names of the models for which the switch doesn't happen:

curl localhost:11434/api/generate -d "{\"model\":\"gemma3\"}"
ollama ps
curl localhost:11434/api/generate -d "{\"model\":\"qwen3\"}"
ollama ps

Server logs may aid in debugging.

<!-- gh-comment-id:3214979642 --> @rick-github commented on GitHub (Aug 22, 2025): > @rick-github did you have the same env variables set incl. new engine and estimates? Yes. Perform this test and show the results. Replace `gemma3` and `qwen3` with the names of the models for which the switch doesn't happen: ```console curl localhost:11434/api/generate -d "{\"model\":\"gemma3\"}" ollama ps curl localhost:11434/api/generate -d "{\"model\":\"qwen3\"}" ollama ps ``` [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) may aid in debugging.
GiteaMirror commented 2026-04-12 20:10:50 -05:00
Author
Owner
Copy Link

@mcr-ksh commented on GitHub (Aug 24, 2025):

I've reinstalled and tested again and now it's working at least at the beginning. I've had the process killed and restarted multiple times but the loading didn't work and now it did for a few times until it gets stuck again

For the sake of completeness the server log:

time=2025-08-24T08:07:32.191Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:07:32.201Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:07:32.207Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 64712"
time=2025-08-24T08:07:32.209Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1
time=2025-08-24T08:07:32.219Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.6 GiB" free_swap="35.9 GiB"
time=2025-08-24T08:07:32.219Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:07:32.289Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:07:32.324Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:64712"
time=2025-08-24T08:07:32.334Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:07:32.360Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:07:32.529Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:07:32.677Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:07:32.970Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:07:32.971Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU"
time=2025-08-24T08:07:32.971Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:07:32.971Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU"
time=2025-08-24T08:07:32.971Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB"
time=2025-08-24T08:07:32.971Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB"
time=2025-08-24T08:07:32.971Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB"
time=2025-08-24T08:07:32.971Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB"
time=2025-08-24T08:07:32.971Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-08-24T08:07:32.971Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB"
time=2025-08-24T08:07:32.971Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:07:32.971Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:07:32.972Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:07:50.268Z level=INFO source=server.go:1272 msg="llama runner started in 18.06 seconds"
time=2025-08-24T08:07:50.374Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=21812 keep=4 new=9200
[GIN] 2025/08/24 - 08:08:06 | 200 |   34.7778902s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:08:07.017Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=21812 keep=4 new=9200
[GIN] 2025/08/24 - 08:08:24 | 200 |   17.9491192s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:08:25.156Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=21897 keep=4 new=9200
[GIN] 2025/08/24 - 08:08:40 | 200 |   15.6296422s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:08:40.988Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=21897 keep=4 new=9200
[GIN] 2025/08/24 - 08:08:57 | 200 |   17.0038725s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:08:58.469Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:08:58.480Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:08:58.488Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 64741"
time=2025-08-24T08:08:58.491Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1
time=2025-08-24T08:08:58.500Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.4 GiB" free_swap="35.7 GiB"
time=2025-08-24T08:08:58.500Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:08:58.569Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:08:58.599Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:64741"
time=2025-08-24T08:08:58.610Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:08:58.642Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:08:58.829Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:08:58.974Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:08:59.150Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:08:59.151Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB"
time=2025-08-24T08:08:59.151Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB"
time=2025-08-24T08:08:59.151Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB"
time=2025-08-24T08:08:59.151Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB"
time=2025-08-24T08:08:59.151Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB"
time=2025-08-24T08:08:59.151Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB"
time=2025-08-24T08:08:59.151Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:08:59.151Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:08:59.151Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:08:59.152Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU"
time=2025-08-24T08:08:59.152Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:08:59.152Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU"
time=2025-08-24T08:09:08.928Z level=INFO source=server.go:1272 msg="llama runner started in 10.44 seconds"
[GIN] 2025/08/24 - 08:12:10 | 200 |         3m13s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:12:32 | 200 |    21.061812s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:12:55 | 200 |   22.8297054s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:13:06 | 200 |   10.9356254s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:13:09 | 200 |    2.7860686s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:13:12 | 200 |    3.6968598s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:13:16 | 200 |    3.3646689s |    192.168.1.17 | POST     "/api/generate"
time=2025-08-24T08:13:22.116Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:13:22.126Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:13:22.133Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 64812"
time=2025-08-24T08:13:22.136Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1
time=2025-08-24T08:13:22.144Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.7 GiB" free_swap="36.1 GiB"
time=2025-08-24T08:13:22.144Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:13:22.214Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:13:22.247Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:64812"
time=2025-08-24T08:13:22.251Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:13:22.275Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:13:22.431Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:13:22.631Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:13:22.929Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:13:22.929Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU"
time=2025-08-24T08:13:22.929Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:13:22.929Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU"
time=2025-08-24T08:13:22.930Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB"
time=2025-08-24T08:13:22.930Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB"
time=2025-08-24T08:13:22.930Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB"
time=2025-08-24T08:13:22.930Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB"
time=2025-08-24T08:13:22.930Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-08-24T08:13:22.930Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB"
time=2025-08-24T08:13:22.930Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:13:22.930Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:13:22.930Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:13:38.974Z level=INFO source=server.go:1272 msg="llama runner started in 16.84 seconds"
[GIN] 2025/08/24 - 08:13:42 | 200 |   20.8934571s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:13:44 | 200 |    1.8952135s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:13:49 | 200 |    4.2768875s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:13:52 | 200 |    3.0859709s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:13:55 | 200 |     2.730771s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:13:59 | 200 |    3.9742797s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:14:07 | 200 |    7.8356668s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:14:10 | 200 |    2.6263678s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:14:17 | 200 |    6.6245268s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:14:17.701Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:14:17.711Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:14:17.718Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 64835"
time=2025-08-24T08:14:17.722Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1
time=2025-08-24T08:14:17.730Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.7 GiB" free_swap="36.1 GiB"
time=2025-08-24T08:14:17.730Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:14:17.806Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:14:17.836Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:64835"
time=2025-08-24T08:14:17.848Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:14:17.875Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:14:18.049Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:14:18.180Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:14:18.358Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:14:18.358Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU"
time=2025-08-24T08:14:18.358Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:14:18.358Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU"
time=2025-08-24T08:14:18.358Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB"
time=2025-08-24T08:14:18.358Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB"
time=2025-08-24T08:14:18.358Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB"
time=2025-08-24T08:14:18.358Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB"
time=2025-08-24T08:14:18.358Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB"
time=2025-08-24T08:14:18.358Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB"
time=2025-08-24T08:14:18.358Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:14:18.358Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:14:18.359Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:14:27.386Z level=INFO source=server.go:1272 msg="llama runner started in 9.67 seconds"
[GIN] 2025/08/24 - 08:14:32 | 200 |   15.1381859s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:14:36 | 200 |    4.0328043s |    192.168.1.17 | POST     "/api/generate"
time=2025-08-24T08:17:38.311Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:17:38.320Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:17:38.328Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 64897"
time=2025-08-24T08:17:38.330Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1
time=2025-08-24T08:17:38.339Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.6 GiB" free_swap="36.1 GiB"
time=2025-08-24T08:17:38.340Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:17:38.408Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:17:38.441Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:64897"
time=2025-08-24T08:17:38.447Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:17:38.472Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:17:38.634Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:17:38.837Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:17:39.150Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:17:39.150Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU"
time=2025-08-24T08:17:39.150Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:17:39.150Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU"
time=2025-08-24T08:17:39.151Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB"
time=2025-08-24T08:17:39.151Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB"
time=2025-08-24T08:17:39.151Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB"
time=2025-08-24T08:17:39.151Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB"
time=2025-08-24T08:17:39.151Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-08-24T08:17:39.151Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB"
time=2025-08-24T08:17:39.151Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:17:39.151Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:17:39.151Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:17:55.198Z level=INFO source=server.go:1272 msg="llama runner started in 16.87 seconds"
time=2025-08-24T08:17:55.303Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17679 keep=4 new=9200
[GIN] 2025/08/24 - 08:18:11 | 200 |   33.8854044s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:18:11.711Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=21812 keep=4 new=9200
[GIN] 2025/08/24 - 08:18:24 | 200 |   46.0562294s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:18:24.949Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17679 keep=4 new=9200
[GIN] 2025/08/24 - 08:18:38 | 200 |   26.6680352s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:18:38.538Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=21812 keep=4 new=9200
[GIN] 2025/08/24 - 08:18:55 | 200 |   30.6227418s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:18:55.749Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17764 keep=4 new=9200
[GIN] 2025/08/24 - 08:19:11 | 200 |    32.348392s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:19:11.047Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=21897 keep=4 new=9200
[GIN] 2025/08/24 - 08:19:27 | 200 |   31.8002034s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:19:27.736Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17764 keep=4 new=9200
[GIN] 2025/08/24 - 08:19:44 | 200 |   33.1371324s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:19:44.408Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=21897 keep=4 new=9200
[GIN] 2025/08/24 - 08:20:02 | 200 |   34.2558054s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:20:02.540Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:20:02.550Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:20:02.557Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 64973"
time=2025-08-24T08:20:02.559Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1
time=2025-08-24T08:20:02.571Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.7 GiB" free_swap="36.2 GiB"
time=2025-08-24T08:20:02.571Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:20:02.637Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:20:02.666Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:64973"
time=2025-08-24T08:20:02.677Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:20:02.709Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:20:02.866Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:20:03.014Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:20:03.208Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:20:03.208Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU"
time=2025-08-24T08:20:03.208Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:20:03.208Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU"
time=2025-08-24T08:20:03.208Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB"
time=2025-08-24T08:20:03.208Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB"
time=2025-08-24T08:20:03.208Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB"
time=2025-08-24T08:20:03.208Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB"
time=2025-08-24T08:20:03.208Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB"
time=2025-08-24T08:20:03.208Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB"
time=2025-08-24T08:20:03.208Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:20:03.208Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:20:03.210Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:20:11.980Z level=INFO source=server.go:1272 msg="llama runner started in 9.42 seconds"
[GIN] 2025/08/24 - 08:20:21 | 200 |   37.4275719s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:20:44 | 200 |   42.3701852s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:20:49 | 200 |   27.9177488s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:21:00 | 200 |   15.4601114s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:21:14 | 200 |   24.0274647s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:21:23 | 200 |   23.3699805s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:21:29 | 200 |    14.910289s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:21:36 | 200 |     12.79915s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:21:39 | 200 |    3.0346262s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:21:42 | 200 |    2.6803838s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:21:45 | 200 |    2.9273598s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:21:48 | 200 |    2.7270342s |    192.168.1.17 | POST     "/api/generate"
time=2025-08-24T08:22:19.262Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:22:19.272Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:22:19.280Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 65059"
time=2025-08-24T08:22:19.282Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1
time=2025-08-24T08:22:19.291Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.3 GiB" free_swap="35.8 GiB"
time=2025-08-24T08:22:19.291Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:22:19.362Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:22:19.395Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65059"
time=2025-08-24T08:22:19.397Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:22:19.420Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:22:19.609Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:22:19.758Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:22:20.049Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:22:20.050Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU"
time=2025-08-24T08:22:20.050Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:22:20.050Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU"
time=2025-08-24T08:22:20.050Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB"
time=2025-08-24T08:22:20.050Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB"
time=2025-08-24T08:22:20.050Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB"
time=2025-08-24T08:22:20.050Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB"
time=2025-08-24T08:22:20.050Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-08-24T08:22:20.050Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB"
time=2025-08-24T08:22:20.050Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:22:20.050Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:22:20.050Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:22:35.591Z level=INFO source=server.go:1272 msg="llama runner started in 16.31 seconds"
time=2025-08-24T08:22:37.120Z level=ERROR source=server.go:1442 msg="post predict" error="Post \"http://127.0.0.1:65059/completion\": context canceled"
[GIN] 2025/08/24 - 08:22:37 | 200 |   18.3661583s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:23:52.324Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17679 keep=4 new=9200
[GIN] 2025/08/24 - 08:24:06 | 200 |   14.6810069s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:24:11 | 200 |   19.1207952s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:24:11.405Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17679 keep=4 new=9200
[GIN] 2025/08/24 - 08:24:27 | 200 |   20.1823008s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:24:30 | 200 |   18.6093555s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:24:30.277Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17764 keep=4 new=9200
[GIN] 2025/08/24 - 08:24:47 | 200 |   20.4410958s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:24:51 | 200 |    20.913231s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:24:51.227Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17764 keep=4 new=9200
[GIN] 2025/08/24 - 08:25:08 | 200 |   20.4097597s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:25:13 | 200 |    21.751285s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:25:13.506Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:25:13.515Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:25:13.523Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 65120"
time=2025-08-24T08:25:13.526Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1
time=2025-08-24T08:25:13.535Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.3 GiB" free_swap="35.8 GiB"
time=2025-08-24T08:25:13.535Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:25:13.606Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:25:13.639Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65120"
time=2025-08-24T08:25:13.642Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:25:13.668Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:25:13.825Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:25:13.975Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:25:14.156Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:25:14.156Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU"
time=2025-08-24T08:25:14.156Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:25:14.156Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU"
time=2025-08-24T08:25:14.156Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB"
time=2025-08-24T08:25:14.156Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB"
time=2025-08-24T08:25:14.156Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB"
time=2025-08-24T08:25:14.156Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB"
time=2025-08-24T08:25:14.156Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB"
time=2025-08-24T08:25:14.156Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB"
time=2025-08-24T08:25:14.156Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:25:14.156Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:25:14.156Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:25:22.438Z level=INFO source=server.go:1272 msg="llama runner started in 8.92 seconds"
[GIN] 2025/08/24 - 08:25:38 | 200 |   29.7982689s |    192.168.1.17 | POST     "/api/generate"
time=2025-08-24T08:25:38.751Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:25:38.761Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:25:38.768Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 65136"
time=2025-08-24T08:25:38.770Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1
time=2025-08-24T08:25:38.780Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.4 GiB" free_swap="35.8 GiB"
time=2025-08-24T08:25:38.780Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:25:38.851Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:25:38.880Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65136"
time=2025-08-24T08:25:38.883Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:25:38.906Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:25:39.068Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:25:39.207Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:25:39.469Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:25:39.469Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU"
time=2025-08-24T08:25:39.469Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:25:39.469Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU"
time=2025-08-24T08:25:39.469Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB"
time=2025-08-24T08:25:39.469Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB"
time=2025-08-24T08:25:39.469Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB"
time=2025-08-24T08:25:39.469Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB"
time=2025-08-24T08:25:39.469Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-08-24T08:25:39.469Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB"
time=2025-08-24T08:25:39.470Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:25:39.470Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:25:39.470Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:25:54.508Z level=INFO source=server.go:1272 msg="llama runner started in 15.74 seconds"
[GIN] 2025/08/24 - 08:25:59 | 200 |   45.9285195s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:25:59.660Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:25:59.671Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:25:59.678Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 65150"
time=2025-08-24T08:25:59.682Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1
time=2025-08-24T08:25:59.694Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.4 GiB" free_swap="35.8 GiB"
time=2025-08-24T08:25:59.694Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:25:59.760Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:25:59.790Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65150"
time=2025-08-24T08:25:59.800Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:25:59.828Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:25:59.983Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:26:00.134Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:26:00.309Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:26:00.310Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU"
time=2025-08-24T08:26:00.310Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:26:00.310Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU"
time=2025-08-24T08:26:00.310Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB"
time=2025-08-24T08:26:00.310Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB"
time=2025-08-24T08:26:00.310Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB"
time=2025-08-24T08:26:00.310Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB"
time=2025-08-24T08:26:00.310Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB"
time=2025-08-24T08:26:00.310Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB"
time=2025-08-24T08:26:00.310Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:26:00.310Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:26:00.311Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:26:08.833Z level=INFO source=server.go:1272 msg="llama runner started in 9.16 seconds"
[GIN] 2025/08/24 - 08:26:19 | 200 |   40.5837591s |    192.168.1.17 | POST     "/api/generate"
time=2025-08-24T08:26:19.474Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:26:19.484Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:26:19.492Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 65165"
time=2025-08-24T08:26:19.495Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1
time=2025-08-24T08:26:19.503Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.3 GiB" free_swap="35.8 GiB"
time=2025-08-24T08:26:19.503Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:26:19.571Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:26:19.603Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65165"
time=2025-08-24T08:26:19.609Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:26:19.633Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:26:19.793Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:26:19.931Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:26:20.203Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:26:20.203Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU"
time=2025-08-24T08:26:20.203Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:26:20.203Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU"
time=2025-08-24T08:26:20.203Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB"
time=2025-08-24T08:26:20.203Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB"
time=2025-08-24T08:26:20.203Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB"
time=2025-08-24T08:26:20.203Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB"
time=2025-08-24T08:26:20.203Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-08-24T08:26:20.203Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB"
time=2025-08-24T08:26:20.203Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:26:20.203Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:26:20.204Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:26:35.498Z level=INFO source=server.go:1272 msg="llama runner started in 16.01 seconds"
[GIN] 2025/08/24 - 08:26:44 | 200 |   44.6729655s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:26:44.439Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:26:44.449Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:26:44.458Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 65186"
time=2025-08-24T08:26:44.460Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1
time=2025-08-24T08:26:44.469Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.2 GiB" free_swap="35.7 GiB"
time=2025-08-24T08:26:44.469Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:26:44.539Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:26:44.577Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65186"
time=2025-08-24T08:26:44.579Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:26:44.613Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:26:44.781Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:26:44.911Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:26:45.110Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:26:45.111Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU"
time=2025-08-24T08:26:45.111Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:26:45.111Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU"
time=2025-08-24T08:26:45.111Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB"
time=2025-08-24T08:26:45.111Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB"
time=2025-08-24T08:26:45.111Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB"
time=2025-08-24T08:26:45.111Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB"
time=2025-08-24T08:26:45.111Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB"
time=2025-08-24T08:26:45.111Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB"
time=2025-08-24T08:26:45.111Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:26:45.111Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:26:45.111Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:26:53.138Z level=INFO source=server.go:1272 msg="llama runner started in 8.68 seconds"
[GIN] 2025/08/24 - 08:27:04 | 200 |   45.4527574s |    192.168.1.17 | POST     "/api/generate"
time=2025-08-24T08:27:05.032Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:27:05.042Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:27:05.049Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 65202"
time=2025-08-24T08:27:05.051Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1
time=2025-08-24T08:27:05.061Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.6 GiB" free_swap="35.8 GiB"
time=2025-08-24T08:27:05.061Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:27:05.132Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:27:05.163Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65202"
time=2025-08-24T08:27:05.167Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:27:05.192Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:27:05.351Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:27:05.495Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:27:05.754Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:27:05.755Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU"
time=2025-08-24T08:27:05.755Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:27:05.755Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU"
time=2025-08-24T08:27:05.755Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB"
time=2025-08-24T08:27:05.755Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB"
time=2025-08-24T08:27:05.755Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB"
time=2025-08-24T08:27:05.755Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB"
time=2025-08-24T08:27:05.755Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-08-24T08:27:05.755Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB"
time=2025-08-24T08:27:05.755Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:27:05.755Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:27:05.755Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:27:21.295Z level=INFO source=server.go:1272 msg="llama runner started in 16.25 seconds"
[GIN] 2025/08/24 - 08:27:28 | 200 |   43.9324582s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:27:28.824Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:27:28.835Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:27:28.842Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 65220"
time=2025-08-24T08:27:28.845Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1
time=2025-08-24T08:27:28.856Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.7 GiB" free_swap="35.9 GiB"
time=2025-08-24T08:27:28.856Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:27:28.923Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:27:28.952Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65220"
time=2025-08-24T08:27:28.962Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:27:28.990Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:27:29.161Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:27:29.367Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:27:29.546Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:27:29.546Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU"
time=2025-08-24T08:27:29.546Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:27:29.546Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU"
time=2025-08-24T08:27:29.546Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB"
time=2025-08-24T08:27:29.546Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB"
time=2025-08-24T08:27:29.546Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB"
time=2025-08-24T08:27:29.546Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB"
time=2025-08-24T08:27:29.546Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB"
time=2025-08-24T08:27:29.546Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB"
time=2025-08-24T08:27:29.546Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:27:29.546Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:27:29.546Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:27:37.573Z level=INFO source=server.go:1272 msg="llama runner started in 8.73 seconds"
[GIN] 2025/08/24 - 08:27:42 | 200 |   37.5892486s |    192.168.1.17 | POST     "/api/generate"
time=2025-08-24T08:27:42.742Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:27:42.751Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:27:42.759Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 65233"
time=2025-08-24T08:27:42.761Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1
time=2025-08-24T08:27:42.773Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.7 GiB" free_swap="35.9 GiB"
time=2025-08-24T08:27:42.773Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:27:42.840Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:27:42.870Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65233"
time=2025-08-24T08:27:42.877Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:27:42.901Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:27:43.066Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:27:43.209Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:27:43.465Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:27:43.465Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU"
time=2025-08-24T08:27:43.465Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:27:43.465Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU"
time=2025-08-24T08:27:43.466Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB"
time=2025-08-24T08:27:43.466Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB"
time=2025-08-24T08:27:43.466Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB"
time=2025-08-24T08:27:43.466Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB"
time=2025-08-24T08:27:43.466Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-08-24T08:27:43.466Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB"
time=2025-08-24T08:27:43.466Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:27:43.467Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:27:43.471Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:27:47.479Z level=INFO source=server.go:1272 msg="llama runner started in 4.72 seconds"
[GIN] 2025/08/24 - 08:27:56 | 200 |   28.0871306s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:28:02 | 200 |     5.697337s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:28:03.227Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:28:03.237Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:28:03.244Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 65246"
time=2025-08-24T08:28:03.246Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1
time=2025-08-24T08:28:03.256Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.6 GiB" free_swap="35.9 GiB"
time=2025-08-24T08:28:03.256Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:28:03.335Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:28:03.367Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65246"
time=2025-08-24T08:28:03.373Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:28:03.420Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:28:03.571Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:28:03.702Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:28:03.885Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:28:03.886Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU"
time=2025-08-24T08:28:03.886Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:28:03.886Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU"
time=2025-08-24T08:28:03.886Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB"
time=2025-08-24T08:28:03.886Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB"
time=2025-08-24T08:28:03.886Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB"
time=2025-08-24T08:28:03.886Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB"
time=2025-08-24T08:28:03.886Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB"
time=2025-08-24T08:28:03.886Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB"
time=2025-08-24T08:28:03.886Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:28:03.886Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:28:03.887Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:28:12.657Z level=INFO source=server.go:1272 msg="llama runner started in 9.41 seconds"
[GIN] 2025/08/24 - 08:28:17 | 200 |   14.7165468s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:28:22 | 200 |    4.6953768s |    192.168.1.17 | POST     "/api/generate"
time=2025-08-24T08:31:04.363Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:31:04.373Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:31:04.381Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 56277"
time=2025-08-24T08:31:04.384Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1
time=2025-08-24T08:31:04.394Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.6 GiB" free_swap="35.8 GiB"
time=2025-08-24T08:31:04.394Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:31:04.467Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:31:04.498Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:56277"
time=2025-08-24T08:31:04.509Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:31:04.534Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:31:04.736Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:31:04.873Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:31:05.174Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:31:05.175Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU"
time=2025-08-24T08:31:05.175Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:31:05.175Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU"
time=2025-08-24T08:31:05.175Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB"
time=2025-08-24T08:31:05.175Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB"
time=2025-08-24T08:31:05.175Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB"
time=2025-08-24T08:31:05.175Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB"
time=2025-08-24T08:31:05.175Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-08-24T08:31:05.175Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB"
time=2025-08-24T08:31:05.175Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:31:05.175Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:31:05.175Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:31:20.210Z level=INFO source=server.go:1272 msg="llama runner started in 15.83 seconds"
[GIN] 2025/08/24 - 08:31:23 | 200 |   19.9755304s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:31:27 | 200 |    3.2055066s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:31:33 | 200 |    3.9072038s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:31:36 | 200 |    2.5509373s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:31:40 | 200 |    3.9435223s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:31:47 | 200 |    6.3619846s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:31:49 | 200 |    2.6050403s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:31:56 | 200 |    5.8860515s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:31:56.663Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:31:56.672Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:31:56.680Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 56300"
time=2025-08-24T08:31:56.682Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1
time=2025-08-24T08:31:56.696Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.6 GiB" free_swap="35.9 GiB"
time=2025-08-24T08:31:56.696Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:31:56.759Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:31:56.787Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:56300"
time=2025-08-24T08:31:56.792Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:31:56.819Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:31:56.984Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:31:57.120Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:31:57.298Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:31:57.298Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU"
time=2025-08-24T08:31:57.298Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:31:57.298Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU"
time=2025-08-24T08:31:57.298Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB"
time=2025-08-24T08:31:57.298Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB"
time=2025-08-24T08:31:57.298Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB"
time=2025-08-24T08:31:57.298Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB"
time=2025-08-24T08:31:57.298Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB"
time=2025-08-24T08:31:57.298Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB"
time=2025-08-24T08:31:57.298Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:31:57.298Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:31:57.299Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:32:05.842Z level=INFO source=server.go:1272 msg="llama runner started in 9.16 seconds"
[GIN] 2025/08/24 - 08:32:11 | 200 |   14.8728921s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:32:14 | 200 |    3.5070745s |    192.168.1.17 | POST     "/api/generate"
time=2025-08-24T08:33:15.942Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:33:15.952Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:33:15.959Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 56330"
time=2025-08-24T08:33:15.962Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1
time=2025-08-24T08:33:15.971Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.5 GiB" free_swap="36.1 GiB"
time=2025-08-24T08:33:15.971Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:33:16.051Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:33:16.080Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:56330"
time=2025-08-24T08:33:16.089Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:33:16.116Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:33:16.289Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:33:16.471Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:33:16.764Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:33:16.764Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU"
time=2025-08-24T08:33:16.764Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:33:16.764Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU"
time=2025-08-24T08:33:16.764Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB"
time=2025-08-24T08:33:16.764Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB"
time=2025-08-24T08:33:16.764Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB"
time=2025-08-24T08:33:16.764Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB"
time=2025-08-24T08:33:16.764Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-08-24T08:33:16.764Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB"
time=2025-08-24T08:33:16.764Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:33:16.764Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:33:16.765Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:33:32.052Z level=INFO source=server.go:1272 msg="llama runner started in 16.09 seconds"
[GIN] 2025/08/24 - 08:33:35 | 200 |   20.2775093s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:33:38 | 200 |    2.9531411s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:33:42 | 200 |    3.7989308s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:33:45 | 200 |    2.1928378s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:33:48 | 200 |    3.7262304s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:33:56 | 200 |    6.8345257s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:33:58 | 200 |    2.2374723s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 08:34:06 | 200 |    7.6498364s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T08:34:06.995Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T08:34:07.005Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T08:34:07.012Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 56351"
time=2025-08-24T08:34:07.014Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1
time=2025-08-24T08:34:07.023Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.5 GiB" free_swap="36.1 GiB"
time=2025-08-24T08:34:07.023Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T08:34:07.094Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T08:34:07.126Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:56351"
time=2025-08-24T08:34:07.129Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:34:07.159Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T08:34:07.320Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T08:34:07.452Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:34:07.628Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T08:34:07.628Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU"
time=2025-08-24T08:34:07.628Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T08:34:07.628Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU"
time=2025-08-24T08:34:07.628Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB"
time=2025-08-24T08:34:07.628Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB"
time=2025-08-24T08:34:07.628Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB"
time=2025-08-24T08:34:07.628Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB"
time=2025-08-24T08:34:07.628Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB"
time=2025-08-24T08:34:07.628Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB"
time=2025-08-24T08:34:07.628Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T08:34:07.628Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T08:34:07.629Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T08:34:16.152Z level=INFO source=server.go:1272 msg="llama runner started in 9.14 seconds"
[GIN] 2025/08/24 - 08:34:20 | 200 |   14.3440295s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 08:34:24 | 200 |    3.6528641s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/24 - 09:20:46 | 200 |     733.919ms |    192.168.1.17 | GET      "/api/tags"
[GIN] 2025/08/24 - 09:20:49 | 200 |     90.2055ms |    192.168.1.17 | GET      "/api/tags"
time=2025-08-24T09:21:10.052Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-24T09:21:10.063Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-24T09:21:10.070Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 57113"
time=2025-08-24T09:21:10.073Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1
time=2025-08-24T09:21:10.083Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.6 GiB" free_swap="36.1 GiB"
time=2025-08-24T09:21:10.083Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-24T09:21:10.152Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-24T09:21:10.191Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:57113"
time=2025-08-24T09:21:10.194Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T09:21:10.218Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-24T09:21:10.406Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-24T09:21:10.548Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T09:21:10.857Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-24T09:21:10.857Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU"
time=2025-08-24T09:21:10.857Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-24T09:21:10.857Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU"
time=2025-08-24T09:21:10.857Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB"
time=2025-08-24T09:21:10.857Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB"
time=2025-08-24T09:21:10.857Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB"
time=2025-08-24T09:21:10.857Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB"
time=2025-08-24T09:21:10.857Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-08-24T09:21:10.857Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB"
time=2025-08-24T09:21:10.857Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-24T09:21:10.857Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding"
time=2025-08-24T09:21:10.857Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-24T09:21:28.147Z level=INFO source=server.go:1272 msg="llama runner started in 18.08 seconds"
time=2025-08-24T09:21:28.249Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17679 keep=4 new=9200
[GIN] 2025/08/24 - 09:21:44 | 200 |   34.5049141s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 09:21:46 | 200 |   34.7076766s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 09:21:48 | 200 |   35.3747928s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T09:21:48.355Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17679 keep=4 new=9200
[GIN] 2025/08/24 - 09:22:02 | 200 |   18.3972338s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 09:22:05 | 200 |   19.3592633s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 09:22:06 | 200 |   18.4042719s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T09:22:06.942Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17764 keep=4 new=9200
[GIN] 2025/08/24 - 09:22:20 | 200 |    17.642561s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 09:22:25 | 200 |   19.3262876s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 09:22:26 | 200 |   19.6994169s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-24T09:22:26.681Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17764 keep=4 new=9200
[GIN] 2025/08/24 - 09:22:43 | 200 |   22.4530331s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 09:22:47 | 200 |   22.4339148s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 09:22:49 | 200 |    22.764364s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 09:22:53 | 200 |    5.6252216s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 09:22:55 | 200 |    5.9669936s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 09:23:02 | 200 |    9.0069872s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 09:23:08 | 200 |   12.6981106s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 09:23:10 | 200 |      8.13966s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 09:23:13 | 200 |    4.6126592s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 09:23:20 | 200 |    9.4045521s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/24 - 09:23:26 | 200 |   12.9598829s |    192.168.1.17 | POST     "/api/chat"
<!-- gh-comment-id:3217941616 --> @mcr-ksh commented on GitHub (Aug 24, 2025): I've reinstalled and tested again and now it's working at least at the beginning. I've had the process killed and restarted multiple times but the loading didn't work and now it did for a few times until it gets stuck again For the sake of completeness the server log: ``` time=2025-08-24T08:07:32.191Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:07:32.201Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:07:32.207Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 64712" time=2025-08-24T08:07:32.209Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1 time=2025-08-24T08:07:32.219Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.6 GiB" free_swap="35.9 GiB" time=2025-08-24T08:07:32.219Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:07:32.289Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:07:32.324Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:64712" time=2025-08-24T08:07:32.334Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:07:32.360Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:07:32.529Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:07:32.677Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:07:32.970Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:07:32.971Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU" time=2025-08-24T08:07:32.971Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:07:32.971Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU" time=2025-08-24T08:07:32.971Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB" time=2025-08-24T08:07:32.971Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB" time=2025-08-24T08:07:32.971Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB" time=2025-08-24T08:07:32.971Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB" time=2025-08-24T08:07:32.971Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB" time=2025-08-24T08:07:32.971Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB" time=2025-08-24T08:07:32.971Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:07:32.971Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:07:32.972Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:07:50.268Z level=INFO source=server.go:1272 msg="llama runner started in 18.06 seconds" time=2025-08-24T08:07:50.374Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=21812 keep=4 new=9200 [GIN] 2025/08/24 - 08:08:06 | 200 | 34.7778902s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:08:07.017Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=21812 keep=4 new=9200 [GIN] 2025/08/24 - 08:08:24 | 200 | 17.9491192s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:08:25.156Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=21897 keep=4 new=9200 [GIN] 2025/08/24 - 08:08:40 | 200 | 15.6296422s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:08:40.988Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=21897 keep=4 new=9200 [GIN] 2025/08/24 - 08:08:57 | 200 | 17.0038725s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:08:58.469Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:08:58.480Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:08:58.488Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 64741" time=2025-08-24T08:08:58.491Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1 time=2025-08-24T08:08:58.500Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.4 GiB" free_swap="35.7 GiB" time=2025-08-24T08:08:58.500Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:08:58.569Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:08:58.599Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:64741" time=2025-08-24T08:08:58.610Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:08:58.642Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:08:58.829Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:08:58.974Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:08:59.150Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:08:59.151Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB" time=2025-08-24T08:08:59.151Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB" time=2025-08-24T08:08:59.151Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB" time=2025-08-24T08:08:59.151Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB" time=2025-08-24T08:08:59.151Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB" time=2025-08-24T08:08:59.151Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB" time=2025-08-24T08:08:59.151Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:08:59.151Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:08:59.151Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:08:59.152Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU" time=2025-08-24T08:08:59.152Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:08:59.152Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU" time=2025-08-24T08:09:08.928Z level=INFO source=server.go:1272 msg="llama runner started in 10.44 seconds" [GIN] 2025/08/24 - 08:12:10 | 200 | 3m13s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:12:32 | 200 | 21.061812s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:12:55 | 200 | 22.8297054s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:13:06 | 200 | 10.9356254s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:13:09 | 200 | 2.7860686s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:13:12 | 200 | 3.6968598s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:13:16 | 200 | 3.3646689s | 192.168.1.17 | POST "/api/generate" time=2025-08-24T08:13:22.116Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:13:22.126Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:13:22.133Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 64812" time=2025-08-24T08:13:22.136Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1 time=2025-08-24T08:13:22.144Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.7 GiB" free_swap="36.1 GiB" time=2025-08-24T08:13:22.144Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:13:22.214Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:13:22.247Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:64812" time=2025-08-24T08:13:22.251Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:13:22.275Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:13:22.431Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:13:22.631Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:13:22.929Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:13:22.929Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU" time=2025-08-24T08:13:22.929Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:13:22.929Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU" time=2025-08-24T08:13:22.930Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB" time=2025-08-24T08:13:22.930Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB" time=2025-08-24T08:13:22.930Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB" time=2025-08-24T08:13:22.930Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB" time=2025-08-24T08:13:22.930Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB" time=2025-08-24T08:13:22.930Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB" time=2025-08-24T08:13:22.930Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:13:22.930Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:13:22.930Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:13:38.974Z level=INFO source=server.go:1272 msg="llama runner started in 16.84 seconds" [GIN] 2025/08/24 - 08:13:42 | 200 | 20.8934571s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:13:44 | 200 | 1.8952135s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:13:49 | 200 | 4.2768875s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:13:52 | 200 | 3.0859709s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:13:55 | 200 | 2.730771s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:13:59 | 200 | 3.9742797s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:14:07 | 200 | 7.8356668s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:14:10 | 200 | 2.6263678s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:14:17 | 200 | 6.6245268s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:14:17.701Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:14:17.711Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:14:17.718Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 64835" time=2025-08-24T08:14:17.722Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1 time=2025-08-24T08:14:17.730Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.7 GiB" free_swap="36.1 GiB" time=2025-08-24T08:14:17.730Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:14:17.806Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:14:17.836Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:64835" time=2025-08-24T08:14:17.848Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:14:17.875Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:14:18.049Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:14:18.180Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:14:18.358Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:14:18.358Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU" time=2025-08-24T08:14:18.358Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:14:18.358Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU" time=2025-08-24T08:14:18.358Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB" time=2025-08-24T08:14:18.358Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB" time=2025-08-24T08:14:18.358Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB" time=2025-08-24T08:14:18.358Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB" time=2025-08-24T08:14:18.358Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB" time=2025-08-24T08:14:18.358Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB" time=2025-08-24T08:14:18.358Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:14:18.358Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:14:18.359Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:14:27.386Z level=INFO source=server.go:1272 msg="llama runner started in 9.67 seconds" [GIN] 2025/08/24 - 08:14:32 | 200 | 15.1381859s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:14:36 | 200 | 4.0328043s | 192.168.1.17 | POST "/api/generate" time=2025-08-24T08:17:38.311Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:17:38.320Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:17:38.328Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 64897" time=2025-08-24T08:17:38.330Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1 time=2025-08-24T08:17:38.339Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.6 GiB" free_swap="36.1 GiB" time=2025-08-24T08:17:38.340Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:17:38.408Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:17:38.441Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:64897" time=2025-08-24T08:17:38.447Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:17:38.472Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:17:38.634Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:17:38.837Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:17:39.150Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:17:39.150Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU" time=2025-08-24T08:17:39.150Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:17:39.150Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU" time=2025-08-24T08:17:39.151Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB" time=2025-08-24T08:17:39.151Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB" time=2025-08-24T08:17:39.151Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB" time=2025-08-24T08:17:39.151Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB" time=2025-08-24T08:17:39.151Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB" time=2025-08-24T08:17:39.151Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB" time=2025-08-24T08:17:39.151Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:17:39.151Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:17:39.151Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:17:55.198Z level=INFO source=server.go:1272 msg="llama runner started in 16.87 seconds" time=2025-08-24T08:17:55.303Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17679 keep=4 new=9200 [GIN] 2025/08/24 - 08:18:11 | 200 | 33.8854044s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:18:11.711Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=21812 keep=4 new=9200 [GIN] 2025/08/24 - 08:18:24 | 200 | 46.0562294s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:18:24.949Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17679 keep=4 new=9200 [GIN] 2025/08/24 - 08:18:38 | 200 | 26.6680352s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:18:38.538Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=21812 keep=4 new=9200 [GIN] 2025/08/24 - 08:18:55 | 200 | 30.6227418s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:18:55.749Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17764 keep=4 new=9200 [GIN] 2025/08/24 - 08:19:11 | 200 | 32.348392s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:19:11.047Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=21897 keep=4 new=9200 [GIN] 2025/08/24 - 08:19:27 | 200 | 31.8002034s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:19:27.736Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17764 keep=4 new=9200 [GIN] 2025/08/24 - 08:19:44 | 200 | 33.1371324s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:19:44.408Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=21897 keep=4 new=9200 [GIN] 2025/08/24 - 08:20:02 | 200 | 34.2558054s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:20:02.540Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:20:02.550Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:20:02.557Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 64973" time=2025-08-24T08:20:02.559Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1 time=2025-08-24T08:20:02.571Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.7 GiB" free_swap="36.2 GiB" time=2025-08-24T08:20:02.571Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:20:02.637Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:20:02.666Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:64973" time=2025-08-24T08:20:02.677Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:20:02.709Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:20:02.866Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:20:03.014Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:20:03.208Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:20:03.208Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU" time=2025-08-24T08:20:03.208Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:20:03.208Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU" time=2025-08-24T08:20:03.208Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB" time=2025-08-24T08:20:03.208Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB" time=2025-08-24T08:20:03.208Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB" time=2025-08-24T08:20:03.208Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB" time=2025-08-24T08:20:03.208Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB" time=2025-08-24T08:20:03.208Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB" time=2025-08-24T08:20:03.208Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:20:03.208Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:20:03.210Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:20:11.980Z level=INFO source=server.go:1272 msg="llama runner started in 9.42 seconds" [GIN] 2025/08/24 - 08:20:21 | 200 | 37.4275719s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:20:44 | 200 | 42.3701852s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:20:49 | 200 | 27.9177488s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:21:00 | 200 | 15.4601114s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:21:14 | 200 | 24.0274647s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:21:23 | 200 | 23.3699805s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:21:29 | 200 | 14.910289s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:21:36 | 200 | 12.79915s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:21:39 | 200 | 3.0346262s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:21:42 | 200 | 2.6803838s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:21:45 | 200 | 2.9273598s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:21:48 | 200 | 2.7270342s | 192.168.1.17 | POST "/api/generate" time=2025-08-24T08:22:19.262Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:22:19.272Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:22:19.280Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 65059" time=2025-08-24T08:22:19.282Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1 time=2025-08-24T08:22:19.291Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.3 GiB" free_swap="35.8 GiB" time=2025-08-24T08:22:19.291Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:22:19.362Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:22:19.395Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65059" time=2025-08-24T08:22:19.397Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:22:19.420Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:22:19.609Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:22:19.758Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:22:20.049Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:22:20.050Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU" time=2025-08-24T08:22:20.050Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:22:20.050Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU" time=2025-08-24T08:22:20.050Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB" time=2025-08-24T08:22:20.050Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB" time=2025-08-24T08:22:20.050Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB" time=2025-08-24T08:22:20.050Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB" time=2025-08-24T08:22:20.050Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB" time=2025-08-24T08:22:20.050Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB" time=2025-08-24T08:22:20.050Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:22:20.050Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:22:20.050Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:22:35.591Z level=INFO source=server.go:1272 msg="llama runner started in 16.31 seconds" time=2025-08-24T08:22:37.120Z level=ERROR source=server.go:1442 msg="post predict" error="Post \"http://127.0.0.1:65059/completion\": context canceled" [GIN] 2025/08/24 - 08:22:37 | 200 | 18.3661583s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:23:52.324Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17679 keep=4 new=9200 [GIN] 2025/08/24 - 08:24:06 | 200 | 14.6810069s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:24:11 | 200 | 19.1207952s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:24:11.405Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17679 keep=4 new=9200 [GIN] 2025/08/24 - 08:24:27 | 200 | 20.1823008s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:24:30 | 200 | 18.6093555s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:24:30.277Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17764 keep=4 new=9200 [GIN] 2025/08/24 - 08:24:47 | 200 | 20.4410958s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:24:51 | 200 | 20.913231s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:24:51.227Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17764 keep=4 new=9200 [GIN] 2025/08/24 - 08:25:08 | 200 | 20.4097597s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:25:13 | 200 | 21.751285s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:25:13.506Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:25:13.515Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:25:13.523Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 65120" time=2025-08-24T08:25:13.526Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1 time=2025-08-24T08:25:13.535Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.3 GiB" free_swap="35.8 GiB" time=2025-08-24T08:25:13.535Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:25:13.606Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:25:13.639Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65120" time=2025-08-24T08:25:13.642Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:25:13.668Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:25:13.825Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:25:13.975Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:25:14.156Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:25:14.156Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU" time=2025-08-24T08:25:14.156Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:25:14.156Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU" time=2025-08-24T08:25:14.156Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB" time=2025-08-24T08:25:14.156Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB" time=2025-08-24T08:25:14.156Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB" time=2025-08-24T08:25:14.156Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB" time=2025-08-24T08:25:14.156Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB" time=2025-08-24T08:25:14.156Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB" time=2025-08-24T08:25:14.156Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:25:14.156Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:25:14.156Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:25:22.438Z level=INFO source=server.go:1272 msg="llama runner started in 8.92 seconds" [GIN] 2025/08/24 - 08:25:38 | 200 | 29.7982689s | 192.168.1.17 | POST "/api/generate" time=2025-08-24T08:25:38.751Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:25:38.761Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:25:38.768Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 65136" time=2025-08-24T08:25:38.770Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1 time=2025-08-24T08:25:38.780Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.4 GiB" free_swap="35.8 GiB" time=2025-08-24T08:25:38.780Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:25:38.851Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:25:38.880Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65136" time=2025-08-24T08:25:38.883Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:25:38.906Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:25:39.068Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:25:39.207Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:25:39.469Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:25:39.469Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU" time=2025-08-24T08:25:39.469Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:25:39.469Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU" time=2025-08-24T08:25:39.469Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB" time=2025-08-24T08:25:39.469Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB" time=2025-08-24T08:25:39.469Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB" time=2025-08-24T08:25:39.469Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB" time=2025-08-24T08:25:39.469Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB" time=2025-08-24T08:25:39.469Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB" time=2025-08-24T08:25:39.470Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:25:39.470Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:25:39.470Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:25:54.508Z level=INFO source=server.go:1272 msg="llama runner started in 15.74 seconds" [GIN] 2025/08/24 - 08:25:59 | 200 | 45.9285195s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:25:59.660Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:25:59.671Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:25:59.678Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 65150" time=2025-08-24T08:25:59.682Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1 time=2025-08-24T08:25:59.694Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.4 GiB" free_swap="35.8 GiB" time=2025-08-24T08:25:59.694Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:25:59.760Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:25:59.790Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65150" time=2025-08-24T08:25:59.800Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:25:59.828Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:25:59.983Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:26:00.134Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:26:00.309Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:26:00.310Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU" time=2025-08-24T08:26:00.310Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:26:00.310Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU" time=2025-08-24T08:26:00.310Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB" time=2025-08-24T08:26:00.310Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB" time=2025-08-24T08:26:00.310Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB" time=2025-08-24T08:26:00.310Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB" time=2025-08-24T08:26:00.310Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB" time=2025-08-24T08:26:00.310Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB" time=2025-08-24T08:26:00.310Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:26:00.310Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:26:00.311Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:26:08.833Z level=INFO source=server.go:1272 msg="llama runner started in 9.16 seconds" [GIN] 2025/08/24 - 08:26:19 | 200 | 40.5837591s | 192.168.1.17 | POST "/api/generate" time=2025-08-24T08:26:19.474Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:26:19.484Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:26:19.492Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 65165" time=2025-08-24T08:26:19.495Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1 time=2025-08-24T08:26:19.503Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.3 GiB" free_swap="35.8 GiB" time=2025-08-24T08:26:19.503Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:26:19.571Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:26:19.603Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65165" time=2025-08-24T08:26:19.609Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:26:19.633Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:26:19.793Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:26:19.931Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:26:20.203Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:26:20.203Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU" time=2025-08-24T08:26:20.203Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:26:20.203Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU" time=2025-08-24T08:26:20.203Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB" time=2025-08-24T08:26:20.203Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB" time=2025-08-24T08:26:20.203Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB" time=2025-08-24T08:26:20.203Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB" time=2025-08-24T08:26:20.203Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB" time=2025-08-24T08:26:20.203Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB" time=2025-08-24T08:26:20.203Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:26:20.203Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:26:20.204Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:26:35.498Z level=INFO source=server.go:1272 msg="llama runner started in 16.01 seconds" [GIN] 2025/08/24 - 08:26:44 | 200 | 44.6729655s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:26:44.439Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:26:44.449Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:26:44.458Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 65186" time=2025-08-24T08:26:44.460Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1 time=2025-08-24T08:26:44.469Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.2 GiB" free_swap="35.7 GiB" time=2025-08-24T08:26:44.469Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:26:44.539Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:26:44.577Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65186" time=2025-08-24T08:26:44.579Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:26:44.613Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:26:44.781Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:26:44.911Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:26:45.110Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:26:45.111Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU" time=2025-08-24T08:26:45.111Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:26:45.111Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU" time=2025-08-24T08:26:45.111Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB" time=2025-08-24T08:26:45.111Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB" time=2025-08-24T08:26:45.111Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB" time=2025-08-24T08:26:45.111Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB" time=2025-08-24T08:26:45.111Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB" time=2025-08-24T08:26:45.111Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB" time=2025-08-24T08:26:45.111Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:26:45.111Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:26:45.111Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:26:53.138Z level=INFO source=server.go:1272 msg="llama runner started in 8.68 seconds" [GIN] 2025/08/24 - 08:27:04 | 200 | 45.4527574s | 192.168.1.17 | POST "/api/generate" time=2025-08-24T08:27:05.032Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:27:05.042Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:27:05.049Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 65202" time=2025-08-24T08:27:05.051Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1 time=2025-08-24T08:27:05.061Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.6 GiB" free_swap="35.8 GiB" time=2025-08-24T08:27:05.061Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:27:05.132Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:27:05.163Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65202" time=2025-08-24T08:27:05.167Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:27:05.192Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:27:05.351Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:27:05.495Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:27:05.754Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:27:05.755Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU" time=2025-08-24T08:27:05.755Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:27:05.755Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU" time=2025-08-24T08:27:05.755Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB" time=2025-08-24T08:27:05.755Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB" time=2025-08-24T08:27:05.755Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB" time=2025-08-24T08:27:05.755Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB" time=2025-08-24T08:27:05.755Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB" time=2025-08-24T08:27:05.755Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB" time=2025-08-24T08:27:05.755Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:27:05.755Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:27:05.755Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:27:21.295Z level=INFO source=server.go:1272 msg="llama runner started in 16.25 seconds" [GIN] 2025/08/24 - 08:27:28 | 200 | 43.9324582s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:27:28.824Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:27:28.835Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:27:28.842Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 65220" time=2025-08-24T08:27:28.845Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1 time=2025-08-24T08:27:28.856Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.7 GiB" free_swap="35.9 GiB" time=2025-08-24T08:27:28.856Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:27:28.923Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:27:28.952Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65220" time=2025-08-24T08:27:28.962Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:27:28.990Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:27:29.161Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:27:29.367Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:27:29.546Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:27:29.546Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU" time=2025-08-24T08:27:29.546Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:27:29.546Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU" time=2025-08-24T08:27:29.546Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB" time=2025-08-24T08:27:29.546Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB" time=2025-08-24T08:27:29.546Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB" time=2025-08-24T08:27:29.546Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB" time=2025-08-24T08:27:29.546Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB" time=2025-08-24T08:27:29.546Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB" time=2025-08-24T08:27:29.546Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:27:29.546Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:27:29.546Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:27:37.573Z level=INFO source=server.go:1272 msg="llama runner started in 8.73 seconds" [GIN] 2025/08/24 - 08:27:42 | 200 | 37.5892486s | 192.168.1.17 | POST "/api/generate" time=2025-08-24T08:27:42.742Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:27:42.751Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:27:42.759Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 65233" time=2025-08-24T08:27:42.761Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1 time=2025-08-24T08:27:42.773Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.7 GiB" free_swap="35.9 GiB" time=2025-08-24T08:27:42.773Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:27:42.840Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:27:42.870Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65233" time=2025-08-24T08:27:42.877Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:27:42.901Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:27:43.066Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:27:43.209Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:27:43.465Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:27:43.465Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU" time=2025-08-24T08:27:43.465Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:27:43.465Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU" time=2025-08-24T08:27:43.466Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB" time=2025-08-24T08:27:43.466Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB" time=2025-08-24T08:27:43.466Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB" time=2025-08-24T08:27:43.466Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB" time=2025-08-24T08:27:43.466Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB" time=2025-08-24T08:27:43.466Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB" time=2025-08-24T08:27:43.466Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:27:43.467Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:27:43.471Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:27:47.479Z level=INFO source=server.go:1272 msg="llama runner started in 4.72 seconds" [GIN] 2025/08/24 - 08:27:56 | 200 | 28.0871306s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:28:02 | 200 | 5.697337s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:28:03.227Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:28:03.237Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:28:03.244Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 65246" time=2025-08-24T08:28:03.246Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1 time=2025-08-24T08:28:03.256Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.6 GiB" free_swap="35.9 GiB" time=2025-08-24T08:28:03.256Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:28:03.335Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:28:03.367Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:65246" time=2025-08-24T08:28:03.373Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:28:03.420Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:28:03.571Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:28:03.702Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:28:03.885Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:28:03.886Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU" time=2025-08-24T08:28:03.886Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:28:03.886Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU" time=2025-08-24T08:28:03.886Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB" time=2025-08-24T08:28:03.886Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB" time=2025-08-24T08:28:03.886Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB" time=2025-08-24T08:28:03.886Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB" time=2025-08-24T08:28:03.886Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB" time=2025-08-24T08:28:03.886Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB" time=2025-08-24T08:28:03.886Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:28:03.886Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:28:03.887Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:28:12.657Z level=INFO source=server.go:1272 msg="llama runner started in 9.41 seconds" [GIN] 2025/08/24 - 08:28:17 | 200 | 14.7165468s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:28:22 | 200 | 4.6953768s | 192.168.1.17 | POST "/api/generate" time=2025-08-24T08:31:04.363Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:31:04.373Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:31:04.381Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 56277" time=2025-08-24T08:31:04.384Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1 time=2025-08-24T08:31:04.394Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.6 GiB" free_swap="35.8 GiB" time=2025-08-24T08:31:04.394Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:31:04.467Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:31:04.498Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:56277" time=2025-08-24T08:31:04.509Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:31:04.534Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:31:04.736Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:31:04.873Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:31:05.174Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:31:05.175Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU" time=2025-08-24T08:31:05.175Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:31:05.175Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU" time=2025-08-24T08:31:05.175Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB" time=2025-08-24T08:31:05.175Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB" time=2025-08-24T08:31:05.175Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB" time=2025-08-24T08:31:05.175Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB" time=2025-08-24T08:31:05.175Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB" time=2025-08-24T08:31:05.175Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB" time=2025-08-24T08:31:05.175Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:31:05.175Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:31:05.175Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:31:20.210Z level=INFO source=server.go:1272 msg="llama runner started in 15.83 seconds" [GIN] 2025/08/24 - 08:31:23 | 200 | 19.9755304s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:31:27 | 200 | 3.2055066s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:31:33 | 200 | 3.9072038s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:31:36 | 200 | 2.5509373s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:31:40 | 200 | 3.9435223s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:31:47 | 200 | 6.3619846s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:31:49 | 200 | 2.6050403s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:31:56 | 200 | 5.8860515s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:31:56.663Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:31:56.672Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:31:56.680Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 56300" time=2025-08-24T08:31:56.682Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1 time=2025-08-24T08:31:56.696Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.6 GiB" free_swap="35.9 GiB" time=2025-08-24T08:31:56.696Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:31:56.759Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:31:56.787Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:56300" time=2025-08-24T08:31:56.792Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:31:56.819Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:31:56.984Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:31:57.120Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:31:57.298Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:31:57.298Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU" time=2025-08-24T08:31:57.298Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:31:57.298Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU" time=2025-08-24T08:31:57.298Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB" time=2025-08-24T08:31:57.298Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB" time=2025-08-24T08:31:57.298Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB" time=2025-08-24T08:31:57.298Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB" time=2025-08-24T08:31:57.298Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB" time=2025-08-24T08:31:57.298Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB" time=2025-08-24T08:31:57.298Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:31:57.298Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:31:57.299Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:32:05.842Z level=INFO source=server.go:1272 msg="llama runner started in 9.16 seconds" [GIN] 2025/08/24 - 08:32:11 | 200 | 14.8728921s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:32:14 | 200 | 3.5070745s | 192.168.1.17 | POST "/api/generate" time=2025-08-24T08:33:15.942Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:33:15.952Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:33:15.959Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 56330" time=2025-08-24T08:33:15.962Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1 time=2025-08-24T08:33:15.971Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.5 GiB" free_swap="36.1 GiB" time=2025-08-24T08:33:15.971Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:33:16.051Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:33:16.080Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:56330" time=2025-08-24T08:33:16.089Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:33:16.116Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:33:16.289Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:33:16.471Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:33:16.764Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:33:16.764Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU" time=2025-08-24T08:33:16.764Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:33:16.764Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU" time=2025-08-24T08:33:16.764Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB" time=2025-08-24T08:33:16.764Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB" time=2025-08-24T08:33:16.764Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB" time=2025-08-24T08:33:16.764Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB" time=2025-08-24T08:33:16.764Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB" time=2025-08-24T08:33:16.764Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB" time=2025-08-24T08:33:16.764Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:33:16.764Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:33:16.765Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:33:32.052Z level=INFO source=server.go:1272 msg="llama runner started in 16.09 seconds" [GIN] 2025/08/24 - 08:33:35 | 200 | 20.2775093s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:33:38 | 200 | 2.9531411s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:33:42 | 200 | 3.7989308s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:33:45 | 200 | 2.1928378s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:33:48 | 200 | 3.7262304s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:33:56 | 200 | 6.8345257s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:33:58 | 200 | 2.2374723s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 08:34:06 | 200 | 7.6498364s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T08:34:06.995Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T08:34:07.005Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T08:34:07.012Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 56351" time=2025-08-24T08:34:07.014Z level=INFO source=server.go:659 msg="loading model" "model layers"=33 requested=-1 time=2025-08-24T08:34:07.023Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.5 GiB" free_swap="36.1 GiB" time=2025-08-24T08:34:07.023Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T08:34:07.094Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T08:34:07.126Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:56351" time=2025-08-24T08:34:07.129Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:34:07.159Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T08:34:07.320Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T08:34:07.452Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:34:07.628Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T08:34:07.628Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU" time=2025-08-24T08:34:07.628Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T08:34:07.628Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU" time=2025-08-24T08:34:07.628Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB" time=2025-08-24T08:34:07.628Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB" time=2025-08-24T08:34:07.628Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB" time=2025-08-24T08:34:07.628Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB" time=2025-08-24T08:34:07.628Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB" time=2025-08-24T08:34:07.628Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB" time=2025-08-24T08:34:07.628Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T08:34:07.628Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T08:34:07.629Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T08:34:16.152Z level=INFO source=server.go:1272 msg="llama runner started in 9.14 seconds" [GIN] 2025/08/24 - 08:34:20 | 200 | 14.3440295s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 08:34:24 | 200 | 3.6528641s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/24 - 09:20:46 | 200 | 733.919ms | 192.168.1.17 | GET "/api/tags" [GIN] 2025/08/24 - 09:20:49 | 200 | 90.2055ms | 192.168.1.17 | GET "/api/tags" time=2025-08-24T09:21:10.052Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-24T09:21:10.063Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-24T09:21:10.070Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 57113" time=2025-08-24T09:21:10.073Z level=INFO source=server.go:659 msg="loading model" "model layers"=49 requested=-1 time=2025-08-24T09:21:10.083Z level=INFO source=server.go:665 msg="system memory" total="48.0 GiB" free="35.6 GiB" free_swap="36.1 GiB" time=2025-08-24T09:21:10.083Z level=INFO source=server.go:669 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-24T09:21:10.152Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-24T09:21:10.191Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:57113" time=2025-08-24T09:21:10.194Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T09:21:10.218Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-24T09:21:10.406Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-24T09:21:10.548Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T09:21:10.857Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-24T09:21:10.857Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU" time=2025-08-24T09:21:10.857Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-24T09:21:10.857Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU" time=2025-08-24T09:21:10.857Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB" time=2025-08-24T09:21:10.857Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB" time=2025-08-24T09:21:10.857Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB" time=2025-08-24T09:21:10.857Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB" time=2025-08-24T09:21:10.857Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB" time=2025-08-24T09:21:10.857Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB" time=2025-08-24T09:21:10.857Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-24T09:21:10.857Z level=INFO source=server.go:1234 msg="waiting for llama runner to start responding" time=2025-08-24T09:21:10.857Z level=INFO source=server.go:1268 msg="waiting for server to become available" status="llm server loading model" time=2025-08-24T09:21:28.147Z level=INFO source=server.go:1272 msg="llama runner started in 18.08 seconds" time=2025-08-24T09:21:28.249Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17679 keep=4 new=9200 [GIN] 2025/08/24 - 09:21:44 | 200 | 34.5049141s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 09:21:46 | 200 | 34.7076766s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 09:21:48 | 200 | 35.3747928s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T09:21:48.355Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17679 keep=4 new=9200 [GIN] 2025/08/24 - 09:22:02 | 200 | 18.3972338s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 09:22:05 | 200 | 19.3592633s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 09:22:06 | 200 | 18.4042719s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T09:22:06.942Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17764 keep=4 new=9200 [GIN] 2025/08/24 - 09:22:20 | 200 | 17.642561s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 09:22:25 | 200 | 19.3262876s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 09:22:26 | 200 | 19.6994169s | 192.168.1.17 | POST "/api/chat" time=2025-08-24T09:22:26.681Z level=WARN source=runner.go:158 msg="truncating input prompt" limit=9200 prompt=17764 keep=4 new=9200 [GIN] 2025/08/24 - 09:22:43 | 200 | 22.4530331s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 09:22:47 | 200 | 22.4339148s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 09:22:49 | 200 | 22.764364s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 09:22:53 | 200 | 5.6252216s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 09:22:55 | 200 | 5.9669936s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 09:23:02 | 200 | 9.0069872s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 09:23:08 | 200 | 12.6981106s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 09:23:10 | 200 | 8.13966s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 09:23:13 | 200 | 4.6126592s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 09:23:20 | 200 | 9.4045521s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/24 - 09:23:26 | 200 | 12.9598829s | 192.168.1.17 | POST "/api/chat" ```
GiteaMirror commented 2026-04-12 20:10:51 -05:00
Author
Owner
Copy Link

@mcr-ksh commented on GitHub (Aug 26, 2025):

Here now with 11.6 the same Problem:

time=2025-08-26T08:02:24.903Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-26T08:02:24.914Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-26T08:02:24.921Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 51825"
time=2025-08-26T08:02:24.922Z level=INFO source=server.go:656 msg="loading model" "model layers"=49 requested=-1
time=2025-08-26T08:02:24.931Z level=INFO source=server.go:662 msg="system memory" total="48.0 GiB" free="37.6 GiB" free_swap="38.0 GiB"
time=2025-08-26T08:02:24.931Z level=INFO source=server.go:666 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="8.6 GiB" free="9.0 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-26T08:02:25.023Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-26T08:02:25.060Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:51825"
time=2025-08-26T08:02:25.069Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-26T08:02:25.110Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-26T08:02:25.343Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-26T08:02:25.605Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-26T08:02:25.927Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-26T08:02:25.928Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU"
time=2025-08-26T08:02:25.928Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-26T08:02:25.928Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU"
time=2025-08-26T08:02:25.928Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB"
time=2025-08-26T08:02:25.928Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB"
time=2025-08-26T08:02:25.928Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB"
time=2025-08-26T08:02:25.928Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB"
time=2025-08-26T08:02:25.928Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-08-26T08:02:25.928Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB"
time=2025-08-26T08:02:25.928Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-26T08:02:25.928Z level=INFO source=server.go:1231 msg="waiting for llama runner to start responding"
time=2025-08-26T08:02:25.928Z level=INFO source=server.go:1265 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-26T08:03:05.046Z level=INFO source=server.go:1269 msg="llama runner started in 40.12 seconds"
[GIN] 2025/08/26 - 08:03:06 | 200 |   42.4961426s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/26 - 08:03:08 | 200 |    1.2075999s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/26 - 08:03:09 | 200 |    967.5473ms |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/26 - 08:03:12 | 200 |    2.7365714s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/26 - 08:03:14 | 200 |    2.7190384s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/26 - 08:03:16 | 200 |    1.0703386s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/26 - 08:03:19 | 200 |    3.1117582s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/26 - 08:03:22 | 200 |    3.0295867s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/26 - 08:03:27 | 200 |    5.4446743s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/26 - 08:03:30 | 200 |    2.8743713s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/26 - 08:03:33 | 200 |    2.4033308s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/26 - 08:03:38 | 200 |    4.9939526s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/26 - 08:03:43 | 200 |    5.0560281s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/26 - 08:03:47 | 200 |    3.7502583s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-26T08:03:48.070Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-26T08:03:48.080Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-26T08:03:48.091Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 51857"
time=2025-08-26T08:03:48.096Z level=INFO source=server.go:656 msg="loading model" "model layers"=33 requested=-1
time=2025-08-26T08:03:48.106Z level=INFO source=server.go:662 msg="system memory" total="48.0 GiB" free="37.5 GiB" free_swap="37.9 GiB"
time=2025-08-26T08:03:48.106Z level=INFO source=server.go:666 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="8.6 GiB" free="9.0 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-26T08:03:48.199Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-26T08:03:48.218Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:51857"
time=2025-08-26T08:03:48.226Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-26T08:03:48.274Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-26T08:03:48.517Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-26T08:03:48.695Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-26T08:03:48.884Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-26T08:03:48.884Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU"
time=2025-08-26T08:03:48.884Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-26T08:03:48.884Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU"
time=2025-08-26T08:03:48.884Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB"
time=2025-08-26T08:03:48.884Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB"
time=2025-08-26T08:03:48.884Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB"
time=2025-08-26T08:03:48.884Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB"
time=2025-08-26T08:03:48.884Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB"
time=2025-08-26T08:03:48.884Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB"
time=2025-08-26T08:03:48.885Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-26T08:03:48.885Z level=INFO source=server.go:1231 msg="waiting for llama runner to start responding"
time=2025-08-26T08:03:48.885Z level=INFO source=server.go:1265 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-26T08:04:09.462Z level=INFO source=server.go:1269 msg="llama runner started in 21.37 seconds"
[GIN] 2025/08/26 - 08:04:13 | 200 |   26.2388548s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/26 - 08:04:18 | 200 |    4.9124216s |    192.168.1.17 | POST     "/api/generate"
time=2025-08-26T08:04:19.307Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-26T08:04:19.318Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-26T08:04:19.328Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 51878"
time=2025-08-26T08:04:19.332Z level=INFO source=server.go:656 msg="loading model" "model layers"=49 requested=-1
time=2025-08-26T08:04:19.351Z level=INFO source=server.go:662 msg="system memory" total="48.0 GiB" free="37.5 GiB" free_swap="37.8 GiB"
time=2025-08-26T08:04:19.351Z level=INFO source=server.go:666 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="8.6 GiB" free="9.0 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-26T08:04:19.463Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-26T08:04:19.491Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:51878"
time=2025-08-26T08:04:19.496Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-26T08:04:19.527Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-26T08:04:19.692Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-26T08:04:19.862Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-26T08:04:20.134Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-26T08:04:20.134Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU"
time=2025-08-26T08:04:20.134Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-26T08:04:20.134Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU"
time=2025-08-26T08:04:20.134Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB"
time=2025-08-26T08:04:20.134Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB"
time=2025-08-26T08:04:20.134Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB"
time=2025-08-26T08:04:20.134Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB"
time=2025-08-26T08:04:20.134Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-08-26T08:04:20.134Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB"
time=2025-08-26T08:04:20.134Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-26T08:04:20.134Z level=INFO source=server.go:1231 msg="waiting for llama runner to start responding"
time=2025-08-26T08:04:20.135Z level=INFO source=server.go:1265 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-26T08:05:04.247Z level=INFO source=server.go:1269 msg="llama runner started in 44.92 seconds"
[GIN] 2025/08/26 - 08:05:09 | 200 |   50.8547151s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/26 - 08:05:14 | 200 |   55.3683063s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-26T08:09:19.429Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection"
[GIN] 2025/08/26 - 08:09:19 | 200 |          5m0s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-26T08:10:10.382Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection"
[GIN] 2025/08/26 - 08:10:10 | 200 |          5m0s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/26 - 08:10:18 | 200 |          6m0s |    192.168.1.17 | POST     "/api/chat"
<!-- gh-comment-id:3223099124 --> @mcr-ksh commented on GitHub (Aug 26, 2025): Here now with 11.6 the same Problem: ``` time=2025-08-26T08:02:24.903Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-26T08:02:24.914Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-26T08:02:24.921Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 51825" time=2025-08-26T08:02:24.922Z level=INFO source=server.go:656 msg="loading model" "model layers"=49 requested=-1 time=2025-08-26T08:02:24.931Z level=INFO source=server.go:662 msg="system memory" total="48.0 GiB" free="37.6 GiB" free_swap="38.0 GiB" time=2025-08-26T08:02:24.931Z level=INFO source=server.go:666 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="8.6 GiB" free="9.0 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-26T08:02:25.023Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-26T08:02:25.060Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:51825" time=2025-08-26T08:02:25.069Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-26T08:02:25.110Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-26T08:02:25.343Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-26T08:02:25.605Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-26T08:02:25.927Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-26T08:02:25.928Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU" time=2025-08-26T08:02:25.928Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-26T08:02:25.928Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU" time=2025-08-26T08:02:25.928Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB" time=2025-08-26T08:02:25.928Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB" time=2025-08-26T08:02:25.928Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB" time=2025-08-26T08:02:25.928Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB" time=2025-08-26T08:02:25.928Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB" time=2025-08-26T08:02:25.928Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB" time=2025-08-26T08:02:25.928Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-26T08:02:25.928Z level=INFO source=server.go:1231 msg="waiting for llama runner to start responding" time=2025-08-26T08:02:25.928Z level=INFO source=server.go:1265 msg="waiting for server to become available" status="llm server loading model" time=2025-08-26T08:03:05.046Z level=INFO source=server.go:1269 msg="llama runner started in 40.12 seconds" [GIN] 2025/08/26 - 08:03:06 | 200 | 42.4961426s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/26 - 08:03:08 | 200 | 1.2075999s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/26 - 08:03:09 | 200 | 967.5473ms | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/26 - 08:03:12 | 200 | 2.7365714s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/26 - 08:03:14 | 200 | 2.7190384s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/26 - 08:03:16 | 200 | 1.0703386s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/26 - 08:03:19 | 200 | 3.1117582s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/26 - 08:03:22 | 200 | 3.0295867s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/26 - 08:03:27 | 200 | 5.4446743s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/26 - 08:03:30 | 200 | 2.8743713s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/26 - 08:03:33 | 200 | 2.4033308s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/26 - 08:03:38 | 200 | 4.9939526s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/26 - 08:03:43 | 200 | 5.0560281s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/26 - 08:03:47 | 200 | 3.7502583s | 192.168.1.17 | POST "/api/chat" time=2025-08-26T08:03:48.070Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-26T08:03:48.080Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-26T08:03:48.091Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 51857" time=2025-08-26T08:03:48.096Z level=INFO source=server.go:656 msg="loading model" "model layers"=33 requested=-1 time=2025-08-26T08:03:48.106Z level=INFO source=server.go:662 msg="system memory" total="48.0 GiB" free="37.5 GiB" free_swap="37.9 GiB" time=2025-08-26T08:03:48.106Z level=INFO source=server.go:666 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="8.6 GiB" free="9.0 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-26T08:03:48.199Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-26T08:03:48.218Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:51857" time=2025-08-26T08:03:48.226Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-26T08:03:48.274Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-26T08:03:48.517Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-26T08:03:48.695Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-26T08:03:48.884Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-26T08:03:48.884Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU" time=2025-08-26T08:03:48.884Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-26T08:03:48.884Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU" time=2025-08-26T08:03:48.884Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB" time=2025-08-26T08:03:48.884Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB" time=2025-08-26T08:03:48.884Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB" time=2025-08-26T08:03:48.884Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB" time=2025-08-26T08:03:48.884Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB" time=2025-08-26T08:03:48.884Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB" time=2025-08-26T08:03:48.885Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-26T08:03:48.885Z level=INFO source=server.go:1231 msg="waiting for llama runner to start responding" time=2025-08-26T08:03:48.885Z level=INFO source=server.go:1265 msg="waiting for server to become available" status="llm server loading model" time=2025-08-26T08:04:09.462Z level=INFO source=server.go:1269 msg="llama runner started in 21.37 seconds" [GIN] 2025/08/26 - 08:04:13 | 200 | 26.2388548s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/26 - 08:04:18 | 200 | 4.9124216s | 192.168.1.17 | POST "/api/generate" time=2025-08-26T08:04:19.307Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-26T08:04:19.318Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-26T08:04:19.328Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 51878" time=2025-08-26T08:04:19.332Z level=INFO source=server.go:656 msg="loading model" "model layers"=49 requested=-1 time=2025-08-26T08:04:19.351Z level=INFO source=server.go:662 msg="system memory" total="48.0 GiB" free="37.5 GiB" free_swap="37.8 GiB" time=2025-08-26T08:04:19.351Z level=INFO source=server.go:666 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="8.6 GiB" free="9.0 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-26T08:04:19.463Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-26T08:04:19.491Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:51878" time=2025-08-26T08:04:19.496Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-26T08:04:19.527Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-26T08:04:19.692Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-26T08:04:19.862Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-26T08:04:20.134Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-26T08:04:20.134Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU" time=2025-08-26T08:04:20.134Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-26T08:04:20.134Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU" time=2025-08-26T08:04:20.134Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB" time=2025-08-26T08:04:20.134Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB" time=2025-08-26T08:04:20.134Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB" time=2025-08-26T08:04:20.134Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB" time=2025-08-26T08:04:20.134Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB" time=2025-08-26T08:04:20.134Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB" time=2025-08-26T08:04:20.134Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-26T08:04:20.134Z level=INFO source=server.go:1231 msg="waiting for llama runner to start responding" time=2025-08-26T08:04:20.135Z level=INFO source=server.go:1265 msg="waiting for server to become available" status="llm server loading model" time=2025-08-26T08:05:04.247Z level=INFO source=server.go:1269 msg="llama runner started in 44.92 seconds" [GIN] 2025/08/26 - 08:05:09 | 200 | 50.8547151s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/26 - 08:05:14 | 200 | 55.3683063s | 192.168.1.17 | POST "/api/chat" time=2025-08-26T08:09:19.429Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection" [GIN] 2025/08/26 - 08:09:19 | 200 | 5m0s | 192.168.1.17 | POST "/api/chat" time=2025-08-26T08:10:10.382Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection" [GIN] 2025/08/26 - 08:10:10 | 200 | 5m0s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/26 - 08:10:18 | 200 | 6m0s | 192.168.1.17 | POST "/api/chat" ```
GiteaMirror commented 2026-04-12 20:10:52 -05:00
Author
Owner
Copy Link

@mcr-ksh commented on GitHub (Aug 26, 2025):

another day another hang:

time=2025-08-26T11:17:26.967Z level=INFO source=routes.go:1331 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:16384 OLLAMA_DEBUG:INFO OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:24h0m0s OLLAMA_KV_CACHE_TYPE:q4_0 OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:30m0s OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:O:\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:true OLLAMA_NEW_ESTIMATES:true OLLAMA_NOHISTORY:true OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]"
time=2025-08-26T11:17:28.104Z level=INFO source=images.go:477 msg="total blobs: 220"
time=2025-08-26T11:17:28.213Z level=INFO source=images.go:484 msg="total unused blobs removed: 0"
time=2025-08-26T11:17:28.515Z level=INFO source=routes.go:1384 msg="Listening on [::]:11434 (version 0.11.7)"
time=2025-08-26T11:17:28.549Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:167 msg=packages count=32
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=0 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=1 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=2 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=3 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=4 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=5 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=6 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=7 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=8 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=9 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=10 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=11 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=12 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=13 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=14 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=15 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=16 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=17 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=18 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=19 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=20 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=21 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=22 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=23 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=24 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=25 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=26 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=27 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=28 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=29 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=30 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=31 cores=1 efficiency=0 threads=1
time=2025-08-26T11:17:30.886Z level=INFO source=gpu.go:321 msg="detected OS VRAM overhead" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db library=cuda compute=7.5 driver=12.9 name="NVIDIA GeForce RTX 2080 Ti" overhead="899.9 MiB"
time=2025-08-26T11:17:30.898Z level=INFO source=types.go:130 msg="inference compute" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db library=cuda variant=v12 compute=7.5 driver=12.9 name="NVIDIA GeForce RTX 2080 Ti" total="11.0 GiB" available="9.9 GiB"
time=2025-08-26T11:17:30.898Z level=INFO source=routes.go:1425 msg="entering low vram mode" "total vram"="11.0 GiB" threshold="20.0 GiB"
time=2025-08-26T11:18:04.841Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-26T11:18:04.852Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-26T11:18:04.866Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 63551"
time=2025-08-26T11:18:04.877Z level=INFO source=server.go:656 msg="loading model" "model layers"=33 requested=-1
time=2025-08-26T11:18:04.887Z level=INFO source=server.go:662 msg="system memory" total="48.0 GiB" free="42.4 GiB" free_swap="45.5 GiB"
time=2025-08-26T11:18:04.887Z level=INFO source=server.go:666 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-26T11:18:05.014Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-26T11:18:05.081Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:63551"
time=2025-08-26T11:18:05.083Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-26T11:18:05.190Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-26T11:18:22.889Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-26T11:18:23.857Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-26T11:18:24.082Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-26T11:18:24.082Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB"
time=2025-08-26T11:18:24.082Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB"
time=2025-08-26T11:18:24.082Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB"
time=2025-08-26T11:18:24.082Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB"
time=2025-08-26T11:18:24.082Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB"
time=2025-08-26T11:18:24.082Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB"
time=2025-08-26T11:18:24.082Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-26T11:18:24.082Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU"
time=2025-08-26T11:18:24.103Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-26T11:18:24.103Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU"
time=2025-08-26T11:18:24.082Z level=INFO source=server.go:1231 msg="waiting for llama runner to start responding"
time=2025-08-26T11:18:24.104Z level=INFO source=server.go:1265 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-26T11:18:43.905Z level=INFO source=server.go:1269 msg="llama runner started in 39.04 seconds"
[GIN] 2025/08/26 - 11:18:58 | 200 |   54.7391993s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/26 - 11:19:02 | 200 |    3.2860431s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/26 - 11:19:10 | 200 |    7.9544442s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/26 - 11:19:17 | 200 |    7.6219234s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/26 - 11:19:24 | 200 |    6.1769249s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/26 - 11:19:32 | 200 |    4.5129122s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/26 - 11:19:37 | 200 |    4.5694705s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/26 - 11:19:41 | 200 |    3.5813144s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/26 - 11:19:45 | 200 |    3.7467903s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/26 - 11:19:48 | 200 |    3.2640409s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/26 - 11:19:51 | 200 |    3.4220544s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/26 - 13:46:07 | 200 |    4.8144772s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/26 - 13:46:10 | 200 |    5.6861347s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/26 - 13:46:21 | 200 |   15.1774774s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/26 - 13:46:25 | 200 |   18.0390962s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/26 - 13:46:28 | 200 |   17.6985458s |    192.168.1.17 | POST     "/api/generate"
[GIN] 2025/08/26 - 13:46:31 | 200 |   10.1354179s |    192.168.1.17 | POST     "/api/generate"
time=2025-08-26T13:46:32.219Z level=INFO source=server.go:166 msg="enabling new memory estimates"
time=2025-08-26T13:46:32.229Z level=INFO source=server.go:211 msg="enabling flash attention"
time=2025-08-26T13:46:32.236Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 49802"
time=2025-08-26T13:46:32.238Z level=INFO source=server.go:656 msg="loading model" "model layers"=49 requested=-1
time=2025-08-26T13:46:32.247Z level=INFO source=server.go:662 msg="system memory" total="48.0 GiB" free="41.7 GiB" free_swap="44.9 GiB"
time=2025-08-26T13:46:32.247Z level=INFO source=server.go:666 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-08-26T13:46:32.309Z level=INFO source=runner.go:1006 msg="starting ollama engine"
time=2025-08-26T13:46:32.339Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:49802"
time=2025-08-26T13:46:32.346Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-26T13:46:32.371Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db
load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-08-26T13:46:32.575Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-08-26T13:46:32.807Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-26T13:46:33.055Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-08-26T13:46:33.055Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU"
time=2025-08-26T13:46:33.055Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU"
time=2025-08-26T13:46:33.055Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU"
time=2025-08-26T13:46:33.055Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB"
time=2025-08-26T13:46:33.055Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB"
time=2025-08-26T13:46:33.055Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB"
time=2025-08-26T13:46:33.055Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB"
time=2025-08-26T13:46:33.055Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-08-26T13:46:33.055Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB"
time=2025-08-26T13:46:33.055Z level=INFO source=sched.go:473 msg="loaded runners" count=1
time=2025-08-26T13:46:33.055Z level=INFO source=server.go:1231 msg="waiting for llama runner to start responding"
time=2025-08-26T13:46:33.055Z level=INFO source=server.go:1265 msg="waiting for server to become available" status="llm server loading model"
time=2025-08-26T13:47:10.122Z level=INFO source=server.go:1269 msg="llama runner started in 37.89 seconds"
time=2025-08-26T13:51:26.297Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection"
[GIN] 2025/08/26 - 13:51:26 | 200 |          5m0s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-26T13:56:29.474Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection"
[GIN] 2025/08/26 - 13:56:29 | 200 |          5m0s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-26T14:01:35.626Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection"
[GIN] 2025/08/26 - 14:01:35 | 200 |          5m0s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-26T14:06:37.864Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection"
[GIN] 2025/08/26 - 14:06:37 | 200 |          5m0s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-26T14:11:40.021Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection"
[GIN] 2025/08/26 - 14:11:40 | 200 |          5m0s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-26T14:16:41.842Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection"
[GIN] 2025/08/26 - 14:16:41 | 200 |          5m0s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-26T14:21:43.514Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection"
[GIN] 2025/08/26 - 14:21:43 | 200 |          5m0s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-26T14:26:45.155Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection"
[GIN] 2025/08/26 - 14:26:45 | 200 |          5m0s |    192.168.1.17 | POST     "/api/chat"
time=2025-08-26T14:31:46.815Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection"
[GIN] 2025/08/26 - 14:31:46 | 200 |          5m0s |    192.168.1.17 | POST     "/api/chat"
[GIN] 2025/08/26 - 14:36:27 | 200 |    190.2768ms |    192.168.1.17 | GET      "/api/tags"

I use parameters to the model:

  1. model Summarizer (runs)
    Image
  2. model Lamark (runs)
    Image

Model flow: model Summarizer (works) -> model Lamark (works) -> workflow restart -> model Summerizer (hang)

also ollama run granite3.3 hello fails at this point:

Image

more attempts:

Image

let me know what else I can provide to help debug the issue!

<!-- gh-comment-id:3224478161 --> @mcr-ksh commented on GitHub (Aug 26, 2025): another day another hang: ``` time=2025-08-26T11:17:26.967Z level=INFO source=routes.go:1331 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:16384 OLLAMA_DEBUG:INFO OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:24h0m0s OLLAMA_KV_CACHE_TYPE:q4_0 OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:30m0s OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:O:\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:true OLLAMA_NEW_ESTIMATES:true OLLAMA_NOHISTORY:true OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]" time=2025-08-26T11:17:28.104Z level=INFO source=images.go:477 msg="total blobs: 220" time=2025-08-26T11:17:28.213Z level=INFO source=images.go:484 msg="total unused blobs removed: 0" time=2025-08-26T11:17:28.515Z level=INFO source=routes.go:1384 msg="Listening on [::]:11434 (version 0.11.7)" time=2025-08-26T11:17:28.549Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:167 msg=packages count=32 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=0 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=1 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=2 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=3 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=4 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=5 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=6 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=7 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=8 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=9 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=10 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=11 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=12 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=13 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=14 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=15 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=16 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=17 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=18 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=19 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=20 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=21 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=22 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=23 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=24 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=25 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=26 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=27 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=28 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=29 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=30 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:28.657Z level=INFO source=gpu_windows.go:214 msg="" package=31 cores=1 efficiency=0 threads=1 time=2025-08-26T11:17:30.886Z level=INFO source=gpu.go:321 msg="detected OS VRAM overhead" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db library=cuda compute=7.5 driver=12.9 name="NVIDIA GeForce RTX 2080 Ti" overhead="899.9 MiB" time=2025-08-26T11:17:30.898Z level=INFO source=types.go:130 msg="inference compute" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db library=cuda variant=v12 compute=7.5 driver=12.9 name="NVIDIA GeForce RTX 2080 Ti" total="11.0 GiB" available="9.9 GiB" time=2025-08-26T11:17:30.898Z level=INFO source=routes.go:1425 msg="entering low vram mode" "total vram"="11.0 GiB" threshold="20.0 GiB" time=2025-08-26T11:18:04.841Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-26T11:18:04.852Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-26T11:18:04.866Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87 --port 63551" time=2025-08-26T11:18:04.877Z level=INFO source=server.go:656 msg="loading model" "model layers"=33 requested=-1 time=2025-08-26T11:18:04.887Z level=INFO source=server.go:662 msg="system memory" total="48.0 GiB" free="42.4 GiB" free_swap="45.5 GiB" time=2025-08-26T11:18:04.887Z level=INFO source=server.go:666 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-26T11:18:05.014Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-26T11:18:05.081Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:63551" time=2025-08-26T11:18:05.083Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-26T11:18:05.190Z level=INFO source=ggml.go:130 msg="" architecture=llama file_type=Q4_0 name="Meta Llama 3.1 8B Instruct" description="" num_tensors=291 num_key_values=30 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-26T11:18:22.889Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-26T11:18:23.857Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-26T11:18:24.082Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:26624 KvCacheType:q4_0 NumThreads:32 GPULayers:33[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:33(0..32)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-26T11:18:24.082Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="4.1 GiB" time=2025-08-26T11:18:24.082Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="281.8 MiB" time=2025-08-26T11:18:24.082Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="936.0 MiB" time=2025-08-26T11:18:24.082Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="98.0 MiB" time=2025-08-26T11:18:24.082Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="8.0 MiB" time=2025-08-26T11:18:24.082Z level=INFO source=backend.go:342 msg="total memory" size="5.4 GiB" time=2025-08-26T11:18:24.082Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-26T11:18:24.082Z level=INFO source=ggml.go:486 msg="offloading 32 repeating layers to GPU" time=2025-08-26T11:18:24.103Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-26T11:18:24.103Z level=INFO source=ggml.go:497 msg="offloaded 33/33 layers to GPU" time=2025-08-26T11:18:24.082Z level=INFO source=server.go:1231 msg="waiting for llama runner to start responding" time=2025-08-26T11:18:24.104Z level=INFO source=server.go:1265 msg="waiting for server to become available" status="llm server loading model" time=2025-08-26T11:18:43.905Z level=INFO source=server.go:1269 msg="llama runner started in 39.04 seconds" [GIN] 2025/08/26 - 11:18:58 | 200 | 54.7391993s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/26 - 11:19:02 | 200 | 3.2860431s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/26 - 11:19:10 | 200 | 7.9544442s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/26 - 11:19:17 | 200 | 7.6219234s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/26 - 11:19:24 | 200 | 6.1769249s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/26 - 11:19:32 | 200 | 4.5129122s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/26 - 11:19:37 | 200 | 4.5694705s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/26 - 11:19:41 | 200 | 3.5813144s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/26 - 11:19:45 | 200 | 3.7467903s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/26 - 11:19:48 | 200 | 3.2640409s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/26 - 11:19:51 | 200 | 3.4220544s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/26 - 13:46:07 | 200 | 4.8144772s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/26 - 13:46:10 | 200 | 5.6861347s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/26 - 13:46:21 | 200 | 15.1774774s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/26 - 13:46:25 | 200 | 18.0390962s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/26 - 13:46:28 | 200 | 17.6985458s | 192.168.1.17 | POST "/api/generate" [GIN] 2025/08/26 - 13:46:31 | 200 | 10.1354179s | 192.168.1.17 | POST "/api/generate" time=2025-08-26T13:46:32.219Z level=INFO source=server.go:166 msg="enabling new memory estimates" time=2025-08-26T13:46:32.229Z level=INFO source=server.go:211 msg="enabling flash attention" time=2025-08-26T13:46:32.236Z level=INFO source=server.go:383 msg="starting runner" cmd="C:\\Users\\ksh.IRONSOFTWARE\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model O:\\models\\blobs\\sha256-657cef1d1ae3ab8b8c58622a8b01b12e364ebfc6777d23964edaa1b20ac979b8 --port 49802" time=2025-08-26T13:46:32.238Z level=INFO source=server.go:656 msg="loading model" "model layers"=49 requested=-1 time=2025-08-26T13:46:32.247Z level=INFO source=server.go:662 msg="system memory" total="48.0 GiB" free="41.7 GiB" free_swap="44.9 GiB" time=2025-08-26T13:46:32.247Z level=INFO source=server.go:666 msg="gpu memory" id=GPU-620f3fa0-47a8-a31e-eff8-c87476f589db available="9.4 GiB" free="9.9 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-08-26T13:46:32.309Z level=INFO source=runner.go:1006 msg="starting ollama engine" time=2025-08-26T13:46:32.339Z level=INFO source=runner.go:1043 msg="Server listening on 127.0.0.1:49802" time=2025-08-26T13:46:32.346Z level=INFO source=runner.go:925 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-26T13:46:32.371Z level=INFO source=ggml.go:130 msg="" architecture=qwen2 file_type=Q4_K_M name="Lamarck 14b Converge Della Linear" description="" num_tensors=579 num_key_values=64 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes, ID: GPU-620f3fa0-47a8-a31e-eff8-c87476f589db load_backend: loaded CUDA backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\ksh.IRONSOFTWARE\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-08-26T13:46:32.575Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-08-26T13:46:32.807Z level=INFO source=runner.go:925 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-26T13:46:33.055Z level=INFO source=runner.go:925 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:true KvSize:9200 KvCacheType:q4_0 NumThreads:32 GPULayers:49[ID:GPU-620f3fa0-47a8-a31e-eff8-c87476f589db Layers:49(0..48)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-08-26T13:46:33.055Z level=INFO source=ggml.go:486 msg="offloading 48 repeating layers to GPU" time=2025-08-26T13:46:33.055Z level=INFO source=ggml.go:492 msg="offloading output layer to GPU" time=2025-08-26T13:46:33.055Z level=INFO source=ggml.go:497 msg="offloaded 49/49 layers to GPU" time=2025-08-26T13:46:33.055Z level=INFO source=backend.go:310 msg="model weights" device=CUDA0 size="8.0 GiB" time=2025-08-26T13:46:33.055Z level=INFO source=backend.go:315 msg="model weights" device=CPU size="416.6 MiB" time=2025-08-26T13:46:33.055Z level=INFO source=backend.go:321 msg="kv cache" device=CUDA0 size="486.0 MiB" time=2025-08-26T13:46:33.055Z level=INFO source=backend.go:332 msg="compute graph" device=CUDA0 size="83.0 MiB" time=2025-08-26T13:46:33.055Z level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB" time=2025-08-26T13:46:33.055Z level=INFO source=backend.go:342 msg="total memory" size="8.9 GiB" time=2025-08-26T13:46:33.055Z level=INFO source=sched.go:473 msg="loaded runners" count=1 time=2025-08-26T13:46:33.055Z level=INFO source=server.go:1231 msg="waiting for llama runner to start responding" time=2025-08-26T13:46:33.055Z level=INFO source=server.go:1265 msg="waiting for server to become available" status="llm server loading model" time=2025-08-26T13:47:10.122Z level=INFO source=server.go:1269 msg="llama runner started in 37.89 seconds" time=2025-08-26T13:51:26.297Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection" [GIN] 2025/08/26 - 13:51:26 | 200 | 5m0s | 192.168.1.17 | POST "/api/chat" time=2025-08-26T13:56:29.474Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection" [GIN] 2025/08/26 - 13:56:29 | 200 | 5m0s | 192.168.1.17 | POST "/api/chat" time=2025-08-26T14:01:35.626Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection" [GIN] 2025/08/26 - 14:01:35 | 200 | 5m0s | 192.168.1.17 | POST "/api/chat" time=2025-08-26T14:06:37.864Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection" [GIN] 2025/08/26 - 14:06:37 | 200 | 5m0s | 192.168.1.17 | POST "/api/chat" time=2025-08-26T14:11:40.021Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection" [GIN] 2025/08/26 - 14:11:40 | 200 | 5m0s | 192.168.1.17 | POST "/api/chat" time=2025-08-26T14:16:41.842Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection" [GIN] 2025/08/26 - 14:16:41 | 200 | 5m0s | 192.168.1.17 | POST "/api/chat" time=2025-08-26T14:21:43.514Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection" [GIN] 2025/08/26 - 14:21:43 | 200 | 5m0s | 192.168.1.17 | POST "/api/chat" time=2025-08-26T14:26:45.155Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection" [GIN] 2025/08/26 - 14:26:45 | 200 | 5m0s | 192.168.1.17 | POST "/api/chat" time=2025-08-26T14:31:46.815Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection" [GIN] 2025/08/26 - 14:31:46 | 200 | 5m0s | 192.168.1.17 | POST "/api/chat" [GIN] 2025/08/26 - 14:36:27 | 200 | 190.2768ms | 192.168.1.17 | GET "/api/tags" ``` I use parameters to the model: 1. model Summarizer (runs) <img width="196" height="412" alt="Image" src="https://github.com/user-attachments/assets/5e62bc1b-3f53-45a2-b57b-23e11d00ce4f" /> 2. model Lamark (runs) <img width="515" height="336" alt="Image" src="https://github.com/user-attachments/assets/0fff06e8-ab4d-4803-bbad-56ca6a1a8eb3" /> Model flow: model Summarizer (works) -> model Lamark (works) -> workflow restart -> model Summerizer (hang) also `ollama run granite3.3 hello` fails at this point: <img width="965" height="622" alt="Image" src="https://github.com/user-attachments/assets/8f08d273-353a-47ae-b080-793cace0f553" /> more attempts: <img width="849" height="638" alt="Image" src="https://github.com/user-attachments/assets/fb8e55d4-3127-497a-838b-e423e41cbf18" /> let me know what else I can provide to help debug the issue!
GiteaMirror commented 2026-04-12 20:10:52 -05:00
Author
Owner
Copy Link

@jessegross commented on GitHub (Aug 26, 2025):

If you can reproduce it with OLLAMA_DEBUG=1 set, that would be helpful to see more of what is going on.

<!-- gh-comment-id:3224949360 --> @jessegross commented on GitHub (Aug 26, 2025): If you can reproduce it with OLLAMA_DEBUG=1 set, that would be helpful to see more of what is going on.
GiteaMirror commented 2026-04-12 20:10:53 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 26, 2025):

time=2025-08-26T13:47:10.122Z level=INFO source=server.go:1269 msg="llama runner started in 37.89 seconds"
time=2025-08-26T13:51:26.297Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection"
[GIN] 2025/08/26 - 13:51:26 | 200 |          5m0s |    192.168.1.17 | POST     "/api/chat"

The model loaded but didn't complete a response to the client before the client timed out. While the client is waiting, what is the GPU doing? What does nvidia-smi show?

<!-- gh-comment-id:3224976867 --> @rick-github commented on GitHub (Aug 26, 2025): ``` time=2025-08-26T13:47:10.122Z level=INFO source=server.go:1269 msg="llama runner started in 37.89 seconds" time=2025-08-26T13:51:26.297Z level=INFO source=server.go:1400 msg="aborting completion request due to client closing the connection" [GIN] 2025/08/26 - 13:51:26 | 200 | 5m0s | 192.168.1.17 | POST "/api/chat" ``` The model loaded but didn't complete a response to the client before the client timed out. While the client is waiting, what is the GPU doing? What does `nvidia-smi` show?
GiteaMirror commented 2026-04-12 20:10:53 -05:00
Author
Owner
Copy Link

@mcr-ksh commented on GitHub (Aug 27, 2025):

@rick-github the GPU is idle. Performance monitor has no Cuda activity.

Image Image
<!-- gh-comment-id:3227489549 --> @mcr-ksh commented on GitHub (Aug 27, 2025): @rick-github the GPU is idle. Performance monitor has no Cuda activity. <img width="738" height="413" alt="Image" src="https://github.com/user-attachments/assets/33be63ef-66b2-43fb-9c68-6e8ebd0d11df" /> <img width="523" height="819" alt="Image" src="https://github.com/user-attachments/assets/8aa59d94-90ff-4a8c-acb6-c3e3d788b395" />
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#7986
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?