github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 12:48:05 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #4887] qwen2:7b-instruct is not running correctly, seems that the model is not correctly loaded. #65120

New Issue
Closed
opened 2026-05-03 19:47:01 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-05-03 19:47:01 -05:00
Owner
Copy Link

Originally created by @henryclw on GitHub (Jun 7, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/4887

What is the issue?

I ran this command

ollama run qwen2:7b-instruct

And ask why is the sky blue?
Then it output GGML_ASK_GGML
And more G

I'm using a nvidia GPU

Click to view logs 

2024-06-06 17:24:50 2024/06/07 00:24:50 routes.go:1007: INFO server config env="map[OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST: OLLAMA_KEEP_ALIVE: OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MAX_VRAM:0 OLLAMA_MODELS: OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:*] OLLAMA_RUNNERS_DIR: OLLAMA_TMPDIR:]"
2024-06-06 17:24:50 time=2024-06-07T00:24:50.384Z level=INFO source=images.go:729 msg="total blobs: 55"
2024-06-06 17:24:50 time=2024-06-07T00:24:50.504Z level=INFO source=images.go:736 msg="total unused blobs removed: 0"
2024-06-06 17:24:50 time=2024-06-07T00:24:50.594Z level=INFO source=routes.go:1053 msg="Listening on [::]:11434 (version 0.1.41)"
2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama1794705039/runners
2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gz
2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gz
2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gz
2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublas.so.11.gz
2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublasLt.so.11.gz
2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcudart.so.11.0.gz
2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/ollama_llama_server.gz
2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/deps.txt.gz
2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/ollama_llama_server.gz
2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu
2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu_avx
2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu_avx2
2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cuda_v11
2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/rocm_v60002
2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 rocm_v60002]"
2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=payload.go:45 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=sched.go:90 msg="starting llm scheduler"
2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=gpu.go:132 msg="Detecting GPUs"
2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=gpu.go:274 msg="Searching for GPU library" name=libcuda.so*
2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=gpu.go:293 msg="gpu library search" globs="[/usr/local/nvidia/lib/libcuda.so** /usr/local/nvidia/lib64/libcuda.so** /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=gpu.go:326 msg="discovered GPU libraries" paths="[/usr/lib/x86_64-linux-gnu/libcuda.so.1 /usr/lib/wsl/drivers/nv_dispui.inf_amd64_74cdc804095ee524/libcuda.so.1.1]"
2024-06-06 17:24:52 CUDA driver version: 12.3
2024-06-06 17:24:52 time=2024-06-07T00:24:52.639Z level=DEBUG source=gpu.go:137 msg="detected GPUs" count=1 library=/usr/lib/x86_64-linux-gnu/libcuda.so.1
2024-06-06 17:24:52 time=2024-06-07T00:24:52.639Z level=DEBUG source=cpu_common.go:11 msg="CPU has AVX2"
2024-06-06 17:24:52 [GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898] CUDA totalMem 12287 mb
2024-06-06 17:24:52 [GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898] CUDA freeMem 11261 mb
2024-06-06 17:24:52 [GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898] Compute Capability 8.6
2024-06-06 17:24:52 time=2024-06-07T00:24:52.821Z level=DEBUG source=amd_linux.go:322 msg="amdgpu driver not detected /sys/module/amdgpu"
2024-06-06 17:24:52 releasing nvcuda library
2024-06-06 17:24:52 time=2024-06-07T00:24:52.821Z level=INFO source=types.go:71 msg="inference compute" id=GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898 library=cuda compute=8.6 driver=12.3 name="NVIDIA GeForce RTX 3060" total="12.0 GiB" available="11.0 GiB"
2024-06-06 17:24:57 time=2024-06-07T00:24:57.329Z level=DEBUG source=gpu.go:132 msg="Detecting GPUs"
2024-06-06 17:24:57 time=2024-06-07T00:24:57.329Z level=DEBUG source=gpu.go:274 msg="Searching for GPU library" name=libcuda.so*
2024-06-06 17:24:57 time=2024-06-07T00:24:57.329Z level=DEBUG source=gpu.go:293 msg="gpu library search" globs="[/usr/local/nvidia/lib/libcuda.so** /usr/local/nvidia/lib64/libcuda.so** /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
2024-06-06 17:24:57 time=2024-06-07T00:24:57.329Z level=DEBUG source=gpu.go:326 msg="discovered GPU libraries" paths="[/usr/lib/x86_64-linux-gnu/libcuda.so.1 /usr/lib/wsl/drivers/nv_dispui.inf_amd64_74cdc804095ee524/libcuda.so.1.1]"
2024-06-06 17:24:57 CUDA driver version: 12.3
2024-06-06 17:24:57 time=2024-06-07T00:24:57.329Z level=DEBUG source=gpu.go:137 msg="detected GPUs" count=1 library=/usr/lib/x86_64-linux-gnu/libcuda.so.1
2024-06-06 17:24:57 time=2024-06-07T00:24:57.329Z level=DEBUG source=cpu_common.go:11 msg="CPU has AVX2"
2024-06-06 17:24:57 [GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898] CUDA totalMem 12287 mb
2024-06-06 17:24:57 [GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898] CUDA freeMem 11261 mb
2024-06-06 17:24:57 [GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898] Compute Capability 8.6
2024-06-06 17:24:57 time=2024-06-07T00:24:57.452Z level=DEBUG source=amd_linux.go:322 msg="amdgpu driver not detected /sys/module/amdgpu"
2024-06-06 17:24:57 releasing nvcuda library
2024-06-06 17:24:57 time=2024-06-07T00:24:57.454Z level=DEBUG source=gguf.go:57 msg="model = &llm.gguf{containerGGUF:(*llm.containerGGUF)(0xc0004269c0), kv:llm.KV{}, tensors:[]*llm.Tensor(nil), parameters:0x0}"
2024-06-06 17:26:00 time=2024-06-07T00:26:00.429Z level=DEBUG source=sched.go:153 msg="loading first model" model=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5
2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=memory.go:44 msg=evaluating library=cuda gpu_count=1 available="11.0 GiB"
2024-06-06 17:24:56 [GIN] 2024/06/07 - 00:24:56 | 200 |   95.864537ms |      172.20.0.3 | GET      "/api/tags"
2024-06-06 17:24:57 [GIN] 2024/06/07 - 00:24:57 | 200 |       14.59µs |       127.0.0.1 | HEAD     "/"
2024-06-06 17:24:57 [GIN] 2024/06/07 - 00:24:57 | 200 |    8.112224ms |       127.0.0.1 | POST     "/api/show"
2024-06-06 17:24:57 [GIN] 2024/06/07 - 00:24:57 | 200 |    10.21512ms |       127.0.0.1 | POST     "/api/show"
2024-06-06 17:26:00 INFO [main] build info | build=1 commit="5921b8f" tid="140205045272576" timestamp=1717719960
2024-06-06 17:26:00 INFO [main] system info | n_threads=12 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="140205045272576" timestamp=1717719960 total_threads=24
2024-06-06 17:26:00 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="23" port="35309" tid="140205045272576" timestamp=1717719960
2024-06-06 17:27:30 DEBUG [initialize] initializing slots | n_slots=1 tid="140205045272576" timestamp=1717720050
2024-06-06 17:27:30 DEBUG [initialize] new slot | n_ctx_slot=2048 slot_id=0 tid="140205045272576" timestamp=1717720050
2024-06-06 17:27:30 INFO [main] model loaded | tid="140205045272576" timestamp=1717720050
2024-06-06 17:27:30 DEBUG [update_slots] all slots are idle and system prompt is empty, clear the KV cache | tid="140205045272576" timestamp=1717720050
2024-06-06 17:27:30 DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=0 tid="140205045272576" timestamp=1717720050
2024-06-06 17:27:30 [GIN] 2024/06/07 - 00:27:30 | 200 |         2m32s |       127.0.0.1 | POST     "/api/chat"
2024-06-06 17:28:00 DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=1 tid="140205045272576" timestamp=1717720080
2024-06-06 17:28:00 DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=2 tid="140205045272576" timestamp=1717720080
2024-06-06 17:28:00 DEBUG [log_server_request] request | method="POST" params={} path="/tokenize" remote_addr="127.0.0.1" remote_port=59764 status=200 tid="140204593274880" timestamp=1717720080
2024-06-06 17:28:00 DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=3 tid="140205045272576" timestamp=1717720080
2024-06-06 17:28:00 DEBUG [launch_slot_with_data] slot is processing task | slot_id=0 task_id=4 tid="140205045272576" timestamp=1717720080
2024-06-06 17:28:00 DEBUG [update_slots] slot progression | ga_i=0 n_past=0 n_past_se=0 n_prompt_tokens_processed=14 slot_id=0 task_id=4 tid="140205045272576" timestamp=1717720080
2024-06-06 17:28:00 DEBUG [update_slots] kv cache rm [p0, end) | p0=0 slot_id=0 task_id=4 tid="140205045272576" timestamp=1717720080
2024-06-06 17:28:01 DEBUG [print_timings] prompt eval time     =     103.85 ms /    14 tokens (    7.42 ms per token,   134.81 tokens per second) | n_prompt_tokens_processed=14 n_tokens_second=134.80592760921687 slot_id=0 t_prompt_processing=103.853 t_token=7.418071428571428 task_id=4 tid="140205045272576" timestamp=1717720081
2024-06-06 17:28:01 DEBUG [print_timings] generation eval time =     183.47 ms /    10 runs   (   18.35 ms per token,    54.50 tokens per second) | n_decoded=10 n_tokens_second=54.50333832947268 slot_id=0 t_token=18.3475 t_token_generation=183.475 task_id=4 tid="140205045272576" timestamp=1717720081
2024-06-06 17:28:01 DEBUG [print_timings]           total time =     287.33 ms | slot_id=0 t_prompt_processing=103.853 t_token_generation=183.475 t_total=287.328 task_id=4 tid="140205045272576" timestamp=1717720081
2024-06-06 17:28:01 DEBUG [update_slots] slot released | n_cache_tokens=24 n_ctx=2048 n_past=23 n_system_tokens=0 slot_id=0 task_id=4 tid="140205045272576" timestamp=1717720081 truncated=false
2024-06-06 17:28:01 DEBUG [log_server_request] request | method="POST" params={} path="/completion" remote_addr="127.0.0.1" remote_port=59764 status=200 tid="140204593274880" timestamp=1717720081
2024-06-06 17:28:01 [GIN] 2024/06/07 - 00:28:01 | 200 |   446.95734ms |       127.0.0.1 | POST     "/api/chat"
2024-06-06 17:28:21 DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=17 tid="140205045272576" timestamp=1717720101
2024-06-06 17:28:21 DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=18 tid="140205045272576" timestamp=1717720101
2024-06-06 17:28:22 DEBUG [log_server_request] request | method="POST" params={} path="/tokenize" remote_addr="127.0.0.1" remote_port=55234 status=200 tid="140204509753344" timestamp=1717720102
2024-06-06 17:28:22 DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=19 tid="140205045272576" timestamp=1717720102
2024-06-06 17:28:22 DEBUG [log_server_request] request | method="POST" params={} path="/tokenize" remote_addr="127.0.0.1" remote_port=55234 status=200 tid="140204509753344" timestamp=1717720102
2024-06-06 17:28:22 DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=20 tid="140205045272576" timestamp=1717720102
2024-06-06 17:28:22 DEBUG [launch_slot_with_data] slot is processing task | slot_id=0 task_id=21 tid="140205045272576" timestamp=1717720102
2024-06-06 17:28:22 DEBUG [update_slots] slot progression | ga_i=0 n_past=14 n_past_se=0 n_prompt_tokens_processed=23 slot_id=0 task_id=21 tid="140205045272576" timestamp=1717720102
2024-06-06 17:28:22 DEBUG [update_slots] kv cache rm [p0, end) | p0=14 slot_id=0 task_id=21 tid="140205045272576" timestamp=1717720102
2024-06-06 17:28:23 [GIN] 2024/06/07 - 00:28:23 | 200 |  1.363873173s |       127.0.0.1 | POST     "/api/chat"
2024-06-06 17:28:23 DEBUG [log_server_request] request | method="POST" params={} path="/completion" remote_addr="127.0.0.1" remote_port=55242 status=200 tid="140204501360640" timestamp=1717720103
2024-06-06 17:28:23 DEBUG [update_slots] slot released | n_cache_tokens=86 n_ctx=2048 n_past=85 n_system_tokens=0 slot_id=0 task_id=21 tid="140205045272576" timestamp=1717720103 truncated=false
2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=29 memory.available="11.0 GiB" memory.required.full="4.8 GiB" memory.required.partial="4.8 GiB" memory.required.kv="112.0 MiB" memory.weights.total="3.8 GiB" memory.weights.repeating="3.4 GiB" memory.weights.nonrepeating="426.4 MiB" memory.graph.full="304.0 MiB" memory.graph.partial="730.4 MiB"
2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=sched.go:565 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 gpu=GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898 available=11808014336 required="4.8 GiB"
2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=memory.go:44 msg=evaluating library=cuda gpu_count=1 available="11.0 GiB"
2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=29 memory.available="11.0 GiB" memory.required.full="4.8 GiB" memory.required.partial="4.8 GiB" memory.required.kv="112.0 MiB" memory.weights.total="3.8 GiB" memory.weights.repeating="3.4 GiB" memory.weights.nonrepeating="426.4 MiB" memory.graph.full="304.0 MiB" memory.graph.partial="730.4 MiB"
2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu
2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu_avx
2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu_avx2
2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cuda_v11
2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/rocm_v60002
2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu
2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu_avx
2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu_avx2
2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cuda_v11
2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/rocm_v60002
2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=cpu_common.go:11 msg="CPU has AVX2"
2024-06-06 17:26:00 time=2024-06-07T00:26:00.431Z level=INFO source=server.go:341 msg="starting llama server" cmd="/tmp/ollama1794705039/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 29 --verbose --parallel 1 --port 35309"
2024-06-06 17:26:00 time=2024-06-07T00:26:00.431Z level=DEBUG source=server.go:356 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/tmp/ollama1794705039/runners/cuda_v11:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 CUDA_VISIBLE_DEVICES=GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898]"
2024-06-06 17:26:00 time=2024-06-07T00:26:00.431Z level=INFO source=sched.go:338 msg="loaded runners" count=1
2024-06-06 17:26:00 time=2024-06-07T00:26:00.431Z level=INFO source=server.go:529 msg="waiting for llama runner to start responding"
2024-06-06 17:26:00 time=2024-06-07T00:26:00.431Z level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server error"
2024-06-06 17:26:00 llama_model_loader: loaded meta data with 21 key-value pairs and 339 tensors from /root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 (version GGUF V3 (latest))
2024-06-06 17:26:00 llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
2024-06-06 17:26:00 llama_model_loader: - kv   0:                       general.architecture str              = qwen2
2024-06-06 17:26:00 llama_model_loader: - kv   1:                               general.name str              = Qwen2-7B-Instruct
2024-06-06 17:26:00 llama_model_loader: - kv   2:                          qwen2.block_count u32              = 28
2024-06-06 17:26:00 llama_model_loader: - kv   3:                       qwen2.context_length u32              = 32768
2024-06-06 17:26:00 llama_model_loader: - kv   4:                     qwen2.embedding_length u32              = 3584
2024-06-06 17:26:00 llama_model_loader: - kv   5:                  qwen2.feed_forward_length u32              = 18944
2024-06-06 17:26:00 llama_model_loader: - kv   6:                 qwen2.attention.head_count u32              = 28
2024-06-06 17:26:00 llama_model_loader: - kv   7:              qwen2.attention.head_count_kv u32              = 4
2024-06-06 17:26:00 llama_model_loader: - kv   8:                       qwen2.rope.freq_base f32              = 1000000.000000
2024-06-06 17:26:00 llama_model_loader: - kv   9:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
2024-06-06 17:26:00 llama_model_loader: - kv  10:                          general.file_type u32              = 2
2024-06-06 17:26:00 llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = gpt2
2024-06-06 17:26:00 llama_model_loader: - kv  12:                         tokenizer.ggml.pre str              = qwen2
2024-06-06 17:26:00 llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,152064]  = ["!", "\"", "#", "$", "%", "&", "'", ...
2024-06-06 17:26:00 llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,152064]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
2024-06-06 17:26:00 llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
2024-06-06 17:26:00 llama_model_loader: - kv  16:                tokenizer.ggml.eos_token_id u32              = 151645
2024-06-06 17:26:00 llama_model_loader: - kv  17:            tokenizer.ggml.padding_token_id u32              = 151643
2024-06-06 17:26:00 llama_model_loader: - kv  18:                tokenizer.ggml.bos_token_id u32              = 151643
2024-06-06 17:26:00 llama_model_loader: - kv  19:                    tokenizer.chat_template str              = {% for message in messages %}{% if lo...
2024-06-06 17:26:00 llama_model_loader: - kv  20:               general.quantization_version u32              = 2
2024-06-06 17:26:00 llama_model_loader: - type  f32:  141 tensors
2024-06-06 17:26:00 llama_model_loader: - type q4_0:  197 tensors
2024-06-06 17:26:00 llama_model_loader: - type q6_K:    1 tensors
2024-06-06 17:26:00 time=2024-06-07T00:26:00.682Z level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server loading model"
2024-06-06 17:26:00 llm_load_vocab: special tokens cache size = 421
2024-06-06 17:26:00 llm_load_vocab: token to piece cache size = 1.8703 MB
2024-06-06 17:26:00 llm_load_print_meta: format           = GGUF V3 (latest)
2024-06-06 17:26:00 llm_load_print_meta: arch             = qwen2
2024-06-06 17:26:00 llm_load_print_meta: vocab type       = BPE
2024-06-06 17:26:00 llm_load_print_meta: n_vocab          = 152064
2024-06-06 17:26:00 llm_load_print_meta: n_merges         = 151387
2024-06-06 17:26:00 llm_load_print_meta: n_ctx_train      = 32768
2024-06-06 17:26:00 llm_load_print_meta: n_embd           = 3584
2024-06-06 17:26:00 llm_load_print_meta: n_head           = 28
2024-06-06 17:26:00 llm_load_print_meta: n_head_kv        = 4
2024-06-06 17:26:00 llm_load_print_meta: n_layer          = 28
2024-06-06 17:26:00 llm_load_print_meta: n_rot            = 128
2024-06-06 17:26:00 llm_load_print_meta: n_embd_head_k    = 128
2024-06-06 17:26:00 llm_load_print_meta: n_embd_head_v    = 128
2024-06-06 17:26:00 llm_load_print_meta: n_gqa            = 7
2024-06-06 17:26:00 llm_load_print_meta: n_embd_k_gqa     = 512
2024-06-06 17:26:00 llm_load_print_meta: n_embd_v_gqa     = 512
2024-06-06 17:26:00 llm_load_print_meta: f_norm_eps       = 0.0e+00
2024-06-06 17:26:00 llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
2024-06-06 17:26:00 llm_load_print_meta: f_clamp_kqv      = 0.0e+00
2024-06-06 17:26:00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00
2024-06-06 17:26:00 llm_load_print_meta: f_logit_scale    = 0.0e+00
2024-06-06 17:26:00 llm_load_print_meta: n_ff             = 18944
2024-06-06 17:26:00 llm_load_print_meta: n_expert         = 0
2024-06-06 17:26:00 llm_load_print_meta: n_expert_used    = 0
2024-06-06 17:26:00 llm_load_print_meta: causal attn      = 1
2024-06-06 17:26:00 llm_load_print_meta: pooling type     = 0
2024-06-06 17:26:00 llm_load_print_meta: rope type        = 2
2024-06-06 17:26:00 llm_load_print_meta: rope scaling     = linear
2024-06-06 17:26:00 llm_load_print_meta: freq_base_train  = 1000000.0
2024-06-06 17:26:00 llm_load_print_meta: freq_scale_train = 1
2024-06-06 17:26:00 llm_load_print_meta: n_yarn_orig_ctx  = 32768
2024-06-06 17:26:00 llm_load_print_meta: rope_finetuned   = unknown
2024-06-06 17:26:00 llm_load_print_meta: ssm_d_conv       = 0
2024-06-06 17:26:00 llm_load_print_meta: ssm_d_inner      = 0
2024-06-06 17:26:00 llm_load_print_meta: ssm_d_state      = 0
2024-06-06 17:26:00 llm_load_print_meta: ssm_dt_rank      = 0
2024-06-06 17:26:00 llm_load_print_meta: model type       = ?B
2024-06-06 17:26:00 llm_load_print_meta: model ftype      = Q4_0
2024-06-06 17:26:00 llm_load_print_meta: model params     = 7.62 B
2024-06-06 17:26:00 llm_load_print_meta: model size       = 4.12 GiB (4.65 BPW) 
2024-06-06 17:26:00 llm_load_print_meta: general.name     = Qwen2-7B-Instruct
2024-06-06 17:26:00 llm_load_print_meta: BOS token        = 151643 '<|endoftext|>'
2024-06-06 17:26:00 llm_load_print_meta: EOS token        = 151645 '<|im_end|>'
2024-06-06 17:26:00 llm_load_print_meta: PAD token        = 151643 '<|endoftext|>'
2024-06-06 17:26:00 llm_load_print_meta: LF token         = 148848 'ÄĬ'
2024-06-06 17:26:00 llm_load_print_meta: EOT token        = 151645 '<|im_end|>'
2024-06-06 17:26:00 ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   yes
2024-06-06 17:26:00 ggml_cuda_init: CUDA_USE_TENSOR_CORES: no
2024-06-06 17:26:00 ggml_cuda_init: found 1 CUDA devices:
2024-06-06 17:26:00   Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
2024-06-06 17:26:01 llm_load_tensors: ggml ctx size =    0.32 MiB
2024-06-06 17:27:28 llm_load_tensors: offloading 28 repeating layers to GPU
2024-06-06 17:27:28 llm_load_tensors: offloading non-repeating layers to GPU
2024-06-06 17:27:28 llm_load_tensors: offloaded 29/29 layers to GPU
2024-06-06 17:27:28 llm_load_tensors:        CPU buffer size =   292.36 MiB
2024-06-06 17:27:28 llm_load_tensors:      CUDA0 buffer size =  3928.07 MiB
2024-06-06 17:27:28 time=2024-06-07T00:27:28.948Z level=DEBUG source=server.go:578 msg="model load progress 0.07"
2024-06-06 17:27:29 time=2024-06-07T00:27:29.199Z level=DEBUG source=server.go:578 msg="model load progress 0.30"
2024-06-06 17:27:29 time=2024-06-07T00:27:29.449Z level=DEBUG source=server.go:578 msg="model load progress 0.99"
2024-06-06 17:27:29 llama_new_context_with_model: n_ctx      = 2048
2024-06-06 17:27:29 llama_new_context_with_model: n_batch    = 512
2024-06-06 17:27:29 llama_new_context_with_model: n_ubatch   = 512
2024-06-06 17:27:29 llama_new_context_with_model: flash_attn = 0
2024-06-06 17:27:29 llama_new_context_with_model: freq_base  = 1000000.0
2024-06-06 17:27:29 llama_new_context_with_model: freq_scale = 1
2024-06-06 17:27:29 llama_kv_cache_init:      CUDA0 KV buffer size =   112.00 MiB
2024-06-06 17:27:29 llama_new_context_with_model: KV self size  =  112.00 MiB, K (f16):   56.00 MiB, V (f16):   56.00 MiB
2024-06-06 17:27:29 llama_new_context_with_model:  CUDA_Host  output buffer size =     0.59 MiB
2024-06-06 17:27:29 llama_new_context_with_model:      CUDA0 compute buffer size =   304.00 MiB
2024-06-06 17:27:29 llama_new_context_with_model:  CUDA_Host compute buffer size =    11.01 MiB
2024-06-06 17:27:29 llama_new_context_with_model: graph nodes  = 986
2024-06-06 17:27:29 llama_new_context_with_model: graph splits = 2
2024-06-06 17:27:29 time=2024-06-07T00:27:29.700Z level=DEBUG source=server.go:578 msg="model load progress 1.00"
2024-06-06 17:27:29 time=2024-06-07T00:27:29.950Z level=DEBUG source=server.go:581 msg="model load completed, waiting for server to become available" status="llm server loading model"
2024-06-06 17:27:30 time=2024-06-07T00:27:30.201Z level=INFO source=server.go:572 msg="llama runner started in 89.77 seconds"
2024-06-06 17:27:30 time=2024-06-07T00:27:30.201Z level=DEBUG source=sched.go:351 msg="finished setting up runner" model=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5
2024-06-06 17:27:30 time=2024-06-07T00:27:30.201Z level=DEBUG source=prompt.go:172 msg="prompt now fits in context window" required=1 window=2048
2024-06-06 17:27:30 time=2024-06-07T00:27:30.202Z level=DEBUG source=sched.go:355 msg="context for request finished"
2024-06-06 17:27:30 time=2024-06-07T00:27:30.202Z level=DEBUG source=sched.go:237 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 duration=2562047h47m16.854775807s
2024-06-06 17:27:30 time=2024-06-07T00:27:30.202Z level=DEBUG source=sched.go:255 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 refCount=0
2024-06-06 17:28:00 time=2024-06-07T00:28:00.684Z level=DEBUG source=sched.go:447 msg="evaluating already loaded" model=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5
2024-06-06 17:28:00 time=2024-06-07T00:28:00.785Z level=DEBUG source=prompt.go:172 msg="prompt now fits in context window" required=17 window=2048
2024-06-06 17:28:00 time=2024-06-07T00:28:00.785Z level=DEBUG source=routes.go:1301 msg="chat handler" prompt="<|im_start|>user\nwhy is the sky blue?<|im_end|>\n<|im_start|>assistant\n" images=0
2024-06-06 17:28:00 time=2024-06-07T00:28:00.785Z level=DEBUG source=server.go:668 msg="setting token limit to 10x num_ctx" num_ctx=2048 num_predict=20480
2024-06-06 17:28:01 time=2024-06-07T00:28:01.123Z level=DEBUG source=sched.go:304 msg="context for request finished"
2024-06-06 17:28:01 time=2024-06-07T00:28:01.123Z level=DEBUG source=sched.go:237 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 duration=2562047h47m16.854775807s
2024-06-06 17:28:01 time=2024-06-07T00:28:01.123Z level=DEBUG source=sched.go:255 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 refCount=0
2024-06-06 17:28:21 time=2024-06-07T00:28:21.981Z level=DEBUG source=sched.go:447 msg="evaluating already loaded" model=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5
2024-06-06 17:28:22 time=2024-06-07T00:28:22.137Z level=DEBUG source=prompt.go:172 msg="prompt now fits in context window" required=40 window=2048
2024-06-06 17:28:22 time=2024-06-07T00:28:22.137Z level=DEBUG source=routes.go:1301 msg="chat handler" prompt="<|im_start|>user\nwhy is the sky blue?<|im_end|>\n<|im_start|>assistant\nGGML_ASK_GGML<|im_end|>\n<|im_start|>user\nWhy is the sky blue?<|im_end|>\n<|im_start|>assistant\n" images=0
2024-06-06 17:28:22 time=2024-06-07T00:28:22.137Z level=DEBUG source=server.go:668 msg="setting token limit to 10x num_ctx" num_ctx=2048 num_predict=20480
2024-06-06 17:28:23 time=2024-06-07T00:28:23.337Z level=DEBUG source=server.go:781 msg="prediction aborted, token repeat limit reached"
2024-06-06 17:28:23 time=2024-06-07T00:28:23.338Z level=DEBUG source=sched.go:304 msg="context for request finished"
2024-06-06 17:28:23 time=2024-06-07T00:28:23.338Z level=DEBUG source=sched.go:237 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 duration=2562047h47m16.854775807s
2024-06-06 17:28:23 time=2024-06-07T00:28:23.338Z level=DEBUG source=sched.go:255 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 refCount=0

OS

Docker

GPU

Nvidia

CPU

AMD

Ollama version

0.1.41

Originally created by @henryclw on GitHub (Jun 7, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/4887 ### What is the issue? I ran this command ```bash ollama run qwen2:7b-instruct ``` And ask `why is the sky blue?` Then it output `GGML_ASK_GGML` And more G I'm using a nvidia GPU <details> <summary>Click to view logs</summary> ``` 2024-06-06 17:24:50 2024/06/07 00:24:50 routes.go:1007: INFO server config env="map[OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST: OLLAMA_KEEP_ALIVE: OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MAX_VRAM:0 OLLAMA_MODELS: OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:*] OLLAMA_RUNNERS_DIR: OLLAMA_TMPDIR:]" 2024-06-06 17:24:50 time=2024-06-07T00:24:50.384Z level=INFO source=images.go:729 msg="total blobs: 55" 2024-06-06 17:24:50 time=2024-06-07T00:24:50.504Z level=INFO source=images.go:736 msg="total unused blobs removed: 0" 2024-06-06 17:24:50 time=2024-06-07T00:24:50.594Z level=INFO source=routes.go:1053 msg="Listening on [::]:11434 (version 0.1.41)" 2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama1794705039/runners 2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gz 2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gz 2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gz 2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublas.so.11.gz 2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublasLt.so.11.gz 2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcudart.so.11.0.gz 2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/ollama_llama_server.gz 2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/deps.txt.gz 2024-06-06 17:24:50 time=2024-06-07T00:24:50.595Z level=DEBUG source=payload.go:180 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/ollama_llama_server.gz 2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu 2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu_avx 2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu_avx2 2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cuda_v11 2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/rocm_v60002 2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 rocm_v60002]" 2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=payload.go:45 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY" 2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=sched.go:90 msg="starting llm scheduler" 2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=gpu.go:132 msg="Detecting GPUs" 2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=gpu.go:274 msg="Searching for GPU library" name=libcuda.so* 2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=gpu.go:293 msg="gpu library search" globs="[/usr/local/nvidia/lib/libcuda.so** /usr/local/nvidia/lib64/libcuda.so** /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" 2024-06-06 17:24:52 time=2024-06-07T00:24:52.627Z level=DEBUG source=gpu.go:326 msg="discovered GPU libraries" paths="[/usr/lib/x86_64-linux-gnu/libcuda.so.1 /usr/lib/wsl/drivers/nv_dispui.inf_amd64_74cdc804095ee524/libcuda.so.1.1]" 2024-06-06 17:24:52 CUDA driver version: 12.3 2024-06-06 17:24:52 time=2024-06-07T00:24:52.639Z level=DEBUG source=gpu.go:137 msg="detected GPUs" count=1 library=/usr/lib/x86_64-linux-gnu/libcuda.so.1 2024-06-06 17:24:52 time=2024-06-07T00:24:52.639Z level=DEBUG source=cpu_common.go:11 msg="CPU has AVX2" 2024-06-06 17:24:52 [GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898] CUDA totalMem 12287 mb 2024-06-06 17:24:52 [GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898] CUDA freeMem 11261 mb 2024-06-06 17:24:52 [GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898] Compute Capability 8.6 2024-06-06 17:24:52 time=2024-06-07T00:24:52.821Z level=DEBUG source=amd_linux.go:322 msg="amdgpu driver not detected /sys/module/amdgpu" 2024-06-06 17:24:52 releasing nvcuda library 2024-06-06 17:24:52 time=2024-06-07T00:24:52.821Z level=INFO source=types.go:71 msg="inference compute" id=GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898 library=cuda compute=8.6 driver=12.3 name="NVIDIA GeForce RTX 3060" total="12.0 GiB" available="11.0 GiB" 2024-06-06 17:24:57 time=2024-06-07T00:24:57.329Z level=DEBUG source=gpu.go:132 msg="Detecting GPUs" 2024-06-06 17:24:57 time=2024-06-07T00:24:57.329Z level=DEBUG source=gpu.go:274 msg="Searching for GPU library" name=libcuda.so* 2024-06-06 17:24:57 time=2024-06-07T00:24:57.329Z level=DEBUG source=gpu.go:293 msg="gpu library search" globs="[/usr/local/nvidia/lib/libcuda.so** /usr/local/nvidia/lib64/libcuda.so** /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" 2024-06-06 17:24:57 time=2024-06-07T00:24:57.329Z level=DEBUG source=gpu.go:326 msg="discovered GPU libraries" paths="[/usr/lib/x86_64-linux-gnu/libcuda.so.1 /usr/lib/wsl/drivers/nv_dispui.inf_amd64_74cdc804095ee524/libcuda.so.1.1]" 2024-06-06 17:24:57 CUDA driver version: 12.3 2024-06-06 17:24:57 time=2024-06-07T00:24:57.329Z level=DEBUG source=gpu.go:137 msg="detected GPUs" count=1 library=/usr/lib/x86_64-linux-gnu/libcuda.so.1 2024-06-06 17:24:57 time=2024-06-07T00:24:57.329Z level=DEBUG source=cpu_common.go:11 msg="CPU has AVX2" 2024-06-06 17:24:57 [GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898] CUDA totalMem 12287 mb 2024-06-06 17:24:57 [GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898] CUDA freeMem 11261 mb 2024-06-06 17:24:57 [GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898] Compute Capability 8.6 2024-06-06 17:24:57 time=2024-06-07T00:24:57.452Z level=DEBUG source=amd_linux.go:322 msg="amdgpu driver not detected /sys/module/amdgpu" 2024-06-06 17:24:57 releasing nvcuda library 2024-06-06 17:24:57 time=2024-06-07T00:24:57.454Z level=DEBUG source=gguf.go:57 msg="model = &llm.gguf{containerGGUF:(*llm.containerGGUF)(0xc0004269c0), kv:llm.KV{}, tensors:[]*llm.Tensor(nil), parameters:0x0}" 2024-06-06 17:26:00 time=2024-06-07T00:26:00.429Z level=DEBUG source=sched.go:153 msg="loading first model" model=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=memory.go:44 msg=evaluating library=cuda gpu_count=1 available="11.0 GiB" 2024-06-06 17:24:56 [GIN] 2024/06/07 - 00:24:56 | 200 | 95.864537ms | 172.20.0.3 | GET "/api/tags" 2024-06-06 17:24:57 [GIN] 2024/06/07 - 00:24:57 | 200 | 14.59µs | 127.0.0.1 | HEAD "/" 2024-06-06 17:24:57 [GIN] 2024/06/07 - 00:24:57 | 200 | 8.112224ms | 127.0.0.1 | POST "/api/show" 2024-06-06 17:24:57 [GIN] 2024/06/07 - 00:24:57 | 200 | 10.21512ms | 127.0.0.1 | POST "/api/show" 2024-06-06 17:26:00 INFO [main] build info | build=1 commit="5921b8f" tid="140205045272576" timestamp=1717719960 2024-06-06 17:26:00 INFO [main] system info | n_threads=12 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="140205045272576" timestamp=1717719960 total_threads=24 2024-06-06 17:26:00 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="23" port="35309" tid="140205045272576" timestamp=1717719960 2024-06-06 17:27:30 DEBUG [initialize] initializing slots | n_slots=1 tid="140205045272576" timestamp=1717720050 2024-06-06 17:27:30 DEBUG [initialize] new slot | n_ctx_slot=2048 slot_id=0 tid="140205045272576" timestamp=1717720050 2024-06-06 17:27:30 INFO [main] model loaded | tid="140205045272576" timestamp=1717720050 2024-06-06 17:27:30 DEBUG [update_slots] all slots are idle and system prompt is empty, clear the KV cache | tid="140205045272576" timestamp=1717720050 2024-06-06 17:27:30 DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=0 tid="140205045272576" timestamp=1717720050 2024-06-06 17:27:30 [GIN] 2024/06/07 - 00:27:30 | 200 | 2m32s | 127.0.0.1 | POST "/api/chat" 2024-06-06 17:28:00 DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=1 tid="140205045272576" timestamp=1717720080 2024-06-06 17:28:00 DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=2 tid="140205045272576" timestamp=1717720080 2024-06-06 17:28:00 DEBUG [log_server_request] request | method="POST" params={} path="/tokenize" remote_addr="127.0.0.1" remote_port=59764 status=200 tid="140204593274880" timestamp=1717720080 2024-06-06 17:28:00 DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=3 tid="140205045272576" timestamp=1717720080 2024-06-06 17:28:00 DEBUG [launch_slot_with_data] slot is processing task | slot_id=0 task_id=4 tid="140205045272576" timestamp=1717720080 2024-06-06 17:28:00 DEBUG [update_slots] slot progression | ga_i=0 n_past=0 n_past_se=0 n_prompt_tokens_processed=14 slot_id=0 task_id=4 tid="140205045272576" timestamp=1717720080 2024-06-06 17:28:00 DEBUG [update_slots] kv cache rm [p0, end) | p0=0 slot_id=0 task_id=4 tid="140205045272576" timestamp=1717720080 2024-06-06 17:28:01 DEBUG [print_timings] prompt eval time = 103.85 ms / 14 tokens ( 7.42 ms per token, 134.81 tokens per second) | n_prompt_tokens_processed=14 n_tokens_second=134.80592760921687 slot_id=0 t_prompt_processing=103.853 t_token=7.418071428571428 task_id=4 tid="140205045272576" timestamp=1717720081 2024-06-06 17:28:01 DEBUG [print_timings] generation eval time = 183.47 ms / 10 runs ( 18.35 ms per token, 54.50 tokens per second) | n_decoded=10 n_tokens_second=54.50333832947268 slot_id=0 t_token=18.3475 t_token_generation=183.475 task_id=4 tid="140205045272576" timestamp=1717720081 2024-06-06 17:28:01 DEBUG [print_timings] total time = 287.33 ms | slot_id=0 t_prompt_processing=103.853 t_token_generation=183.475 t_total=287.328 task_id=4 tid="140205045272576" timestamp=1717720081 2024-06-06 17:28:01 DEBUG [update_slots] slot released | n_cache_tokens=24 n_ctx=2048 n_past=23 n_system_tokens=0 slot_id=0 task_id=4 tid="140205045272576" timestamp=1717720081 truncated=false 2024-06-06 17:28:01 DEBUG [log_server_request] request | method="POST" params={} path="/completion" remote_addr="127.0.0.1" remote_port=59764 status=200 tid="140204593274880" timestamp=1717720081 2024-06-06 17:28:01 [GIN] 2024/06/07 - 00:28:01 | 200 | 446.95734ms | 127.0.0.1 | POST "/api/chat" 2024-06-06 17:28:21 DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=17 tid="140205045272576" timestamp=1717720101 2024-06-06 17:28:21 DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=18 tid="140205045272576" timestamp=1717720101 2024-06-06 17:28:22 DEBUG [log_server_request] request | method="POST" params={} path="/tokenize" remote_addr="127.0.0.1" remote_port=55234 status=200 tid="140204509753344" timestamp=1717720102 2024-06-06 17:28:22 DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=19 tid="140205045272576" timestamp=1717720102 2024-06-06 17:28:22 DEBUG [log_server_request] request | method="POST" params={} path="/tokenize" remote_addr="127.0.0.1" remote_port=55234 status=200 tid="140204509753344" timestamp=1717720102 2024-06-06 17:28:22 DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=20 tid="140205045272576" timestamp=1717720102 2024-06-06 17:28:22 DEBUG [launch_slot_with_data] slot is processing task | slot_id=0 task_id=21 tid="140205045272576" timestamp=1717720102 2024-06-06 17:28:22 DEBUG [update_slots] slot progression | ga_i=0 n_past=14 n_past_se=0 n_prompt_tokens_processed=23 slot_id=0 task_id=21 tid="140205045272576" timestamp=1717720102 2024-06-06 17:28:22 DEBUG [update_slots] kv cache rm [p0, end) | p0=14 slot_id=0 task_id=21 tid="140205045272576" timestamp=1717720102 2024-06-06 17:28:23 [GIN] 2024/06/07 - 00:28:23 | 200 | 1.363873173s | 127.0.0.1 | POST "/api/chat" 2024-06-06 17:28:23 DEBUG [log_server_request] request | method="POST" params={} path="/completion" remote_addr="127.0.0.1" remote_port=55242 status=200 tid="140204501360640" timestamp=1717720103 2024-06-06 17:28:23 DEBUG [update_slots] slot released | n_cache_tokens=86 n_ctx=2048 n_past=85 n_system_tokens=0 slot_id=0 task_id=21 tid="140205045272576" timestamp=1717720103 truncated=false 2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=29 memory.available="11.0 GiB" memory.required.full="4.8 GiB" memory.required.partial="4.8 GiB" memory.required.kv="112.0 MiB" memory.weights.total="3.8 GiB" memory.weights.repeating="3.4 GiB" memory.weights.nonrepeating="426.4 MiB" memory.graph.full="304.0 MiB" memory.graph.partial="730.4 MiB" 2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=sched.go:565 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 gpu=GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898 available=11808014336 required="4.8 GiB" 2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=memory.go:44 msg=evaluating library=cuda gpu_count=1 available="11.0 GiB" 2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=29 memory.available="11.0 GiB" memory.required.full="4.8 GiB" memory.required.partial="4.8 GiB" memory.required.kv="112.0 MiB" memory.weights.total="3.8 GiB" memory.weights.repeating="3.4 GiB" memory.weights.nonrepeating="426.4 MiB" memory.graph.full="304.0 MiB" memory.graph.partial="730.4 MiB" 2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu 2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu_avx 2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu_avx2 2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cuda_v11 2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/rocm_v60002 2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu 2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu_avx 2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cpu_avx2 2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/cuda_v11 2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1794705039/runners/rocm_v60002 2024-06-06 17:26:00 time=2024-06-07T00:26:00.430Z level=DEBUG source=cpu_common.go:11 msg="CPU has AVX2" 2024-06-06 17:26:00 time=2024-06-07T00:26:00.431Z level=INFO source=server.go:341 msg="starting llama server" cmd="/tmp/ollama1794705039/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 29 --verbose --parallel 1 --port 35309" 2024-06-06 17:26:00 time=2024-06-07T00:26:00.431Z level=DEBUG source=server.go:356 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/tmp/ollama1794705039/runners/cuda_v11:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 CUDA_VISIBLE_DEVICES=GPU-d1fa0558-2f75-4c4b-0720-8d90d6b01898]" 2024-06-06 17:26:00 time=2024-06-07T00:26:00.431Z level=INFO source=sched.go:338 msg="loaded runners" count=1 2024-06-06 17:26:00 time=2024-06-07T00:26:00.431Z level=INFO source=server.go:529 msg="waiting for llama runner to start responding" 2024-06-06 17:26:00 time=2024-06-07T00:26:00.431Z level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server error" 2024-06-06 17:26:00 llama_model_loader: loaded meta data with 21 key-value pairs and 339 tensors from /root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 (version GGUF V3 (latest)) 2024-06-06 17:26:00 llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. 2024-06-06 17:26:00 llama_model_loader: - kv 0: general.architecture str = qwen2 2024-06-06 17:26:00 llama_model_loader: - kv 1: general.name str = Qwen2-7B-Instruct 2024-06-06 17:26:00 llama_model_loader: - kv 2: qwen2.block_count u32 = 28 2024-06-06 17:26:00 llama_model_loader: - kv 3: qwen2.context_length u32 = 32768 2024-06-06 17:26:00 llama_model_loader: - kv 4: qwen2.embedding_length u32 = 3584 2024-06-06 17:26:00 llama_model_loader: - kv 5: qwen2.feed_forward_length u32 = 18944 2024-06-06 17:26:00 llama_model_loader: - kv 6: qwen2.attention.head_count u32 = 28 2024-06-06 17:26:00 llama_model_loader: - kv 7: qwen2.attention.head_count_kv u32 = 4 2024-06-06 17:26:00 llama_model_loader: - kv 8: qwen2.rope.freq_base f32 = 1000000.000000 2024-06-06 17:26:00 llama_model_loader: - kv 9: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001 2024-06-06 17:26:00 llama_model_loader: - kv 10: general.file_type u32 = 2 2024-06-06 17:26:00 llama_model_loader: - kv 11: tokenizer.ggml.model str = gpt2 2024-06-06 17:26:00 llama_model_loader: - kv 12: tokenizer.ggml.pre str = qwen2 2024-06-06 17:26:00 llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,152064] = ["!", "\"", "#", "$", "%", "&", "'", ... 2024-06-06 17:26:00 llama_model_loader: - kv 14: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... 2024-06-06 17:26:00 llama_model_loader: - kv 15: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... 2024-06-06 17:26:00 llama_model_loader: - kv 16: tokenizer.ggml.eos_token_id u32 = 151645 2024-06-06 17:26:00 llama_model_loader: - kv 17: tokenizer.ggml.padding_token_id u32 = 151643 2024-06-06 17:26:00 llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 151643 2024-06-06 17:26:00 llama_model_loader: - kv 19: tokenizer.chat_template str = {% for message in messages %}{% if lo... 2024-06-06 17:26:00 llama_model_loader: - kv 20: general.quantization_version u32 = 2 2024-06-06 17:26:00 llama_model_loader: - type f32: 141 tensors 2024-06-06 17:26:00 llama_model_loader: - type q4_0: 197 tensors 2024-06-06 17:26:00 llama_model_loader: - type q6_K: 1 tensors 2024-06-06 17:26:00 time=2024-06-07T00:26:00.682Z level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server loading model" 2024-06-06 17:26:00 llm_load_vocab: special tokens cache size = 421 2024-06-06 17:26:00 llm_load_vocab: token to piece cache size = 1.8703 MB 2024-06-06 17:26:00 llm_load_print_meta: format = GGUF V3 (latest) 2024-06-06 17:26:00 llm_load_print_meta: arch = qwen2 2024-06-06 17:26:00 llm_load_print_meta: vocab type = BPE 2024-06-06 17:26:00 llm_load_print_meta: n_vocab = 152064 2024-06-06 17:26:00 llm_load_print_meta: n_merges = 151387 2024-06-06 17:26:00 llm_load_print_meta: n_ctx_train = 32768 2024-06-06 17:26:00 llm_load_print_meta: n_embd = 3584 2024-06-06 17:26:00 llm_load_print_meta: n_head = 28 2024-06-06 17:26:00 llm_load_print_meta: n_head_kv = 4 2024-06-06 17:26:00 llm_load_print_meta: n_layer = 28 2024-06-06 17:26:00 llm_load_print_meta: n_rot = 128 2024-06-06 17:26:00 llm_load_print_meta: n_embd_head_k = 128 2024-06-06 17:26:00 llm_load_print_meta: n_embd_head_v = 128 2024-06-06 17:26:00 llm_load_print_meta: n_gqa = 7 2024-06-06 17:26:00 llm_load_print_meta: n_embd_k_gqa = 512 2024-06-06 17:26:00 llm_load_print_meta: n_embd_v_gqa = 512 2024-06-06 17:26:00 llm_load_print_meta: f_norm_eps = 0.0e+00 2024-06-06 17:26:00 llm_load_print_meta: f_norm_rms_eps = 1.0e-06 2024-06-06 17:26:00 llm_load_print_meta: f_clamp_kqv = 0.0e+00 2024-06-06 17:26:00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 2024-06-06 17:26:00 llm_load_print_meta: f_logit_scale = 0.0e+00 2024-06-06 17:26:00 llm_load_print_meta: n_ff = 18944 2024-06-06 17:26:00 llm_load_print_meta: n_expert = 0 2024-06-06 17:26:00 llm_load_print_meta: n_expert_used = 0 2024-06-06 17:26:00 llm_load_print_meta: causal attn = 1 2024-06-06 17:26:00 llm_load_print_meta: pooling type = 0 2024-06-06 17:26:00 llm_load_print_meta: rope type = 2 2024-06-06 17:26:00 llm_load_print_meta: rope scaling = linear 2024-06-06 17:26:00 llm_load_print_meta: freq_base_train = 1000000.0 2024-06-06 17:26:00 llm_load_print_meta: freq_scale_train = 1 2024-06-06 17:26:00 llm_load_print_meta: n_yarn_orig_ctx = 32768 2024-06-06 17:26:00 llm_load_print_meta: rope_finetuned = unknown 2024-06-06 17:26:00 llm_load_print_meta: ssm_d_conv = 0 2024-06-06 17:26:00 llm_load_print_meta: ssm_d_inner = 0 2024-06-06 17:26:00 llm_load_print_meta: ssm_d_state = 0 2024-06-06 17:26:00 llm_load_print_meta: ssm_dt_rank = 0 2024-06-06 17:26:00 llm_load_print_meta: model type = ?B 2024-06-06 17:26:00 llm_load_print_meta: model ftype = Q4_0 2024-06-06 17:26:00 llm_load_print_meta: model params = 7.62 B 2024-06-06 17:26:00 llm_load_print_meta: model size = 4.12 GiB (4.65 BPW) 2024-06-06 17:26:00 llm_load_print_meta: general.name = Qwen2-7B-Instruct 2024-06-06 17:26:00 llm_load_print_meta: BOS token = 151643 '<|endoftext|>' 2024-06-06 17:26:00 llm_load_print_meta: EOS token = 151645 '<|im_end|>' 2024-06-06 17:26:00 llm_load_print_meta: PAD token = 151643 '<|endoftext|>' 2024-06-06 17:26:00 llm_load_print_meta: LF token = 148848 'ÄĬ' 2024-06-06 17:26:00 llm_load_print_meta: EOT token = 151645 '<|im_end|>' 2024-06-06 17:26:00 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: yes 2024-06-06 17:26:00 ggml_cuda_init: CUDA_USE_TENSOR_CORES: no 2024-06-06 17:26:00 ggml_cuda_init: found 1 CUDA devices: 2024-06-06 17:26:00 Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes 2024-06-06 17:26:01 llm_load_tensors: ggml ctx size = 0.32 MiB 2024-06-06 17:27:28 llm_load_tensors: offloading 28 repeating layers to GPU 2024-06-06 17:27:28 llm_load_tensors: offloading non-repeating layers to GPU 2024-06-06 17:27:28 llm_load_tensors: offloaded 29/29 layers to GPU 2024-06-06 17:27:28 llm_load_tensors: CPU buffer size = 292.36 MiB 2024-06-06 17:27:28 llm_load_tensors: CUDA0 buffer size = 3928.07 MiB 2024-06-06 17:27:28 time=2024-06-07T00:27:28.948Z level=DEBUG source=server.go:578 msg="model load progress 0.07" 2024-06-06 17:27:29 time=2024-06-07T00:27:29.199Z level=DEBUG source=server.go:578 msg="model load progress 0.30" 2024-06-06 17:27:29 time=2024-06-07T00:27:29.449Z level=DEBUG source=server.go:578 msg="model load progress 0.99" 2024-06-06 17:27:29 llama_new_context_with_model: n_ctx = 2048 2024-06-06 17:27:29 llama_new_context_with_model: n_batch = 512 2024-06-06 17:27:29 llama_new_context_with_model: n_ubatch = 512 2024-06-06 17:27:29 llama_new_context_with_model: flash_attn = 0 2024-06-06 17:27:29 llama_new_context_with_model: freq_base = 1000000.0 2024-06-06 17:27:29 llama_new_context_with_model: freq_scale = 1 2024-06-06 17:27:29 llama_kv_cache_init: CUDA0 KV buffer size = 112.00 MiB 2024-06-06 17:27:29 llama_new_context_with_model: KV self size = 112.00 MiB, K (f16): 56.00 MiB, V (f16): 56.00 MiB 2024-06-06 17:27:29 llama_new_context_with_model: CUDA_Host output buffer size = 0.59 MiB 2024-06-06 17:27:29 llama_new_context_with_model: CUDA0 compute buffer size = 304.00 MiB 2024-06-06 17:27:29 llama_new_context_with_model: CUDA_Host compute buffer size = 11.01 MiB 2024-06-06 17:27:29 llama_new_context_with_model: graph nodes = 986 2024-06-06 17:27:29 llama_new_context_with_model: graph splits = 2 2024-06-06 17:27:29 time=2024-06-07T00:27:29.700Z level=DEBUG source=server.go:578 msg="model load progress 1.00" 2024-06-06 17:27:29 time=2024-06-07T00:27:29.950Z level=DEBUG source=server.go:581 msg="model load completed, waiting for server to become available" status="llm server loading model" 2024-06-06 17:27:30 time=2024-06-07T00:27:30.201Z level=INFO source=server.go:572 msg="llama runner started in 89.77 seconds" 2024-06-06 17:27:30 time=2024-06-07T00:27:30.201Z level=DEBUG source=sched.go:351 msg="finished setting up runner" model=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 2024-06-06 17:27:30 time=2024-06-07T00:27:30.201Z level=DEBUG source=prompt.go:172 msg="prompt now fits in context window" required=1 window=2048 2024-06-06 17:27:30 time=2024-06-07T00:27:30.202Z level=DEBUG source=sched.go:355 msg="context for request finished" 2024-06-06 17:27:30 time=2024-06-07T00:27:30.202Z level=DEBUG source=sched.go:237 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 duration=2562047h47m16.854775807s 2024-06-06 17:27:30 time=2024-06-07T00:27:30.202Z level=DEBUG source=sched.go:255 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 refCount=0 2024-06-06 17:28:00 time=2024-06-07T00:28:00.684Z level=DEBUG source=sched.go:447 msg="evaluating already loaded" model=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 2024-06-06 17:28:00 time=2024-06-07T00:28:00.785Z level=DEBUG source=prompt.go:172 msg="prompt now fits in context window" required=17 window=2048 2024-06-06 17:28:00 time=2024-06-07T00:28:00.785Z level=DEBUG source=routes.go:1301 msg="chat handler" prompt="<|im_start|>user\nwhy is the sky blue?<|im_end|>\n<|im_start|>assistant\n" images=0 2024-06-06 17:28:00 time=2024-06-07T00:28:00.785Z level=DEBUG source=server.go:668 msg="setting token limit to 10x num_ctx" num_ctx=2048 num_predict=20480 2024-06-06 17:28:01 time=2024-06-07T00:28:01.123Z level=DEBUG source=sched.go:304 msg="context for request finished" 2024-06-06 17:28:01 time=2024-06-07T00:28:01.123Z level=DEBUG source=sched.go:237 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 duration=2562047h47m16.854775807s 2024-06-06 17:28:01 time=2024-06-07T00:28:01.123Z level=DEBUG source=sched.go:255 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 refCount=0 2024-06-06 17:28:21 time=2024-06-07T00:28:21.981Z level=DEBUG source=sched.go:447 msg="evaluating already loaded" model=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 2024-06-06 17:28:22 time=2024-06-07T00:28:22.137Z level=DEBUG source=prompt.go:172 msg="prompt now fits in context window" required=40 window=2048 2024-06-06 17:28:22 time=2024-06-07T00:28:22.137Z level=DEBUG source=routes.go:1301 msg="chat handler" prompt="<|im_start|>user\nwhy is the sky blue?<|im_end|>\n<|im_start|>assistant\nGGML_ASK_GGML<|im_end|>\n<|im_start|>user\nWhy is the sky blue?<|im_end|>\n<|im_start|>assistant\n" images=0 2024-06-06 17:28:22 time=2024-06-07T00:28:22.137Z level=DEBUG source=server.go:668 msg="setting token limit to 10x num_ctx" num_ctx=2048 num_predict=20480 2024-06-06 17:28:23 time=2024-06-07T00:28:23.337Z level=DEBUG source=server.go:781 msg="prediction aborted, token repeat limit reached" 2024-06-06 17:28:23 time=2024-06-07T00:28:23.338Z level=DEBUG source=sched.go:304 msg="context for request finished" 2024-06-06 17:28:23 time=2024-06-07T00:28:23.338Z level=DEBUG source=sched.go:237 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 duration=2562047h47m16.854775807s 2024-06-06 17:28:23 time=2024-06-07T00:28:23.338Z level=DEBUG source=sched.go:255 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 refCount=0 ``` </details> ### OS Docker ### GPU Nvidia ### CPU AMD ### Ollama version 0.1.41
GiteaMirror added the bug label 2026-05-03 19:47:01 -05:00
GiteaMirror commented 2026-05-03 19:47:05 -05:00
Author
Owner
Copy Link

@owenzhao commented on GitHub (Jun 7, 2024):

The same issue is here.

Windows 11 RTX 4060 8G, Ollama 0.1.41.

截屏2024-06-07 09 13 51
<!-- gh-comment-id:2153683111 --> @owenzhao commented on GitHub (Jun 7, 2024): The same issue is here. Windows 11 RTX 4060 8G, Ollama 0.1.41. <img width="992" alt="截屏2024-06-07 09 13 51" src="https://github.com/ollama/ollama/assets/2182896/d8e54be8-ca17-4cd9-a245-8d6760bcf761">
GiteaMirror commented 2026-05-03 19:47:05 -05:00
Author
Owner
Copy Link

@owenzhao commented on GitHub (Jun 7, 2024):

There is a bug that Ollama will use CPU instead of GPU after nvidia driver update. So I update my driver. And I found it works fine with CPU but slow. So the issue is with GPU.

截屏2024-06-07 09 28 13
<!-- gh-comment-id:2153696610 --> @owenzhao commented on GitHub (Jun 7, 2024): There is a bug that Ollama will use CPU instead of GPU after nvidia driver update. So I update my driver. And I found it works fine with CPU but slow. So the issue is with GPU. <img width="983" alt="截屏2024-06-07 09 28 13" src="https://github.com/ollama/ollama/assets/2182896/ccaaa4b0-af77-4f81-b0df-d3b1602e3ee8">
GiteaMirror commented 2026-05-03 19:47:06 -05:00
Author
Owner
Copy Link

@YFengFly commented on GitHub (Jun 7, 2024):

Not only 7b, but also 1.5b have the same problem.
0.5b is normal

<!-- gh-comment-id:2153724114 --> @YFengFly commented on GitHub (Jun 7, 2024): Not only 7b, but also 1.5b have the same problem. 0.5b is normal
GiteaMirror commented 2026-05-03 19:47:08 -05:00
Author
Owner
Copy Link

@CaptainDP commented on GitHub (Jun 7, 2024):

ollama version is 0.1.37
1、ollama run qwen2:7b or ollama run qwen2:0.5b
Error: llama runner process has terminated: signal: aborted (core dumped)
2、ollama run qwen:14b
is ok

<!-- gh-comment-id:2154178799 --> @CaptainDP commented on GitHub (Jun 7, 2024): ollama version is 0.1.37 1、ollama run qwen2:7b or ollama run qwen2:0.5b Error: llama runner process has terminated: signal: aborted (core dumped) 2、ollama run qwen:14b is ok
GiteaMirror commented 2026-05-03 19:47:10 -05:00
Author
Owner
Copy Link

@cgint commented on GitHub (Jun 7, 2024):

I had the same issue - Pre-Release-Version 1.42 works for me

See https://github.com/ollama/ollama/issues/4890 for the answer.

<!-- gh-comment-id:2154775649 --> @cgint commented on GitHub (Jun 7, 2024): I had the same issue - Pre-Release-Version 1.42 works for me See https://github.com/ollama/ollama/issues/4890 for the answer.
GiteaMirror commented 2026-05-03 19:47:11 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Jun 7, 2024):

Hi all sorry about this, it was fixed in Hi all, sorry about this. It is fixed in 0.1.42: https://github.com/ollama/ollama/releases/tag/v0.1.42

<!-- gh-comment-id:2155671897 --> @jmorganca commented on GitHub (Jun 7, 2024): Hi all sorry about this, it was fixed in Hi all, sorry about this. It is fixed in 0.1.42: https://github.com/ollama/ollama/releases/tag/v0.1.42
GiteaMirror commented 2026-05-03 19:47:12 -05:00
Author
Owner
Copy Link

@henryclw commented on GitHub (Jun 8, 2024):

Hi all sorry about this, it was fixed in Hi all, sorry about this. It is fixed in 0.1.42: https://github.com/ollama/ollama/releases/tag/v0.1.42

Thank you so much @jmorganca Have a great weekend!

<!-- gh-comment-id:2156121214 --> @henryclw commented on GitHub (Jun 8, 2024): > Hi all sorry about this, it was fixed in Hi all, sorry about this. It is fixed in 0.1.42: https://github.com/ollama/ollama/releases/tag/v0.1.42 Thank you so much @jmorganca Have a great weekend!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#65120
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?