github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-22 05:31:43 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #6199] Ollama crashes with Deepseek-Coder-V2-Lite-Instruct #3872

New Issue
Closed
opened 2026-04-12 14:42:31 -05:00 by GiteaMirror · 9 comments
GiteaMirror commented 2026-04-12 14:42:31 -05:00
Owner
Copy Link

Originally created by @shockme on GitHub (Aug 6, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6199

What is the issue?

The output is cut in the middle of generation. Here's the log:

Aug 06 15:10:46 user-desktop systemd[4465]: Started Ollama Service.
Aug 06 15:10:46 user-desktop ollama[13639]: 2024/08/06 15:10:46 routes.go:1108: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://127.0.0.1:11435 OL
LAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:2 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/user/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:3 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
Aug 06 15:10:46 user-desktop ollama[13639]: time=2024-08-06T15:10:46.362+03:00 level=INFO source=images.go:781 msg="total blobs: 92"
Aug 06 15:10:46 user-desktop ollama[13639]: time=2024-08-06T15:10:46.365+03:00 level=INFO source=images.go:788 msg="total unused blobs removed: 0"
Aug 06 15:10:46 user-desktop ollama[13639]: time=2024-08-06T15:10:46.366+03:00 level=INFO source=routes.go:1155 msg="Listening on 127.0.0.1:11435 (version 0.3.3)"
Aug 06 15:10:46 user-desktop ollama[13639]: time=2024-08-06T15:10:46.367+03:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama198573251/runners
Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.148+03:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 rocm_v60102]"
Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.148+03:00 level=INFO source=gpu.go:205 msg="looking for compatible GPUs"
Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.472+03:00 level=WARN source=amd_linux.go:59 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.472+03:00 level=INFO source=amd_linux.go:360 msg="no compatible amdgpu devices detected"
Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.472+03:00 level=INFO source=types.go:105 msg="inference compute" id=GPU-5968b8f6-eb32-e5b2-37a0-2a8637c2ae09 library=cuda compute=6.1 driver=12.2 name="Tesla P40" total="23.9 GiB" available="23.7 GiB"
Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.757+03:00 level=INFO source=sched.go:710 msg="new model will fit in available VRAM in single GPU, loading" model=/home/user/.ollama/models/blobs/sha256-373dcfc92e01372709b6164fc836f677a6280e25e9eac5c434c64223207bfc4f gpu=GPU-5968b8f6-eb32-e5b2-37a0-2a8637c2ae09 parallel=3 available=25430458368 required="17.7 GiB"
Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.759+03:00 level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=28 layers.offload=28 layers.split="" memory.available="[23.7 GiB]" memory.required.full="17.7 GiB" memory.required.partial="17.7 GiB" memory.required.kv="1.6 GiB" memory.required.allocations="[17.7 GiB]" memory.weights.total="16.7 GiB" memory.weights.repeating="16.5 GiB" memory.weights.nonrepeating="212.5 MiB" memory.graph.full="228.0 MiB" memory.graph.partial="376.1 MiB"
Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.760+03:00 level=INFO source=server.go:384 msg="starting llama server" cmd="/tmp/ollama198573251/runners/cuda_v11/ollama_llama_server --model /home/user/.ollama/models/blobs/sha256-373dcfc92e01372709b6164fc836f677a6280e25e9eac5c434c64223207bfc4f --ctx-size 6144 --batch-size 512 --embedding --log-disable --n-gpu-layers 28 --parallel 3 --port 40025"
Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.760+03:00 level=INFO source=sched.go:445 msg="loaded runners" count=1
Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.760+03:00 level=INFO source=server.go:584 msg="waiting for llama runner to start responding"
Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.761+03:00 level=INFO source=server.go:618 msg="waiting for server to become available" status="llm server error"
Aug 06 15:10:53 user-desktop ollama[13757]: INFO [main] build info | build=1 commit="6eeaeba" tid="140158488453120" timestamp=1722946253
Aug 06 15:10:53 user-desktop ollama[13757]: INFO [main] system info | n_threads=16 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="140158488453120" timestamp=1722946253 total_threads=32
Aug 06 15:10:53 user-desktop ollama[13757]: INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="31" port="40025" tid="140158488453120" timestamp=1722946253
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: loaded meta data with 38 key-value pairs and 377 tensors from /home/user/.ollama/models/blobs/sha256-373dcfc92e01372709b6164fc836f677a6280e25e9eac5c434c64223207bfc4f (version GGUF V3 (latest))
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv   1:                               general.name str              = DeepSeek-Coder-V2-Lite-Instruct
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv   2:                      deepseek2.block_count u32              = 27
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv   3:                   deepseek2.context_length u32              = 163840
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv   4:                 deepseek2.embedding_length u32              = 2048
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv   5:              deepseek2.feed_forward_length u32              = 10944
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv   6:             deepseek2.attention.head_count u32              = 16
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv   7:          deepseek2.attention.head_count_kv u32              = 16
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv   8:                   deepseek2.rope.freq_base f32              = 10000.000000
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv   9: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  10:                deepseek2.expert_used_count u32              = 6
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  11:                          general.file_type u32              = 7
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  12:        deepseek2.leading_dense_block_count u32              = 1
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  13:                       deepseek2.vocab_size u32              = 102400
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  14:           deepseek2.attention.kv_lora_rank u32              = 512
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  15:             deepseek2.attention.key_length u32              = 192
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  16:           deepseek2.attention.value_length u32              = 128
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  17:       deepseek2.expert_feed_forward_length u32              = 1408
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  18:                     deepseek2.expert_count u32              = 64
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  19:              deepseek2.expert_shared_count u32              = 2
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  20:             deepseek2.expert_weights_scale f32              = 1.000000
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  21:             deepseek2.rope.dimension_count u32              = 64
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  22:                deepseek2.rope.scaling.type str              = yarn
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  23:              deepseek2.rope.scaling.factor f32              = 40.000000
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  24: deepseek2.rope.scaling.original_context_length u32              = 4096
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  25: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.070700
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  26:                       tokenizer.ggml.model str              = gpt2
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  27:                         tokenizer.ggml.pre str              = deepseek-llm
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  28:                      tokenizer.ggml.tokens arr[str,102400]  = ["!", "\"", "#", "$", "%", "&", "'", ...
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  29:                  tokenizer.ggml.token_type arr[i32,102400]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  30:                      tokenizer.ggml.merges arr[str,99757]   = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e...
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  31:                tokenizer.ggml.bos_token_id u32              = 100000
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  32:                tokenizer.ggml.eos_token_id u32              = 100001
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  33:            tokenizer.ggml.padding_token_id u32              = 100001
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  34:               tokenizer.ggml.add_bos_token bool             = true
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  35:               tokenizer.ggml.add_eos_token bool             = false
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  36:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv  37:               general.quantization_version u32              = 2
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - type  f32:  108 tensors
Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - type q8_0:  269 tensors
Aug 06 15:10:54 user-desktop ollama[13639]: time=2024-08-06T15:10:54.014+03:00 level=INFO source=server.go:618 msg="waiting for server to become available" status="llm server loading model"
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_vocab: special tokens cache size = 2400
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_vocab: token to piece cache size = 0.6661 MB
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: format           = GGUF V3 (latest)
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: arch             = deepseek2
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: vocab type       = BPE
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_vocab          = 102400
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_merges         = 99757
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: vocab_only       = 0
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_ctx_train      = 163840
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_embd           = 2048
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_layer          = 27
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_head           = 16
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_head_kv        = 16
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_rot            = 64
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_swa            = 0
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_embd_head_k    = 192
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_embd_head_v    = 128
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_gqa            = 1
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_embd_k_gqa     = 3072
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_embd_v_gqa     = 2048
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: f_norm_eps       = 0.0e+00
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: f_clamp_kqv      = 0.0e+00
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: f_logit_scale    = 0.0e+00
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_ff             = 10944
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_expert         = 64
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_expert_used    = 6
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: causal attn      = 1
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: pooling type     = 0
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: rope type        = 0
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: rope scaling     = yarn
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: freq_base_train  = 10000.0
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: freq_scale_train = 0.025
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_ctx_orig_yarn  = 4096
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: rope_finetuned   = unknown
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: ssm_d_conv       = 0
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: ssm_d_inner      = 0
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: ssm_d_state      = 0
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: ssm_dt_rank      = 0
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: model type       = 16B
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: model ftype      = Q8_0
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: model params     = 15.71 B
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: model size       = 15.55 GiB (8.51 BPW)
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: general.name     = DeepSeek-Coder-V2-Lite-Instruct
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: BOS token        = 100000 '<｜begin▁of▁sentence｜>'
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: EOS token        = 100001 '<｜end▁of▁sentence｜>'
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: PAD token        = 100001 '<｜end▁of▁sentence｜>'
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: LF token         = 126 'Ä'
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: max token length = 256
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_layer_dense_lead   = 1
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_lora_q             = 0
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_lora_kv            = 512
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_ff_exp             = 1408
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_expert_shared      = 2
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: expert_weights_scale = 1.0
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: rope_yarn_log_mul    = 0.0707
Aug 06 15:10:54 user-desktop ollama[13639]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
Aug 06 15:10:54 user-desktop ollama[13639]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Aug 06 15:10:54 user-desktop ollama[13639]: ggml_cuda_init: found 1 CUDA devices:
Aug 06 15:10:54 user-desktop ollama[13639]:   Device 0: Tesla P40, compute capability 6.1, VMM: yes
Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_tensors: ggml ctx size =    0.32 MiB
Aug 06 15:12:43 user-desktop ollama[13639]: llm_load_tensors: offloading 27 repeating layers to GPU
Aug 06 15:12:43 user-desktop ollama[13639]: llm_load_tensors: offloading non-repeating layers to GPU
Aug 06 15:12:43 user-desktop ollama[13639]: llm_load_tensors: offloaded 28/28 layers to GPU
Aug 06 15:12:43 user-desktop ollama[13639]: llm_load_tensors:        CPU buffer size =   212.50 MiB
Aug 06 15:12:43 user-desktop ollama[13639]: llm_load_tensors:      CUDA0 buffer size = 15712.47 MiB
Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: n_ctx      = 6144
Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: n_batch    = 512
Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: n_ubatch   = 512
Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: flash_attn = 0
Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: freq_base  = 10000.0
Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: freq_scale = 0.025
Aug 06 15:12:52 user-desktop ollama[13639]: llama_kv_cache_init:      CUDA0 KV buffer size =  1620.00 MiB
Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: KV self size  = 1620.00 MiB, K (f16):  972.00 MiB, V (f16):  648.00 MiB
Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model:  CUDA_Host  output buffer size =     1.20 MiB
Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model:      CUDA0 compute buffer size =   228.00 MiB
Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model:  CUDA_Host compute buffer size =    16.01 MiB
Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: graph nodes  = 1924
Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: graph splits = 2
Aug 06 15:12:52 user-desktop ollama[13757]: INFO [main] model loaded | tid="140158488453120" timestamp=1722946372
Aug 06 15:12:52 user-desktop ollama[13639]: time=2024-08-06T15:12:52.887+03:00 level=INFO source=server.go:623 msg="llama runner started in 119.13 seconds"
Aug 06 15:12:58 user-desktop ollama[13639]: /go/src/github.com/ollama/ollama/llm/llama.cpp/src/llama.cpp:15093: Deepseek2 does not support K-shift
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13758]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13759]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13760]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13761]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13762]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13763]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13764]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13765]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13766]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13767]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13768]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13769]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13770]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13771]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13772]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13773]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13774]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13775]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13776]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13777]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13778]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13779]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13780]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13781]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13782]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13783]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13784]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13785]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13786]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13787]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13788]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13789]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13790]
Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13791]
Aug 06 15:12:59 user-desktop ollama[16692]: [Thread debugging using libthread_db enabled]
Aug 06 15:12:59 user-desktop ollama[16692]: Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Aug 06 15:12:59 user-desktop ollama[16692]: 0x00007f791682842f in __GI___wait4 (pid=16692, stat_loc=0x7ffeda1838e4, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
Aug 06 15:12:59 user-desktop ollama[13639]: 30        ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
Aug 06 15:12:59 user-desktop ollama[16692]: #0  0x00007f791682842f in __GI___wait4 (pid=16692, stat_loc=0x7ffeda1838e4, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
Aug 06 15:12:59 user-desktop ollama[16692]: 30        in ../sysdeps/unix/sysv/linux/wait4.c
Aug 06 15:12:59 user-desktop ollama[16692]: #1  0x0000000000517028 in ggml_abort ()
Aug 06 15:12:59 user-desktop ollama[16692]: #2  0x0000000000779741 in llama_kv_cache_update ()
Aug 06 15:12:59 user-desktop ollama[16692]: #3  0x0000000000783804 in llama_decode ()
Aug 06 15:12:59 user-desktop ollama[16692]: #4  0x00000000004ba577 in llama_server_context::update_slots() ()
Aug 06 15:12:59 user-desktop ollama[16692]: #5  0x000000000048f437 in llama_server_queue::start_loop() ()
Aug 06 15:12:59 user-desktop ollama[16692]: #6  0x000000000043bd69 in main ()
Aug 06 15:12:59 user-desktop ollama[16692]: [Inferior 1 (process 13757) detached]
Aug 06 15:12:59 user-desktop ollama[13639]: [GIN] 2024/08/06 - 15:12:59 | 200 |          2m6s |       127.0.0.1 | POST     "/api/chat"
Aug 06 15:18:04 user-desktop ollama[13639]: time=2024-08-06T15:18:04.684+03:00 level=WARN source=sched.go:642 msg="gpu VRAM usage didn't recover within timeout" seconds=5.067024501 model=/home/user/.ollama/models/blobs/sha256-373dcfc92e01372709b6164fc836f677a6280e25e9eac5c434c64223207bfc4f
Aug 06 15:18:04 user-desktop ollama[13639]: time=2024-08-06T15:18:04.920+03:00 level=WARN source=sched.go:642 msg="gpu VRAM usage didn't recover within timeout" seconds=5.30378812 model=/home/user/.ollama/models/blobs/sha256-373dcfc92e01372709b6164fc836f677a6280e25e9eac5c434c64223207bfc4f
Aug 06 15:18:05 user-desktop ollama[13639]: time=2024-08-06T15:18:05.173+03:00 level=WARN source=sched.go:642 msg="gpu VRAM usage didn't recover within timeout" seconds=5.556641741 model=/home/user/.ollama/models/blobs/sha256-373dcfc92e01372709b6164fc836f677a6280e25e9eac5c434c64223207bfc4f

This is also reproducible with 0.2.8.

Let me know what other info you need.

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.3.3

Originally created by @shockme on GitHub (Aug 6, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6199 ### What is the issue? The output is cut in the middle of generation. Here's the log: ``` Aug 06 15:10:46 user-desktop systemd[4465]: Started Ollama Service. Aug 06 15:10:46 user-desktop ollama[13639]: 2024/08/06 15:10:46 routes.go:1108: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://127.0.0.1:11435 OL LAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:2 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/user/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:3 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]" Aug 06 15:10:46 user-desktop ollama[13639]: time=2024-08-06T15:10:46.362+03:00 level=INFO source=images.go:781 msg="total blobs: 92" Aug 06 15:10:46 user-desktop ollama[13639]: time=2024-08-06T15:10:46.365+03:00 level=INFO source=images.go:788 msg="total unused blobs removed: 0" Aug 06 15:10:46 user-desktop ollama[13639]: time=2024-08-06T15:10:46.366+03:00 level=INFO source=routes.go:1155 msg="Listening on 127.0.0.1:11435 (version 0.3.3)" Aug 06 15:10:46 user-desktop ollama[13639]: time=2024-08-06T15:10:46.367+03:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama198573251/runners Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.148+03:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 rocm_v60102]" Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.148+03:00 level=INFO source=gpu.go:205 msg="looking for compatible GPUs" Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.472+03:00 level=WARN source=amd_linux.go:59 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.472+03:00 level=INFO source=amd_linux.go:360 msg="no compatible amdgpu devices detected" Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.472+03:00 level=INFO source=types.go:105 msg="inference compute" id=GPU-5968b8f6-eb32-e5b2-37a0-2a8637c2ae09 library=cuda compute=6.1 driver=12.2 name="Tesla P40" total="23.9 GiB" available="23.7 GiB" Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.757+03:00 level=INFO source=sched.go:710 msg="new model will fit in available VRAM in single GPU, loading" model=/home/user/.ollama/models/blobs/sha256-373dcfc92e01372709b6164fc836f677a6280e25e9eac5c434c64223207bfc4f gpu=GPU-5968b8f6-eb32-e5b2-37a0-2a8637c2ae09 parallel=3 available=25430458368 required="17.7 GiB" Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.759+03:00 level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=28 layers.offload=28 layers.split="" memory.available="[23.7 GiB]" memory.required.full="17.7 GiB" memory.required.partial="17.7 GiB" memory.required.kv="1.6 GiB" memory.required.allocations="[17.7 GiB]" memory.weights.total="16.7 GiB" memory.weights.repeating="16.5 GiB" memory.weights.nonrepeating="212.5 MiB" memory.graph.full="228.0 MiB" memory.graph.partial="376.1 MiB" Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.760+03:00 level=INFO source=server.go:384 msg="starting llama server" cmd="/tmp/ollama198573251/runners/cuda_v11/ollama_llama_server --model /home/user/.ollama/models/blobs/sha256-373dcfc92e01372709b6164fc836f677a6280e25e9eac5c434c64223207bfc4f --ctx-size 6144 --batch-size 512 --embedding --log-disable --n-gpu-layers 28 --parallel 3 --port 40025" Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.760+03:00 level=INFO source=sched.go:445 msg="loaded runners" count=1 Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.760+03:00 level=INFO source=server.go:584 msg="waiting for llama runner to start responding" Aug 06 15:10:53 user-desktop ollama[13639]: time=2024-08-06T15:10:53.761+03:00 level=INFO source=server.go:618 msg="waiting for server to become available" status="llm server error" Aug 06 15:10:53 user-desktop ollama[13757]: INFO [main] build info | build=1 commit="6eeaeba" tid="140158488453120" timestamp=1722946253 Aug 06 15:10:53 user-desktop ollama[13757]: INFO [main] system info | n_threads=16 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="140158488453120" timestamp=1722946253 total_threads=32 Aug 06 15:10:53 user-desktop ollama[13757]: INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="31" port="40025" tid="140158488453120" timestamp=1722946253 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: loaded meta data with 38 key-value pairs and 377 tensors from /home/user/.ollama/models/blobs/sha256-373dcfc92e01372709b6164fc836f677a6280e25e9eac5c434c64223207bfc4f (version GGUF V3 (latest)) Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 0: general.architecture str = deepseek2 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 1: general.name str = DeepSeek-Coder-V2-Lite-Instruct Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 2: deepseek2.block_count u32 = 27 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 3: deepseek2.context_length u32 = 163840 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 4: deepseek2.embedding_length u32 = 2048 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 5: deepseek2.feed_forward_length u32 = 10944 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 6: deepseek2.attention.head_count u32 = 16 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 7: deepseek2.attention.head_count_kv u32 = 16 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 8: deepseek2.rope.freq_base f32 = 10000.000000 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 9: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 10: deepseek2.expert_used_count u32 = 6 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 11: general.file_type u32 = 7 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 12: deepseek2.leading_dense_block_count u32 = 1 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 13: deepseek2.vocab_size u32 = 102400 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 14: deepseek2.attention.kv_lora_rank u32 = 512 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 15: deepseek2.attention.key_length u32 = 192 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 16: deepseek2.attention.value_length u32 = 128 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 17: deepseek2.expert_feed_forward_length u32 = 1408 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 18: deepseek2.expert_count u32 = 64 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 19: deepseek2.expert_shared_count u32 = 2 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 20: deepseek2.expert_weights_scale f32 = 1.000000 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 21: deepseek2.rope.dimension_count u32 = 64 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 22: deepseek2.rope.scaling.type str = yarn Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 23: deepseek2.rope.scaling.factor f32 = 40.000000 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 24: deepseek2.rope.scaling.original_context_length u32 = 4096 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 25: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.070700 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 26: tokenizer.ggml.model str = gpt2 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 27: tokenizer.ggml.pre str = deepseek-llm Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 28: tokenizer.ggml.tokens arr[str,102400] = ["!", "\"", "#", "$", "%", "&", "'", ... Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 29: tokenizer.ggml.token_type arr[i32,102400] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 30: tokenizer.ggml.merges arr[str,99757] = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e... Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 31: tokenizer.ggml.bos_token_id u32 = 100000 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 32: tokenizer.ggml.eos_token_id u32 = 100001 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 33: tokenizer.ggml.padding_token_id u32 = 100001 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 34: tokenizer.ggml.add_bos_token bool = true Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 35: tokenizer.ggml.add_eos_token bool = false Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 36: tokenizer.chat_template str = {% if not add_generation_prompt is de... Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - kv 37: general.quantization_version u32 = 2 Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - type f32: 108 tensors Aug 06 15:10:53 user-desktop ollama[13639]: llama_model_loader: - type q8_0: 269 tensors Aug 06 15:10:54 user-desktop ollama[13639]: time=2024-08-06T15:10:54.014+03:00 level=INFO source=server.go:618 msg="waiting for server to become available" status="llm server loading model" Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_vocab: special tokens cache size = 2400 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_vocab: token to piece cache size = 0.6661 MB Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: format = GGUF V3 (latest) Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: arch = deepseek2 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: vocab type = BPE Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_vocab = 102400 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_merges = 99757 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: vocab_only = 0 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_ctx_train = 163840 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_embd = 2048 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_layer = 27 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_head = 16 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_head_kv = 16 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_rot = 64 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_swa = 0 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_embd_head_k = 192 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_embd_head_v = 128 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_gqa = 1 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_embd_k_gqa = 3072 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_embd_v_gqa = 2048 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: f_norm_eps = 0.0e+00 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: f_norm_rms_eps = 1.0e-06 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: f_clamp_kqv = 0.0e+00 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: f_logit_scale = 0.0e+00 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_ff = 10944 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_expert = 64 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_expert_used = 6 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: causal attn = 1 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: pooling type = 0 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: rope type = 0 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: rope scaling = yarn Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: freq_base_train = 10000.0 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: freq_scale_train = 0.025 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_ctx_orig_yarn = 4096 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: rope_finetuned = unknown Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: ssm_d_conv = 0 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: ssm_d_inner = 0 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: ssm_d_state = 0 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: ssm_dt_rank = 0 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: model type = 16B Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: model ftype = Q8_0 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: model params = 15.71 B Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: model size = 15.55 GiB (8.51 BPW) Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: general.name = DeepSeek-Coder-V2-Lite-Instruct Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: BOS token = 100000 '<｜begin▁of▁sentence｜>' Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: EOS token = 100001 '<｜end▁of▁sentence｜>' Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: PAD token = 100001 '<｜end▁of▁sentence｜>' Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: LF token = 126 'Ä' Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: max token length = 256 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_layer_dense_lead = 1 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_lora_q = 0 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_lora_kv = 512 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_ff_exp = 1408 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: n_expert_shared = 2 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: expert_weights_scale = 1.0 Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_print_meta: rope_yarn_log_mul = 0.0707 Aug 06 15:10:54 user-desktop ollama[13639]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no Aug 06 15:10:54 user-desktop ollama[13639]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no Aug 06 15:10:54 user-desktop ollama[13639]: ggml_cuda_init: found 1 CUDA devices: Aug 06 15:10:54 user-desktop ollama[13639]: Device 0: Tesla P40, compute capability 6.1, VMM: yes Aug 06 15:10:54 user-desktop ollama[13639]: llm_load_tensors: ggml ctx size = 0.32 MiB Aug 06 15:12:43 user-desktop ollama[13639]: llm_load_tensors: offloading 27 repeating layers to GPU Aug 06 15:12:43 user-desktop ollama[13639]: llm_load_tensors: offloading non-repeating layers to GPU Aug 06 15:12:43 user-desktop ollama[13639]: llm_load_tensors: offloaded 28/28 layers to GPU Aug 06 15:12:43 user-desktop ollama[13639]: llm_load_tensors: CPU buffer size = 212.50 MiB Aug 06 15:12:43 user-desktop ollama[13639]: llm_load_tensors: CUDA0 buffer size = 15712.47 MiB Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: n_ctx = 6144 Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: n_batch = 512 Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: n_ubatch = 512 Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: flash_attn = 0 Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: freq_base = 10000.0 Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: freq_scale = 0.025 Aug 06 15:12:52 user-desktop ollama[13639]: llama_kv_cache_init: CUDA0 KV buffer size = 1620.00 MiB Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: KV self size = 1620.00 MiB, K (f16): 972.00 MiB, V (f16): 648.00 MiB Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: CUDA_Host output buffer size = 1.20 MiB Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: CUDA0 compute buffer size = 228.00 MiB Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: CUDA_Host compute buffer size = 16.01 MiB Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: graph nodes = 1924 Aug 06 15:12:52 user-desktop ollama[13639]: llama_new_context_with_model: graph splits = 2 Aug 06 15:12:52 user-desktop ollama[13757]: INFO [main] model loaded | tid="140158488453120" timestamp=1722946372 Aug 06 15:12:52 user-desktop ollama[13639]: time=2024-08-06T15:12:52.887+03:00 level=INFO source=server.go:623 msg="llama runner started in 119.13 seconds" Aug 06 15:12:58 user-desktop ollama[13639]: /go/src/github.com/ollama/ollama/llm/llama.cpp/src/llama.cpp:15093: Deepseek2 does not support K-shift Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13758] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13759] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13760] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13761] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13762] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13763] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13764] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13765] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13766] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13767] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13768] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13769] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13770] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13771] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13772] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13773] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13774] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13775] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13776] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13777] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13778] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13779] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13780] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13781] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13782] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13783] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13784] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13785] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13786] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13787] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13788] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13789] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13790] Aug 06 15:12:58 user-desktop ollama[16692]: [New LWP 13791] Aug 06 15:12:59 user-desktop ollama[16692]: [Thread debugging using libthread_db enabled] Aug 06 15:12:59 user-desktop ollama[16692]: Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Aug 06 15:12:59 user-desktop ollama[16692]: 0x00007f791682842f in __GI___wait4 (pid=16692, stat_loc=0x7ffeda1838e4, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30 Aug 06 15:12:59 user-desktop ollama[13639]: 30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory. Aug 06 15:12:59 user-desktop ollama[16692]: #0 0x00007f791682842f in __GI___wait4 (pid=16692, stat_loc=0x7ffeda1838e4, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30 Aug 06 15:12:59 user-desktop ollama[16692]: 30 in ../sysdeps/unix/sysv/linux/wait4.c Aug 06 15:12:59 user-desktop ollama[16692]: #1 0x0000000000517028 in ggml_abort () Aug 06 15:12:59 user-desktop ollama[16692]: #2 0x0000000000779741 in llama_kv_cache_update () Aug 06 15:12:59 user-desktop ollama[16692]: #3 0x0000000000783804 in llama_decode () Aug 06 15:12:59 user-desktop ollama[16692]: #4 0x00000000004ba577 in llama_server_context::update_slots() () Aug 06 15:12:59 user-desktop ollama[16692]: #5 0x000000000048f437 in llama_server_queue::start_loop() () Aug 06 15:12:59 user-desktop ollama[16692]: #6 0x000000000043bd69 in main () Aug 06 15:12:59 user-desktop ollama[16692]: [Inferior 1 (process 13757) detached] Aug 06 15:12:59 user-desktop ollama[13639]: [GIN] 2024/08/06 - 15:12:59 | 200 | 2m6s | 127.0.0.1 | POST "/api/chat" Aug 06 15:18:04 user-desktop ollama[13639]: time=2024-08-06T15:18:04.684+03:00 level=WARN source=sched.go:642 msg="gpu VRAM usage didn't recover within timeout" seconds=5.067024501 model=/home/user/.ollama/models/blobs/sha256-373dcfc92e01372709b6164fc836f677a6280e25e9eac5c434c64223207bfc4f Aug 06 15:18:04 user-desktop ollama[13639]: time=2024-08-06T15:18:04.920+03:00 level=WARN source=sched.go:642 msg="gpu VRAM usage didn't recover within timeout" seconds=5.30378812 model=/home/user/.ollama/models/blobs/sha256-373dcfc92e01372709b6164fc836f677a6280e25e9eac5c434c64223207bfc4f Aug 06 15:18:05 user-desktop ollama[13639]: time=2024-08-06T15:18:05.173+03:00 level=WARN source=sched.go:642 msg="gpu VRAM usage didn't recover within timeout" seconds=5.556641741 model=/home/user/.ollama/models/blobs/sha256-373dcfc92e01372709b6164fc836f677a6280e25e9eac5c434c64223207bfc4f ``` This is also reproducible with 0.2.8. Let me know what other info you need. ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.3.3
GiteaMirror added the bug label 2026-04-12 14:42:31 -05:00
GiteaMirror commented 2026-04-12 14:42:32 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 6, 2024):

Aug 06 15:12:58 user-desktop ollama[13639]: /go/src/github.com/ollama/ollama/llm/llama.cpp/src/llama.cpp:15093: Deepseek2 does not support K-shift

As per the error, there are problems with supporting this model. There are possible fixes from a couple of PRs in https://github.com/ollama/ollama/issues/5975.

<!-- gh-comment-id:2271234442 --> @rick-github commented on GitHub (Aug 6, 2024): ``` Aug 06 15:12:58 user-desktop ollama[13639]: /go/src/github.com/ollama/ollama/llm/llama.cpp/src/llama.cpp:15093: Deepseek2 does not support K-shift ``` As per the error, there are problems with supporting this model. There are possible fixes from a couple of PRs in https://github.com/ollama/ollama/issues/5975.
GiteaMirror commented 2026-04-12 14:42:32 -05:00
Author
Owner
Copy Link

@httpjamesm commented on GitHub (Aug 12, 2024):

I'm also facing this issue on AMD.

When using streaming, I see these messages:

ollama  | /go/src/github.com/ollama/ollama/llm/llama.cpp/src/llama.cpp:15104: Deepseek2 does not support K-shift
ollama  | No symbol table is loaded.  Use the "file" command.
ollama  | ptrace: Operation not permitted.
ollama  | No stack.
ollama  | The program is not being run.
ollama  | [GIN] 2024/08/12 - 19:50:08 | 200 | 19.410496291s |   100.100.86.76 | POST     "/api/generate"
ollama  | time=2024-08-12T19:50:10.659Z level=WARN source=server.go:511 msg="llama runner process no longer running" sys=134 string="signal: aborted (core dumped)"
ollama  | time=2024-08-12T19:50:15.659Z level=WARN source=sched.go:642 msg="gpu VRAM usage didn't recover within timeout" seconds=5.000203552 model=/root/.ollama/models/blobs/sha256-5ff0abeeac1d2dbdd5455c0b49ba3b29a9ce3c1fb181b2eef2e948689d55d046

It seems the llama runner process is crashing.

Version: 0.3.4, replicated on both Docker and normal install on Ubuntu 24.04 LTS and 22.04 LTS

<!-- gh-comment-id:2284795013 --> @httpjamesm commented on GitHub (Aug 12, 2024): I'm also facing this issue on AMD. When using streaming, I see these messages: ``` ollama | /go/src/github.com/ollama/ollama/llm/llama.cpp/src/llama.cpp:15104: Deepseek2 does not support K-shift ollama | No symbol table is loaded. Use the "file" command. ollama | ptrace: Operation not permitted. ollama | No stack. ollama | The program is not being run. ollama | [GIN] 2024/08/12 - 19:50:08 | 200 | 19.410496291s | 100.100.86.76 | POST "/api/generate" ollama | time=2024-08-12T19:50:10.659Z level=WARN source=server.go:511 msg="llama runner process no longer running" sys=134 string="signal: aborted (core dumped)" ollama | time=2024-08-12T19:50:15.659Z level=WARN source=sched.go:642 msg="gpu VRAM usage didn't recover within timeout" seconds=5.000203552 model=/root/.ollama/models/blobs/sha256-5ff0abeeac1d2dbdd5455c0b49ba3b29a9ce3c1fb181b2eef2e948689d55d046 ``` It seems the llama runner process is crashing. Version: 0.3.4, replicated on both Docker and normal install on Ubuntu 24.04 LTS and 22.04 LTS
GiteaMirror commented 2026-04-12 14:42:32 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 13, 2024):

ollama  | /go/src/github.com/ollama/ollama/llm/llama.cpp/src/llama.cpp:15104: Deepseek2 does not support K-shift

As per the error, there are problems with supporting this model. There are possible fixes from a couple of PRs in https://github.com/ollama/ollama/issues/5975.

<!-- gh-comment-id:2285166659 --> @rick-github commented on GitHub (Aug 13, 2024): ``` ollama | /go/src/github.com/ollama/ollama/llm/llama.cpp/src/llama.cpp:15104: Deepseek2 does not support K-shift ``` As per the error, there are problems with supporting this model. There are possible fixes from a couple of PRs in https://github.com/ollama/ollama/issues/5975.
GiteaMirror commented 2026-04-12 14:42:33 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 18, 2024):

Workaround discussed in https://github.com/ggerganov/llama.cpp/issues/8862.

<!-- gh-comment-id:2295280122 --> @rick-github commented on GitHub (Aug 18, 2024): Workaround discussed in https://github.com/ggerganov/llama.cpp/issues/8862.
GiteaMirror commented 2026-04-12 14:42:33 -05:00
Author
Owner
Copy Link

@Walker555 commented on GitHub (Aug 19, 2024):

Workaround discussed in ggerganov/llama.cpp#8862.

Workaround: create an overridden model with correct parameters

tee -a deepseek-coder-v2-16b-lite-instruct-q8-fix.ollama <<EOF
FROM deepseek-coder-v2:16b-lite-instruct-q8_0
PARAMETER temperature 0
PARAMETER num_ctx 24576
PARAMETER num_predict 8192
EOF

ollama create deepseek-coder-v2-16b-lite-instruct-q8-fix -f deepseek-coder-v2-16b-lite-instruct-q8-fix.ollama
<!-- gh-comment-id:2295952982 --> @Walker555 commented on GitHub (Aug 19, 2024): > Workaround discussed in [ggerganov/llama.cpp#8862](https://github.com/ggerganov/llama.cpp/issues/8862). Workaround: create an overridden model with correct parameters ``` tee -a deepseek-coder-v2-16b-lite-instruct-q8-fix.ollama <<EOF FROM deepseek-coder-v2:16b-lite-instruct-q8_0 PARAMETER temperature 0 PARAMETER num_ctx 24576 PARAMETER num_predict 8192 EOF ollama create deepseek-coder-v2-16b-lite-instruct-q8-fix -f deepseek-coder-v2-16b-lite-instruct-q8-fix.ollama ```
GiteaMirror commented 2026-04-12 14:42:33 -05:00
Author
Owner
Copy Link

@U0M0Z commented on GitHub (Aug 21, 2024):

I guess I am having a similar issue using ollama and deepseek-coder-v2 serviced via aider-chat. After updating the Ollama application in my OS I have been encountering an Unknown error using the following code:

import os

from aider.coders import Coder
from aider.models import Model
from aider.io import InputOutput

From prompts import coder_prompt

folder_name = os.path.join(‘results_dir’)

# Define folder and 
e_file = os.path.join(folder_name, "first_experiment.py")
v_file = os.path.join(folder_name, "plotting.py")
n_file = os.path.join(folder_name, "notes.txt")

reference_run = 'first_experiment'
fnames = [e_file, v_file, n_file]

# Define I/O for model 
io = InputOutput(yes=True, chat_history_file=f"{folder_name}/{reference_run}_aider.txt")

# Define main model to leverage
main_model = Model("ollama/deepseek-coder-v2:16b")

# Instantiate coder model
coder = Coder.create(main_model=main_model, fnames=fnames, io=io, stream=False, use_git=False, edit_format="diff”)

coder_out = coder.run(coder_prompt)
print(coder_out)

————————————————————————————————————————————————————————————————————

litellm.APIConnectionError: {"error":"an unknown error was encountered while running the model "}
Traceback (most recent call last):
  File "/Users/<username>/miniconda3/envs/arc/lib/python3.11/site-packages/litellm/main.py", line 2424, in completion
    generator = ollama.get_ollama_response(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/<username>/miniconda3/envs/arc/lib/python3.11/site-packages/litellm/llms/ollama.py", line 268, in get_ollama_response
    raise OllamaError(status_code=response.status_code, message=response.text)
litellm.llms.ollama.OllamaError: {"error":"an unknown error was encountered while running the model "}

These are the packages in my environment:

Package Version

aider-chat 0.49.1
litellm 1.43.4
ollama 0.3.1

I am not really understanding how to solve the problem.

<!-- gh-comment-id:2302834046 --> @U0M0Z commented on GitHub (Aug 21, 2024): I guess I am having a similar issue using ollama and deepseek-coder-v2 serviced via aider-chat. After updating the Ollama application in my OS I have been encountering an Unknown error using the following code: ``` import os from aider.coders import Coder from aider.models import Model from aider.io import InputOutput From prompts import coder_prompt folder_name = os.path.join(‘results_dir’) # Define folder and e_file = os.path.join(folder_name, "first_experiment.py") v_file = os.path.join(folder_name, "plotting.py") n_file = os.path.join(folder_name, "notes.txt") reference_run = 'first_experiment' fnames = [e_file, v_file, n_file] # Define I/O for model io = InputOutput(yes=True, chat_history_file=f"{folder_name}/{reference_run}_aider.txt") # Define main model to leverage main_model = Model("ollama/deepseek-coder-v2:16b") # Instantiate coder model coder = Coder.create(main_model=main_model, fnames=fnames, io=io, stream=False, use_git=False, edit_format="diff”) coder_out = coder.run(coder_prompt) print(coder_out) ———————————————————————————————————————————————————————————————————— litellm.APIConnectionError: {"error":"an unknown error was encountered while running the model "} Traceback (most recent call last): File "/Users/<username>/miniconda3/envs/arc/lib/python3.11/site-packages/litellm/main.py", line 2424, in completion generator = ollama.get_ollama_response( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/<username>/miniconda3/envs/arc/lib/python3.11/site-packages/litellm/llms/ollama.py", line 268, in get_ollama_response raise OllamaError(status_code=response.status_code, message=response.text) litellm.llms.ollama.OllamaError: {"error":"an unknown error was encountered while running the model "} ``` These are the packages in my environment: Package Version --------------------- ------------- aider-chat 0.49.1 litellm 1.43.4 ollama 0.3.1 I am not really understanding how to solve the problem.
GiteaMirror commented 2026-04-12 14:42:34 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 21, 2024):

Server logs may help in debugging.

<!-- gh-comment-id:2302843240 --> @rick-github commented on GitHub (Aug 21, 2024): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) may help in debugging.
GiteaMirror commented 2026-04-12 14:42:34 -05:00
Author
Owner
Copy Link

@nicolasembleton commented on GitHub (Sep 17, 2024):

Workaround discussed in ggerganov/llama.cpp#8862.

Workaround: create an overridden model with correct parameters

tee -a deepseek-coder-v2-16b-lite-instruct-q8-fix.ollama <<EOF
FROM deepseek-coder-v2:16b-lite-instruct-q8_0
PARAMETER temperature 0
PARAMETER num_ctx 24576
PARAMETER num_predict 8192
EOF

ollama create deepseek-coder-v2-16b-lite-instruct-q8-fix -f deepseek-coder-v2-16b-lite-instruct-q8-fix.ollama

Brilliant summary. It worked perfectly. Thanks!

<!-- gh-comment-id:2354887394 --> @nicolasembleton commented on GitHub (Sep 17, 2024): > > Workaround discussed in [ggerganov/llama.cpp#8862](https://github.com/ggerganov/llama.cpp/issues/8862). > > Workaround: create an overridden model with correct parameters > > ``` > tee -a deepseek-coder-v2-16b-lite-instruct-q8-fix.ollama <<EOF > FROM deepseek-coder-v2:16b-lite-instruct-q8_0 > PARAMETER temperature 0 > PARAMETER num_ctx 24576 > PARAMETER num_predict 8192 > EOF > > ollama create deepseek-coder-v2-16b-lite-instruct-q8-fix -f deepseek-coder-v2-16b-lite-instruct-q8-fix.ollama > ``` Brilliant summary. It worked perfectly. Thanks!
GiteaMirror commented 2026-04-12 14:42:34 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Oct 31, 2024):

Dup of #5975

<!-- gh-comment-id:2450545890 --> @dhiltgen commented on GitHub (Oct 31, 2024): Dup of #5975
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#3872
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?