github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 08:02:14 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #4545] Ollama stops serving requests after 10-15 minutes #28610

New Issue
Closed
opened 2026-04-22 07:01:33 -05:00 by GiteaMirror · 54 comments
GiteaMirror commented 2026-04-22 07:01:33 -05:00
Owner
Copy Link

Originally created by @iganev on GitHub (May 20, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/4545

Originally assigned to: @dhiltgen on GitHub.

What is the issue?

Using ollama:latest with nvidia-docker and 2x4090.

Tried blasting a bunch of 256 words long text snippets to ollama for embeddings generation using all-minilm:l6-v2.

Every time I start the task, it works for about 10-15 minutes, then ollama starts initially hanging the requests till timeout, but later requests respond immediately with error {"error":"failed to generate embedding"}.

Looking at the logs:

[GIN] 2024/05/20 - 20:50:30 | 200 |  445.487928ms |      172.20.1.1 | POST     "/api/embeddings"
[GIN] 2024/05/20 - 20:50:30 | 200 |   446.51795ms |      172.20.1.1 | POST     "/api/embeddings"
[GIN] 2024/05/20 - 20:50:30 | 200 |  441.551062ms |      172.20.1.1 | POST     "/api/embeddings"
[GIN] 2024/05/20 - 20:50:30 | 200 |  439.109191ms |      172.20.1.1 | POST     "/api/embeddings"
[GIN] 2024/05/20 - 20:50:30 | 200 |  395.145587ms |      172.20.1.1 | POST     "/api/embeddings"
[GIN] 2024/05/20 - 20:50:30 | 200 |  393.074237ms |      172.20.1.1 | POST     "/api/embeddings"
[GIN] 2024/05/20 - 20:50:30 | 200 |  434.758996ms |      172.20.1.1 | POST     "/api/embeddings"
[GIN] 2024/05/20 - 20:50:30 | 200 |  394.631605ms |      172.20.1.1 | POST     "/api/embeddings"
[GIN] 2024/05/20 - 20:50:30 | 200 |  401.354417ms |      172.20.1.1 | POST     "/api/embeddings"
[GIN] 2024/05/20 - 20:50:30 | 200 |  443.634336ms |      172.20.1.1 | POST     "/api/embeddings"
time=2024-05-20T21:48:01.015Z level=ERROR source=server.go:800 msg="Failed to acquire semaphore" error="context canceled"
time=2024-05-20T21:48:01.015Z level=ERROR source=server.go:800 msg="Failed to acquire semaphore" error="context canceled"
time=2024-05-20T21:48:01.015Z level=ERROR source=server.go:800 msg="Failed to acquire semaphore" error="context canceled"
time=2024-05-20T21:48:01.015Z level=ERROR source=server.go:800 msg="Failed to acquire semaphore" error="context canceled"
time=2024-05-20T21:48:01.015Z level=INFO source=routes.go:400 msg="embedding generation failed: context canceled"
time=2024-05-20T21:48:01.015Z level=INFO source=routes.go:400 msg="embedding generation failed: context canceled"
time=2024-05-20T21:48:01.015Z level=INFO source=routes.go:400 msg="embedding generation failed: context canceled"
time=2024-05-20T21:48:01.015Z level=ERROR source=server.go:800 msg="Failed to acquire semaphore" error="context canceled"
time=2024-05-20T21:48:01.015Z level=INFO source=routes.go:400 msg="embedding generation failed: context canceled"
time=2024-05-20T21:48:01.015Z level=INFO source=routes.go:400 msg="embedding generation failed: do embedding request: Post \"http://127.0.0.1:33007/embedding\": context canceled"
time=2024-05-20T21:48:01.015Z level=INFO source=routes.go:400 msg="embedding generation failed: context canceled"
time=2024-05-20T21:48:01.015Z level=ERROR source=server.go:800 msg="Failed to acquire semaphore" error="context canceled"
time=2024-05-20T21:48:01.015Z level=INFO source=routes.go:400 msg="embedding generation failed: context canceled"
[GIN] 2024/05/20 - 21:48:01 | 500 |        57m30s |      172.20.1.1 | POST     "/api/embeddings"
time=2024-05-20T21:48:01.015Z level=ERROR source=server.go:800 msg="Failed to acquire semaphore" error="context canceled"
time=2024-05-20T21:48:01.016Z level=INFO source=routes.go:400 msg="embedding generation failed: context canceled"
[GIN] 2024/05/20 - 21:48:01 | 500 |        57m30s |      172.20.1.1 | POST     "/api/embeddings"
[GIN] 2024/05/20 - 21:48:01 | 500 |        57m30s |      172.20.1.1 | POST     "/api/embeddings"
[GIN] 2024/05/20 - 21:48:01 | 500 |        57m30s |      172.20.1.1 | POST     "/api/embeddings"
[GIN] 2024/05/20 - 21:48:01 | 500 |        57m30s |      172.20.1.1 | POST     "/api/embeddings"
[GIN] 2024/05/20 - 21:48:01 | 500 |        57m30s |      172.20.1.1 | POST     "/api/embeddings"
[GIN] 2024/05/20 - 21:48:01 | 500 |        57m30s |      172.20.1.1 | POST     "/api/embeddings"
[GIN] 2024/05/20 - 21:48:01 | 500 |        57m30s |      172.20.1.1 | POST     "/api/embeddings"
[GIN] 2024/05/20 - 21:48:38 | 500 |  369.514148ms |      10.252.1.3 | POST     "/api/embeddings"
time=2024-05-20T21:48:38.515Z level=INFO source=routes.go:400 msg="embedding generation failed: no slots available after 10 retries"

Here's my startup log as well:

2024/05/20 19:54:46 routes.go:1008: INFO server config env="map[OLLAMA_DEBUG:false OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MAX_VRAM:0 OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:*] OLLAMA_RUNNERS_DIR: OLLAMA_TMPDIR:]"
time=2024-05-20T19:54:46.610Z level=INFO source=images.go:704 msg="total blobs: 43"
time=2024-05-20T19:54:46.611Z level=INFO source=images.go:711 msg="total unused blobs removed: 0"
time=2024-05-20T19:54:46.612Z level=INFO source=routes.go:1054 msg="Listening on [::]:11434 (version 0.1.38)"
time=2024-05-20T19:54:46.612Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama2944120655/runners
time=2024-05-20T19:54:47.964Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 rocm_v60002]"
time=2024-05-20T19:54:48.961Z level=INFO source=types.go:71 msg="inference compute" id=GPU-d419dbd5-adab-6e8b-e46b-4e45491c3e50 library=cuda compute=8.9 driver=12.4 name="NVIDIA GeForce RTX 4090" total="23.6 GiB" available="23.3 GiB"
time=2024-05-20T19:54:48.961Z level=INFO source=types.go:71 msg="inference compute" id=GPU-6da9f13b-9b65-b30a-fd59-910f358a7824 library=cuda compute=8.9 driver=12.4 name="NVIDIA GeForce RTX 4090" total="23.6 GiB" available="23.3 GiB"

Issue persists until I restart the container, then repeats in the next 10-15 minutes.

Any suggestions?

Thanks in advance <3

OS

Linux, Docker

GPU

Nvidia

CPU

Intel

Ollama version

0.1.38

Originally created by @iganev on GitHub (May 20, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/4545 Originally assigned to: @dhiltgen on GitHub. ### What is the issue? Using `ollama:latest` with nvidia-docker and 2x4090. Tried blasting a bunch of 256 words long text snippets to ollama for embeddings generation using `all-minilm:l6-v2`. Every time I start the task, it works for about 10-15 minutes, then ollama starts initially hanging the requests till timeout, but later requests respond immediately with error `{"error":"failed to generate embedding"}`. Looking at the logs: ``` [GIN] 2024/05/20 - 20:50:30 | 200 | 445.487928ms | 172.20.1.1 | POST "/api/embeddings" [GIN] 2024/05/20 - 20:50:30 | 200 | 446.51795ms | 172.20.1.1 | POST "/api/embeddings" [GIN] 2024/05/20 - 20:50:30 | 200 | 441.551062ms | 172.20.1.1 | POST "/api/embeddings" [GIN] 2024/05/20 - 20:50:30 | 200 | 439.109191ms | 172.20.1.1 | POST "/api/embeddings" [GIN] 2024/05/20 - 20:50:30 | 200 | 395.145587ms | 172.20.1.1 | POST "/api/embeddings" [GIN] 2024/05/20 - 20:50:30 | 200 | 393.074237ms | 172.20.1.1 | POST "/api/embeddings" [GIN] 2024/05/20 - 20:50:30 | 200 | 434.758996ms | 172.20.1.1 | POST "/api/embeddings" [GIN] 2024/05/20 - 20:50:30 | 200 | 394.631605ms | 172.20.1.1 | POST "/api/embeddings" [GIN] 2024/05/20 - 20:50:30 | 200 | 401.354417ms | 172.20.1.1 | POST "/api/embeddings" [GIN] 2024/05/20 - 20:50:30 | 200 | 443.634336ms | 172.20.1.1 | POST "/api/embeddings" time=2024-05-20T21:48:01.015Z level=ERROR source=server.go:800 msg="Failed to acquire semaphore" error="context canceled" time=2024-05-20T21:48:01.015Z level=ERROR source=server.go:800 msg="Failed to acquire semaphore" error="context canceled" time=2024-05-20T21:48:01.015Z level=ERROR source=server.go:800 msg="Failed to acquire semaphore" error="context canceled" time=2024-05-20T21:48:01.015Z level=ERROR source=server.go:800 msg="Failed to acquire semaphore" error="context canceled" time=2024-05-20T21:48:01.015Z level=INFO source=routes.go:400 msg="embedding generation failed: context canceled" time=2024-05-20T21:48:01.015Z level=INFO source=routes.go:400 msg="embedding generation failed: context canceled" time=2024-05-20T21:48:01.015Z level=INFO source=routes.go:400 msg="embedding generation failed: context canceled" time=2024-05-20T21:48:01.015Z level=ERROR source=server.go:800 msg="Failed to acquire semaphore" error="context canceled" time=2024-05-20T21:48:01.015Z level=INFO source=routes.go:400 msg="embedding generation failed: context canceled" time=2024-05-20T21:48:01.015Z level=INFO source=routes.go:400 msg="embedding generation failed: do embedding request: Post \"http://127.0.0.1:33007/embedding\": context canceled" time=2024-05-20T21:48:01.015Z level=INFO source=routes.go:400 msg="embedding generation failed: context canceled" time=2024-05-20T21:48:01.015Z level=ERROR source=server.go:800 msg="Failed to acquire semaphore" error="context canceled" time=2024-05-20T21:48:01.015Z level=INFO source=routes.go:400 msg="embedding generation failed: context canceled" [GIN] 2024/05/20 - 21:48:01 | 500 | 57m30s | 172.20.1.1 | POST "/api/embeddings" time=2024-05-20T21:48:01.015Z level=ERROR source=server.go:800 msg="Failed to acquire semaphore" error="context canceled" time=2024-05-20T21:48:01.016Z level=INFO source=routes.go:400 msg="embedding generation failed: context canceled" [GIN] 2024/05/20 - 21:48:01 | 500 | 57m30s | 172.20.1.1 | POST "/api/embeddings" [GIN] 2024/05/20 - 21:48:01 | 500 | 57m30s | 172.20.1.1 | POST "/api/embeddings" [GIN] 2024/05/20 - 21:48:01 | 500 | 57m30s | 172.20.1.1 | POST "/api/embeddings" [GIN] 2024/05/20 - 21:48:01 | 500 | 57m30s | 172.20.1.1 | POST "/api/embeddings" [GIN] 2024/05/20 - 21:48:01 | 500 | 57m30s | 172.20.1.1 | POST "/api/embeddings" [GIN] 2024/05/20 - 21:48:01 | 500 | 57m30s | 172.20.1.1 | POST "/api/embeddings" [GIN] 2024/05/20 - 21:48:01 | 500 | 57m30s | 172.20.1.1 | POST "/api/embeddings" [GIN] 2024/05/20 - 21:48:38 | 500 | 369.514148ms | 10.252.1.3 | POST "/api/embeddings" time=2024-05-20T21:48:38.515Z level=INFO source=routes.go:400 msg="embedding generation failed: no slots available after 10 retries" ``` Here's my startup log as well: ``` 2024/05/20 19:54:46 routes.go:1008: INFO server config env="map[OLLAMA_DEBUG:false OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MAX_VRAM:0 OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:*] OLLAMA_RUNNERS_DIR: OLLAMA_TMPDIR:]" time=2024-05-20T19:54:46.610Z level=INFO source=images.go:704 msg="total blobs: 43" time=2024-05-20T19:54:46.611Z level=INFO source=images.go:711 msg="total unused blobs removed: 0" time=2024-05-20T19:54:46.612Z level=INFO source=routes.go:1054 msg="Listening on [::]:11434 (version 0.1.38)" time=2024-05-20T19:54:46.612Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama2944120655/runners time=2024-05-20T19:54:47.964Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 rocm_v60002]" time=2024-05-20T19:54:48.961Z level=INFO source=types.go:71 msg="inference compute" id=GPU-d419dbd5-adab-6e8b-e46b-4e45491c3e50 library=cuda compute=8.9 driver=12.4 name="NVIDIA GeForce RTX 4090" total="23.6 GiB" available="23.3 GiB" time=2024-05-20T19:54:48.961Z level=INFO source=types.go:71 msg="inference compute" id=GPU-6da9f13b-9b65-b30a-fd59-910f358a7824 library=cuda compute=8.9 driver=12.4 name="NVIDIA GeForce RTX 4090" total="23.6 GiB" available="23.3 GiB" ``` Issue persists until I restart the container, then repeats in the next 10-15 minutes. Any suggestions? Thanks in advance <3 ### OS Linux, Docker ### GPU Nvidia ### CPU Intel ### Ollama version 0.1.38
GiteaMirror added the needs more infobug labels 2026-04-22 07:01:33 -05:00
GiteaMirror commented 2026-04-22 07:01:36 -05:00
Author
Owner
Copy Link

@iganev commented on GitHub (May 20, 2024):

Tried bumping the queue to 1024 and the parallel num to 10. Watching the log live (-fn100) I see this popping up every once in a while:

[GIN] 2024/05/20 - 22:17:48 | 200 |  262.519652ms |      172.20.1.1 | POST     "/api/embeddings"
GGML_ASSERT: /go/src/github.com/ollama/ollama/llm/llama.cpp/llama.cpp:11010: seq_id < n_tokens && "seq_id cannot be larger than n_tokens with pooling_type == MEAN"
[GIN] 2024/05/20 - 22:17:48 | 200 |  312.790001ms |      172.20.1.1 | POST     "/api/embeddings"
time=2024-05-20T22:17:48.786Z level=INFO source=routes.go:400 msg="embedding generation failed: do embedding request: Post \"http://127.0.0.1:42663/embedding\": EOF"
time=2024-05-20T22:17:48.786Z level=INFO source=routes.go:400 msg="embedding generation failed: do embedding request: Post \"http://127.0.0.1:42663/embedding\": EOF"
[GIN] 2024/05/20 - 22:17:48 | 500 |  260.588554ms |      172.20.1.1 | POST     "/api/embeddings"
[GIN] 2024/05/20 - 22:17:48 | 500 |  253.715301ms |      172.20.1.1 | POST     "/api/embeddings"
time=2024-05-20T22:17:48.789Z level=INFO source=routes.go:400 msg="embedding generation failed: do embedding request: Post \"http://127.0.0.1:42663/embedding\": EOF"
[GIN] 2024/05/20 - 22:17:48 | 500 |  210.774869ms |      172.20.1.1 | POST     "/api/embeddings"
time=2024-05-20T22:17:48.789Z level=INFO source=routes.go:400 msg="embedding generation failed: health resp: Get \"http://127.0.0.1:42663/health\": dial tcp 127.0.0.1:42663: connect: connection refused"
[GIN] 2024/05/20 - 22:17:48 | 500 |   390.45269ms |      172.20.1.1 | POST     "/api/embeddings"
time=2024-05-20T22:17:53.914Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.124544061
time=2024-05-20T22:17:54.163Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.374084634
time=2024-05-20T22:17:54.185Z level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=7 memory.available="23.3 GiB" memory.required.full="508.7 MiB" memory.required.partial="508.7 MiB" memory.required.kv="1.9 MiB" memory.weights.total="42.7 MiB" memory.weights.repeating="20.4 MiB" memory.weights.nonrepeating="22.4 MiB" memory.graph.full="3.8 MiB" memory.graph.partial="3.8 MiB"
time=2024-05-20T22:17:54.185Z level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=7 memory.available="23.3 GiB" memory.required.full="508.7 MiB" memory.required.partial="508.7 MiB" memory.required.kv="1.9 MiB" memory.weights.total="42.7 MiB" memory.weights.repeating="20.4 MiB" memory.weights.nonrepeating="22.4 MiB" memory.graph.full="3.8 MiB" memory.graph.partial="3.8 MiB"
time=2024-05-20T22:17:54.185Z level=INFO source=server.go:320 msg="starting llama server" cmd="/tmp/ollama3420298939/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-797b70c4edf85907fe0a49eb85811256f65fa0f7bf52166b147fd16be2be4662 --ctx-size 2560 --batch-size 512 --embedding --log-disable --n-gpu-layers 7 --parallel 10 --port 45467"
time=2024-05-20T22:17:54.185Z level=INFO source=sched.go:338 msg="loaded runners" count=1
time=2024-05-20T22:17:54.185Z level=INFO source=server.go:504 msg="waiting for llama runner to start responding"
time=2024-05-20T22:17:54.186Z level=INFO source=server.go:540 msg="waiting for server to become available" status="llm server error"
INFO [main] build info | build=1 commit="952d03d" tid="124954675605504" timestamp=1716243474
INFO [main] system info | n_threads=8 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="124954675605504" timestamp=1716243474 total_threads=32
INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="31" port="45467" tid="124954675605504" timestamp=1716243474
llama_model_loader: loaded meta data with 23 key-value pairs and 101 tensors from /root/.ollama/models/blobs/sha256-797b70c4edf85907fe0a49eb85811256f65fa0f7bf52166b147fd16be2be4662 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = bert
llama_model_loader: - kv   1:                               general.name str              = all-MiniLM-L6-v2
llama_model_loader: - kv   2:                           bert.block_count u32              = 6
llama_model_loader: - kv   3:                        bert.context_length u32              = 512
llama_model_loader: - kv   4:                      bert.embedding_length u32              = 384
llama_model_loader: - kv   5:                   bert.feed_forward_length u32              = 1536
llama_model_loader: - kv   6:                  bert.attention.head_count u32              = 12
llama_model_loader: - kv   7:          bert.attention.layer_norm_epsilon f32              = 0.000000
llama_model_loader: - kv   8:                          general.file_type u32              = 1
llama_model_loader: - kv   9:                      bert.attention.causal bool             = false
llama_model_loader: - kv  10:                          bert.pooling_type u32              = 1
llama_model_loader: - kv  11:            tokenizer.ggml.token_type_count u32              = 2
llama_model_loader: - kv  12:                tokenizer.ggml.bos_token_id u32              = 101
llama_model_loader: - kv  13:                tokenizer.ggml.eos_token_id u32              = 102
llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = bert
llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,30522]   = ["[PAD]", "[unused0]", "[unused1]", "...
llama_model_loader: - kv  16:                      tokenizer.ggml.scores arr[f32,30522]   = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,30522]   = [3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 100
llama_model_loader: - kv  19:          tokenizer.ggml.seperator_token_id u32              = 102
llama_model_loader: - kv  20:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  21:                tokenizer.ggml.cls_token_id u32              = 101
llama_model_loader: - kv  22:               tokenizer.ggml.mask_token_id u32              = 103
llama_model_loader: - type  f32:   63 tensors
llama_model_loader: - type  f16:   38 tensors
llm_load_vocab: mismatch in special tokens definition ( 7104/30522 vs 5/30522 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = bert
llm_load_print_meta: vocab type       = WPM
llm_load_print_meta: n_vocab          = 30522
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 512
llm_load_print_meta: n_embd           = 384
llm_load_print_meta: n_head           = 12
llm_load_print_meta: n_head_kv        = 12
llm_load_print_meta: n_layer          = 6
llm_load_print_meta: n_rot            = 32
llm_load_print_meta: n_embd_head_k    = 32
llm_load_print_meta: n_embd_head_v    = 32
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 384
llm_load_print_meta: n_embd_v_gqa     = 384
llm_load_print_meta: f_norm_eps       = 1.0e-12
llm_load_print_meta: f_norm_rms_eps   = 0.0e+00
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 1536
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 0
llm_load_print_meta: pooling type     = 1
llm_load_print_meta: rope type        = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 512
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 22M
llm_load_print_meta: model ftype      = F16
llm_load_print_meta: model params     = 22.57 M
llm_load_print_meta: model size       = 43.10 MiB (16.02 BPW) 
llm_load_print_meta: general.name     = all-MiniLM-L6-v2
llm_load_print_meta: BOS token        = 101 '[CLS]'
llm_load_print_meta: EOS token        = 102 '[SEP]'
llm_load_print_meta: UNK token        = 100 '[UNK]'
llm_load_print_meta: SEP token        = 102 '[SEP]'
llm_load_print_meta: PAD token        = 0 '[PAD]'
llm_load_print_meta: CLS token        = 101 '[CLS]'
llm_load_print_meta: MASK token       = 103 '[MASK]'
llm_load_print_meta: LF token         = 0 '[PAD]'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   yes
ggml_cuda_init: CUDA_USE_TENSOR_CORES: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
llm_load_tensors: ggml ctx size =    0.09 MiB
llm_load_tensors: offloading 6 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 7/7 layers to GPU
llm_load_tensors:        CPU buffer size =    22.73 MiB
llm_load_tensors:      CUDA0 buffer size =    20.37 MiB
...............................
llama_new_context_with_model: n_ctx      = 2560
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      CUDA0 KV buffer size =    22.50 MiB
llama_new_context_with_model: KV self size  =   22.50 MiB, K (f16):   11.25 MiB, V (f16):   11.25 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.00 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =    17.00 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =     3.50 MiB
llama_new_context_with_model: graph nodes  = 221
llama_new_context_with_model: graph splits = 2
time=2024-05-20T22:17:54.413Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.624121318
time=2024-05-20T22:17:54.437Z level=INFO source=server.go:540 msg="waiting for server to become available" status="llm server loading model"
INFO [main] model loaded | tid="124954675605504" timestamp=1716243474
time=2024-05-20T22:17:54.688Z level=INFO source=server.go:545 msg="llama runner started in 0.50 seconds"
[GIN] 2024/05/20 - 22:17:54 | 200 |  6.186218241s |      172.20.1.1 | POST     "/api/embeddings"
<!-- gh-comment-id:2121311971 --> @iganev commented on GitHub (May 20, 2024): Tried bumping the queue to 1024 and the parallel num to 10. Watching the log live (-fn100) I see this popping up every once in a while: ``` [GIN] 2024/05/20 - 22:17:48 | 200 | 262.519652ms | 172.20.1.1 | POST "/api/embeddings" GGML_ASSERT: /go/src/github.com/ollama/ollama/llm/llama.cpp/llama.cpp:11010: seq_id < n_tokens && "seq_id cannot be larger than n_tokens with pooling_type == MEAN" [GIN] 2024/05/20 - 22:17:48 | 200 | 312.790001ms | 172.20.1.1 | POST "/api/embeddings" time=2024-05-20T22:17:48.786Z level=INFO source=routes.go:400 msg="embedding generation failed: do embedding request: Post \"http://127.0.0.1:42663/embedding\": EOF" time=2024-05-20T22:17:48.786Z level=INFO source=routes.go:400 msg="embedding generation failed: do embedding request: Post \"http://127.0.0.1:42663/embedding\": EOF" [GIN] 2024/05/20 - 22:17:48 | 500 | 260.588554ms | 172.20.1.1 | POST "/api/embeddings" [GIN] 2024/05/20 - 22:17:48 | 500 | 253.715301ms | 172.20.1.1 | POST "/api/embeddings" time=2024-05-20T22:17:48.789Z level=INFO source=routes.go:400 msg="embedding generation failed: do embedding request: Post \"http://127.0.0.1:42663/embedding\": EOF" [GIN] 2024/05/20 - 22:17:48 | 500 | 210.774869ms | 172.20.1.1 | POST "/api/embeddings" time=2024-05-20T22:17:48.789Z level=INFO source=routes.go:400 msg="embedding generation failed: health resp: Get \"http://127.0.0.1:42663/health\": dial tcp 127.0.0.1:42663: connect: connection refused" [GIN] 2024/05/20 - 22:17:48 | 500 | 390.45269ms | 172.20.1.1 | POST "/api/embeddings" time=2024-05-20T22:17:53.914Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.124544061 time=2024-05-20T22:17:54.163Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.374084634 time=2024-05-20T22:17:54.185Z level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=7 memory.available="23.3 GiB" memory.required.full="508.7 MiB" memory.required.partial="508.7 MiB" memory.required.kv="1.9 MiB" memory.weights.total="42.7 MiB" memory.weights.repeating="20.4 MiB" memory.weights.nonrepeating="22.4 MiB" memory.graph.full="3.8 MiB" memory.graph.partial="3.8 MiB" time=2024-05-20T22:17:54.185Z level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=7 memory.available="23.3 GiB" memory.required.full="508.7 MiB" memory.required.partial="508.7 MiB" memory.required.kv="1.9 MiB" memory.weights.total="42.7 MiB" memory.weights.repeating="20.4 MiB" memory.weights.nonrepeating="22.4 MiB" memory.graph.full="3.8 MiB" memory.graph.partial="3.8 MiB" time=2024-05-20T22:17:54.185Z level=INFO source=server.go:320 msg="starting llama server" cmd="/tmp/ollama3420298939/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-797b70c4edf85907fe0a49eb85811256f65fa0f7bf52166b147fd16be2be4662 --ctx-size 2560 --batch-size 512 --embedding --log-disable --n-gpu-layers 7 --parallel 10 --port 45467" time=2024-05-20T22:17:54.185Z level=INFO source=sched.go:338 msg="loaded runners" count=1 time=2024-05-20T22:17:54.185Z level=INFO source=server.go:504 msg="waiting for llama runner to start responding" time=2024-05-20T22:17:54.186Z level=INFO source=server.go:540 msg="waiting for server to become available" status="llm server error" INFO [main] build info | build=1 commit="952d03d" tid="124954675605504" timestamp=1716243474 INFO [main] system info | n_threads=8 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="124954675605504" timestamp=1716243474 total_threads=32 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="31" port="45467" tid="124954675605504" timestamp=1716243474 llama_model_loader: loaded meta data with 23 key-value pairs and 101 tensors from /root/.ollama/models/blobs/sha256-797b70c4edf85907fe0a49eb85811256f65fa0f7bf52166b147fd16be2be4662 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = bert llama_model_loader: - kv 1: general.name str = all-MiniLM-L6-v2 llama_model_loader: - kv 2: bert.block_count u32 = 6 llama_model_loader: - kv 3: bert.context_length u32 = 512 llama_model_loader: - kv 4: bert.embedding_length u32 = 384 llama_model_loader: - kv 5: bert.feed_forward_length u32 = 1536 llama_model_loader: - kv 6: bert.attention.head_count u32 = 12 llama_model_loader: - kv 7: bert.attention.layer_norm_epsilon f32 = 0.000000 llama_model_loader: - kv 8: general.file_type u32 = 1 llama_model_loader: - kv 9: bert.attention.causal bool = false llama_model_loader: - kv 10: bert.pooling_type u32 = 1 llama_model_loader: - kv 11: tokenizer.ggml.token_type_count u32 = 2 llama_model_loader: - kv 12: tokenizer.ggml.bos_token_id u32 = 101 llama_model_loader: - kv 13: tokenizer.ggml.eos_token_id u32 = 102 llama_model_loader: - kv 14: tokenizer.ggml.model str = bert llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,30522] = ["[PAD]", "[unused0]", "[unused1]", "... llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,30522] = [-1000.000000, -1000.000000, -1000.00... llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,30522] = [3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 18: tokenizer.ggml.unknown_token_id u32 = 100 llama_model_loader: - kv 19: tokenizer.ggml.seperator_token_id u32 = 102 llama_model_loader: - kv 20: tokenizer.ggml.padding_token_id u32 = 0 llama_model_loader: - kv 21: tokenizer.ggml.cls_token_id u32 = 101 llama_model_loader: - kv 22: tokenizer.ggml.mask_token_id u32 = 103 llama_model_loader: - type f32: 63 tensors llama_model_loader: - type f16: 38 tensors llm_load_vocab: mismatch in special tokens definition ( 7104/30522 vs 5/30522 ). llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = bert llm_load_print_meta: vocab type = WPM llm_load_print_meta: n_vocab = 30522 llm_load_print_meta: n_merges = 0 llm_load_print_meta: n_ctx_train = 512 llm_load_print_meta: n_embd = 384 llm_load_print_meta: n_head = 12 llm_load_print_meta: n_head_kv = 12 llm_load_print_meta: n_layer = 6 llm_load_print_meta: n_rot = 32 llm_load_print_meta: n_embd_head_k = 32 llm_load_print_meta: n_embd_head_v = 32 llm_load_print_meta: n_gqa = 1 llm_load_print_meta: n_embd_k_gqa = 384 llm_load_print_meta: n_embd_v_gqa = 384 llm_load_print_meta: f_norm_eps = 1.0e-12 llm_load_print_meta: f_norm_rms_eps = 0.0e+00 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 1536 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 0 llm_load_print_meta: pooling type = 1 llm_load_print_meta: rope type = 2 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx = 512 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = 22M llm_load_print_meta: model ftype = F16 llm_load_print_meta: model params = 22.57 M llm_load_print_meta: model size = 43.10 MiB (16.02 BPW) llm_load_print_meta: general.name = all-MiniLM-L6-v2 llm_load_print_meta: BOS token = 101 '[CLS]' llm_load_print_meta: EOS token = 102 '[SEP]' llm_load_print_meta: UNK token = 100 '[UNK]' llm_load_print_meta: SEP token = 102 '[SEP]' llm_load_print_meta: PAD token = 0 '[PAD]' llm_load_print_meta: CLS token = 101 '[CLS]' llm_load_print_meta: MASK token = 103 '[MASK]' llm_load_print_meta: LF token = 0 '[PAD]' ggml_cuda_init: GGML_CUDA_FORCE_MMQ: yes ggml_cuda_init: CUDA_USE_TENSOR_CORES: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes llm_load_tensors: ggml ctx size = 0.09 MiB llm_load_tensors: offloading 6 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 7/7 layers to GPU llm_load_tensors: CPU buffer size = 22.73 MiB llm_load_tensors: CUDA0 buffer size = 20.37 MiB ............................... llama_new_context_with_model: n_ctx = 2560 llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: CUDA0 KV buffer size = 22.50 MiB llama_new_context_with_model: KV self size = 22.50 MiB, K (f16): 11.25 MiB, V (f16): 11.25 MiB llama_new_context_with_model: CPU output buffer size = 0.00 MiB llama_new_context_with_model: CUDA0 compute buffer size = 17.00 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 3.50 MiB llama_new_context_with_model: graph nodes = 221 llama_new_context_with_model: graph splits = 2 time=2024-05-20T22:17:54.413Z level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.624121318 time=2024-05-20T22:17:54.437Z level=INFO source=server.go:540 msg="waiting for server to become available" status="llm server loading model" INFO [main] model loaded | tid="124954675605504" timestamp=1716243474 time=2024-05-20T22:17:54.688Z level=INFO source=server.go:545 msg="llama runner started in 0.50 seconds" [GIN] 2024/05/20 - 22:17:54 | 200 | 6.186218241s | 172.20.1.1 | POST "/api/embeddings" ```
GiteaMirror commented 2026-04-22 07:01:37 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (May 20, 2024):

Sorry this happened – may I ask how you're sending the embedding requests? Hoping to reproduce on my side ASAP!

<!-- gh-comment-id:2121325570 --> @jmorganca commented on GitHub (May 20, 2024): Sorry this happened – may I ask how you're sending the embedding requests? Hoping to reproduce on my side ASAP!
GiteaMirror commented 2026-04-22 07:01:38 -05:00
Author
Owner
Copy Link

@iganev commented on GitHub (May 20, 2024):

I am using ollama-rs:

let ollama_host = env::var("OLLAMA_HOST").expect("Ollama host not configured.");
let ollama = Ollama::new(format!("http://{}", ollama_host), 11434);

let res = ollama
        .generate_embeddings("all-minilm:l6-v2".to_string(), chunk.clone(), None)
        .await;

Nothing out of the ordinary. Happens every 5-10 seconds now. A bunch of requests pass without issue and then it glitches and resumes few moments later. I will check if the payload itself causes a reproducible error sequence or not. Just a sec.

<!-- gh-comment-id:2121335243 --> @iganev commented on GitHub (May 20, 2024): I am using [ollama-rs](https://crates.io/crates/ollama-rs): ```rust let ollama_host = env::var("OLLAMA_HOST").expect("Ollama host not configured."); let ollama = Ollama::new(format!("http://{}", ollama_host), 11434); let res = ollama .generate_embeddings("all-minilm:l6-v2".to_string(), chunk.clone(), None) .await; ``` Nothing out of the ordinary. Happens every 5-10 seconds now. A bunch of requests pass without issue and then it glitches and resumes few moments later. I will check if the payload itself causes a reproducible error sequence or not. Just a sec.
GiteaMirror commented 2026-04-22 07:01:39 -05:00
Author
Owner
Copy Link

@iganev commented on GitHub (May 20, 2024):

It doesn't seem to be payload-dependent. I tested the payloads manually using curl with the last successful embedding before an error and also with the first occurring error. Unsuccessful requests turn out as successful when retried few seconds later when the service reloads.

Might have something to do with the fact that I run 10 workers in parallel blasting requests at the same time.

My OLLAMA_NUM_PARALLEL is 10 at the moment and the queue at 1024. In theory I shouldn't be landing the queue at all because the number of workers matches the number of parallel requests expected by ollama.

Do you need me to test something specific and report back the result?

<!-- gh-comment-id:2121349737 --> @iganev commented on GitHub (May 20, 2024): It doesn't seem to be payload-dependent. I tested the payloads manually using curl with the last successful embedding before an error and also with the first occurring error. Unsuccessful requests turn out as successful when retried few seconds later when the service reloads. Might have something to do with the fact that I run 10 workers in parallel blasting requests at the same time. My `OLLAMA_NUM_PARALLEL` is 10 at the moment and the queue at 1024. In theory I shouldn't be landing the queue at all because the number of workers matches the number of parallel requests expected by ollama. Do you need me to test something specific and report back the result?
GiteaMirror commented 2026-04-22 07:01:40 -05:00
Author
Owner
Copy Link

@iganev commented on GitHub (May 20, 2024):

Ok, here's a reproducible example. I asked llama3 for exactly 256 words placeholder text sounding like a news article with a "{}" placeholder to be replaced by a number. Made a mini-version of what I am actually doing with Ollama and let it blast 1 million requests, 8 at a time in parallel. First error occurs sometime in the first few dozens of requests. One time was 63rd, then 41st request.

Here's the exact code I am using:

use std::env;

use anyhow::Result;
use clap::Parser;
use futures::{stream::FuturesUnordered, StreamExt};
use ollama_rs::Ollama;

pub const CONCURRENT_WORKERS: usize = 8;

#[derive(Parser, Debug)]
#[command(version, about, long_about = None)]
struct Args {
    /// Number of threads
    #[arg(short = 't', long)]
    threads: Option<usize>,
}

#[tokio::main]
async fn main() -> Result<()> {
    dotenvy::dotenv()?;

    let args = Args::parse();

    let tasks_limit = if let Some(threads) = args.threads {
        threads
    } else {
        CONCURRENT_WORKERS
    };

    let ollama_host = env::var("OLLAMA_HOST").expect("Ollama host not configured.");
    let ollama = Ollama::new(format!("http://{}", ollama_host), 11434);

    println!("Working...");
    let mut futs = FuturesUnordered::new();
    for i in 1..1_000_000 {
        let chunk = format!(
            r#"Breaking News: Economy Sees Slight Uptick in {} Quarter
        According to recent reports, the economy has seen a slight increase in growth over the past few months. Experts attribute this upswing to a combination of factors, including increased consumer spending and a boost in small business confidence.
        The news comes as a welcome relief after a tumultuous year that saw market fluctuations and economic uncertainty. "This is a promising sign for the economy," said Dr. Jane Smith, a leading economist at Harvard University. "We're seeing a renewed sense of optimism among consumers and businesses alike."
        In related news, the housing market has also seen a surge in recent months. With interest rates at an all-time low, many Americans are taking advantage of the opportunity to purchase or refinance their homes.
        However, not everyone is convinced that this upswing will last. "We need to be cautious," said Senator John Doe, a leading voice on economic policy. "While this news is certainly welcome, we can't ignore the underlying issues that still plague our economy."
        As the year draws to a close, economists and policymakers alike will be watching closely to see if this trend continues. Will it be enough to propel us towards a full recovery? Only time will tell.
        In other news, the {} annual conference on economic development is set to take place next month in Washington D.C. The event will bring together leading experts from around the world to discuss the latest trends and strategies for promoting economic growth."#,
            i, i
        );

        let fut = ollama_worker(chunk, &ollama);

        futs.push(fut);

        if futs.len() == tasks_limit {
            let _ = futs.next().await;
        }
    }

    // wait for the rest
    while let Some(()) = futs.next().await {}

    Ok(())
}

pub async fn ollama_worker(chunk: String, ollama: &Ollama) {
    let res = ollama
        .generate_embeddings("all-minilm:l6-v2".to_string(), chunk.clone(), None)
        .await;

    match res {
        Ok(_embeddings) => {
            //println!("Generated embeddings for: {}", &chunk);
        }
        Err(err) => {
            eprintln!(
                "Error fetching embeddings from Ollama: {}\nChunk: {}",
                err, &chunk
            );
        }
    }
}

Cargo.toml:

[package]
name = "ollama-test"
version = "0.1.0"
edition = "2021"

[dependencies]
anyhow = "1"
tokio = { version = "1", features = ["full", "rt-multi-thread"] }
futures = "0"
clap = { version = "4", features = ["derive"] }
dotenvy = "0"
ollama-rs = { version = "0", features = ["rustls", "tokio", "tokio-stream"] }

.env file:

OLLAMA_HOST="localhost"

The Ollama docker-compose.yml:

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    restart: always
    privileged: true
    ports:
      - '11434:11434'
    volumes:
      - ollama-data:/root/.ollama
      - /dev:/dev
    environment:
      - OLLAMA_KEEP_ALIVE=-1
      - OLLAMA_MAX_QUEUE=1024
      - OLLAMA_NUM_PARALLEL=20
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  ollama-ui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: ollama-ui
    restart: always
    depends_on:
      - ollama
    ports:
      - '8080:8080'
    volumes:
      - ollama-ui-data:/app/backend/data
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
volumes:
  ollama-data:
  ollama-ui-data:
<!-- gh-comment-id:2121385941 --> @iganev commented on GitHub (May 20, 2024): Ok, here's a reproducible example. I asked llama3 for exactly 256 words placeholder text sounding like a news article with a "{}" placeholder to be replaced by a number. Made a mini-version of what I am actually doing with Ollama and let it blast 1 million requests, 8 at a time in parallel. First error occurs sometime in the first few dozens of requests. One time was 63rd, then 41st request. Here's the exact code I am using: ```rust use std::env; use anyhow::Result; use clap::Parser; use futures::{stream::FuturesUnordered, StreamExt}; use ollama_rs::Ollama; pub const CONCURRENT_WORKERS: usize = 8; #[derive(Parser, Debug)] #[command(version, about, long_about = None)] struct Args { /// Number of threads #[arg(short = 't', long)] threads: Option<usize>, } #[tokio::main] async fn main() -> Result<()> { dotenvy::dotenv()?; let args = Args::parse(); let tasks_limit = if let Some(threads) = args.threads { threads } else { CONCURRENT_WORKERS }; let ollama_host = env::var("OLLAMA_HOST").expect("Ollama host not configured."); let ollama = Ollama::new(format!("http://{}", ollama_host), 11434); println!("Working..."); let mut futs = FuturesUnordered::new(); for i in 1..1_000_000 { let chunk = format!( r#"Breaking News: Economy Sees Slight Uptick in {} Quarter According to recent reports, the economy has seen a slight increase in growth over the past few months. Experts attribute this upswing to a combination of factors, including increased consumer spending and a boost in small business confidence. The news comes as a welcome relief after a tumultuous year that saw market fluctuations and economic uncertainty. "This is a promising sign for the economy," said Dr. Jane Smith, a leading economist at Harvard University. "We're seeing a renewed sense of optimism among consumers and businesses alike." In related news, the housing market has also seen a surge in recent months. With interest rates at an all-time low, many Americans are taking advantage of the opportunity to purchase or refinance their homes. However, not everyone is convinced that this upswing will last. "We need to be cautious," said Senator John Doe, a leading voice on economic policy. "While this news is certainly welcome, we can't ignore the underlying issues that still plague our economy." As the year draws to a close, economists and policymakers alike will be watching closely to see if this trend continues. Will it be enough to propel us towards a full recovery? Only time will tell. In other news, the {} annual conference on economic development is set to take place next month in Washington D.C. The event will bring together leading experts from around the world to discuss the latest trends and strategies for promoting economic growth."#, i, i ); let fut = ollama_worker(chunk, &ollama); futs.push(fut); if futs.len() == tasks_limit { let _ = futs.next().await; } } // wait for the rest while let Some(()) = futs.next().await {} Ok(()) } pub async fn ollama_worker(chunk: String, ollama: &Ollama) { let res = ollama .generate_embeddings("all-minilm:l6-v2".to_string(), chunk.clone(), None) .await; match res { Ok(_embeddings) => { //println!("Generated embeddings for: {}", &chunk); } Err(err) => { eprintln!( "Error fetching embeddings from Ollama: {}\nChunk: {}", err, &chunk ); } } } ``` Cargo.toml: ```toml [package] name = "ollama-test" version = "0.1.0" edition = "2021" [dependencies] anyhow = "1" tokio = { version = "1", features = ["full", "rt-multi-thread"] } futures = "0" clap = { version = "4", features = ["derive"] } dotenvy = "0" ollama-rs = { version = "0", features = ["rustls", "tokio", "tokio-stream"] } ``` .env file: ``` OLLAMA_HOST="localhost" ``` The Ollama docker-compose.yml: ``` services: ollama: image: ollama/ollama container_name: ollama restart: always privileged: true ports: - '11434:11434' volumes: - ollama-data:/root/.ollama - /dev:/dev environment: - OLLAMA_KEEP_ALIVE=-1 - OLLAMA_MAX_QUEUE=1024 - OLLAMA_NUM_PARALLEL=20 deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] ollama-ui: image: ghcr.io/open-webui/open-webui:main container_name: ollama-ui restart: always depends_on: - ollama ports: - '8080:8080' volumes: - ollama-ui-data:/app/backend/data environment: - OLLAMA_BASE_URL=http://ollama:11434 volumes: ollama-data: ollama-ui-data: ```
GiteaMirror commented 2026-04-22 07:01:41 -05:00
Author
Owner
Copy Link

@iganev commented on GitHub (May 20, 2024):

Here's a debug-enabled log from a test run of the above setup:
ollama_log.txt

Took around 100 requests to encounter an error, so the log isn't all too crowded.

Let me know if I can be of any more help.

<!-- gh-comment-id:2121406260 --> @iganev commented on GitHub (May 20, 2024): Here's a debug-enabled log from a test run of the above setup: [ollama_log.txt](https://github.com/ollama/ollama/files/15382623/ollama_log.txt) Took around 100 requests to encounter an error, so the log isn't all too crowded. Let me know if I can be of any more help.
GiteaMirror commented 2026-04-22 07:01:42 -05:00
Author
Owner
Copy Link

@lyriccoder commented on GitHub (May 21, 2024):

@jmorganca I am encountering the same issue, though it appears to be without Docker. I installed Ollama using curl, and the service starts automatically. I am unaware of Docker running within this service, but I experience the same problem: after 20-30 minutes, the service hangs. I am using Ubuntu 23.04 with two 4090 GPUs. Consequently, I have to restart the service every 30 minutes.

Here is how I send requests:

def send_req_llama(prompt, ip, port):
    fail = 0
    while (fail < 10):
        try:
            resp = requests.post(url=f'http://{ip}:{port}/api/generate', data=json.dumps({"model": "llama3", "prompt": prompt, 'stream': False}))
            status_code = resp.status_code
            res = resp.json()['response']
            break
        except Exception as e:
            fail += 1
            time.sleep(5)
            res = resp.content
            print(resp, status_code)
    return res, status_code

Could you please help?

<!-- gh-comment-id:2122455172 --> @lyriccoder commented on GitHub (May 21, 2024): @jmorganca I am encountering the same issue, though it appears to be without Docker. I installed Ollama using curl, and the service starts automatically. I am unaware of Docker running within this service, but I experience the same problem: after 20-30 minutes, the service hangs. I am using Ubuntu 23.04 with two 4090 GPUs. Consequently, I have to restart the service every 30 minutes. Here is how I send requests: ``` def send_req_llama(prompt, ip, port): fail = 0 while (fail < 10): try: resp = requests.post(url=f'http://{ip}:{port}/api/generate', data=json.dumps({"model": "llama3", "prompt": prompt, 'stream': False})) status_code = resp.status_code res = resp.json()['response'] break except Exception as e: fail += 1 time.sleep(5) res = resp.content print(resp, status_code) return res, status_code ``` Could you please help?
GiteaMirror commented 2026-04-22 07:01:42 -05:00
Author
Owner
Copy Link

@Levichev commented on GitHub (May 21, 2024):

@lyriccoder same issue

<!-- gh-comment-id:2122462847 --> @Levichev commented on GitHub (May 21, 2024): @lyriccoder same issue
GiteaMirror commented 2026-04-22 07:01:43 -05:00
Author
Owner
Copy Link

@ChenhuaYang commented on GitHub (May 21, 2024):

we have same issue

<!-- gh-comment-id:2122503276 --> @ChenhuaYang commented on GitHub (May 21, 2024): we have same issue
GiteaMirror commented 2026-04-22 07:01:45 -05:00
Author
Owner
Copy Link

@Adefful commented on GitHub (May 21, 2024):

I thought I was the only one with this problem. I have an RTX 4090. After 20 minutes, it freezes. I have to restart it. Six months ago, I had the same problem, and it hasn't gone away.

<!-- gh-comment-id:2122698410 --> @Adefful commented on GitHub (May 21, 2024): I thought I was the only one with this problem. I have an RTX 4090. After 20 minutes, it freezes. I have to restart it. Six months ago, I had the same problem, and it hasn't gone away.
GiteaMirror commented 2026-04-22 07:01:47 -05:00
Author
Owner
Copy Link

@lucasbarrosomk6 commented on GitHub (May 21, 2024):

I have an RTX 4050, running llama3 and getting this issue under high load.
[GIN] 2024/05/21 - 14:36:23 | 500 | 2.0974796s | 127.0.0.1 | POST "/api/generate" time=2024-05-21T14:36:23.400-07:00 level=ERROR source=server.go:624 msg="Failed to acquire semaphore" error="context canceled" [GIN] 2024/05/21 - 14:36:23 | 500 | 3.2332771s | 127.0.0.1 | POST "/api/generate" time=2024-05-21T14:36:23.400-07:00 level=ERROR source=server.go:624 msg="Failed to acquire semaphore" error="context canceled" [GIN] 2024/05/21 - 14:36:23 | 500 | 805.2666ms | 127.0.0.1 | POST "/api/generate" time=2024-05-21T14:36:23.401-07:00 level=ERROR source=server.go:624 msg="Failed to acquire semaphore" error="context canceled" [GIN] 2024/05/21 - 14:36:23 | 500 | 5.8394386s | 127.0.0.1 | POST "/api/generate" [GIN] 2024/05/21 - 14:36:23 | 500 | 4.7348779s | 127.0.0.1 | POST "/api/generate"

<!-- gh-comment-id:2123487876 --> @lucasbarrosomk6 commented on GitHub (May 21, 2024): I have an RTX 4050, running llama3 and getting this issue under high load. `[GIN] 2024/05/21 - 14:36:23 | 500 | 2.0974796s | 127.0.0.1 | POST "/api/generate" time=2024-05-21T14:36:23.400-07:00 level=ERROR source=server.go:624 msg="Failed to acquire semaphore" error="context canceled" [GIN] 2024/05/21 - 14:36:23 | 500 | 3.2332771s | 127.0.0.1 | POST "/api/generate" time=2024-05-21T14:36:23.400-07:00 level=ERROR source=server.go:624 msg="Failed to acquire semaphore" error="context canceled" [GIN] 2024/05/21 - 14:36:23 | 500 | 805.2666ms | 127.0.0.1 | POST "/api/generate" time=2024-05-21T14:36:23.401-07:00 level=ERROR source=server.go:624 msg="Failed to acquire semaphore" error="context canceled" [GIN] 2024/05/21 - 14:36:23 | 500 | 5.8394386s | 127.0.0.1 | POST "/api/generate" [GIN] 2024/05/21 - 14:36:23 | 500 | 4.7348779s | 127.0.0.1 | POST "/api/generate"`
GiteaMirror commented 2026-04-22 07:01:48 -05:00
Author
Owner
Copy Link

@iganev commented on GitHub (May 21, 2024):

Got myself a piece of weirdly broken text that crashes ollama every time. The problem I am experiencing is partly because of payloads like that, apparently. It reliably crashes it every time. Not sure how that relates to the rest of the issues.

I got this from poppler when reading a PDF file... there's something seriously messed up with it. Any version of poppler I tried produced the same text and it does indeed make ollama scream.

116                                                                                                                                 

P.S.: Turns out those are coming from PDFs with "CID Fonts" which have "CID Maps" or "CMaps" basically mapping their font character to a regular unicode character, usually by adding a certain byte value, in my case F000. These 100% always crash ollama.

This is some clue to what's going on, but certainly not the full picture, as I can make it crash regularly with "normal" text.

<!-- gh-comment-id:2123563009 --> @iganev commented on GitHub (May 21, 2024): Got myself a piece of weirdly broken text that crashes ollama every time. The problem I am experiencing is partly because of payloads like that, apparently. It reliably crashes it every time. Not sure how that relates to the rest of the issues. I got this from poppler when reading a PDF file... there's something seriously messed up with it. Any version of poppler I tried produced the same text and it does indeed make ollama scream. ``` 116                                                                                                                                  ``` P.S.: Turns out those are coming from PDFs with "CID Fonts" which have "CID Maps" or "CMaps" basically mapping their font character to a regular unicode character, usually by adding a certain byte value, in my case F000. These 100% always crash ollama. This is some clue to what's going on, but certainly not the full picture, as I can make it crash regularly with "normal" text.
GiteaMirror commented 2026-04-22 07:01:49 -05:00
Author
Owner
Copy Link

@iganev commented on GitHub (May 21, 2024):

Ok, here's a reproducible example. I asked llama3 for exactly 256 words placeholder text sounding like a news article with a "{}" placeholder to be replaced by a number. Made a mini-version of what I am actually doing with Ollama and let it blast 1 million requests, 8 at a time in parallel. First error occurs sometime in the first few dozens of requests. One time was 63rd, then 41st request.

Here's the exact code I am using:

use std::env;

use anyhow::Result;
use clap::Parser;
use futures::{stream::FuturesUnordered, StreamExt};
use ollama_rs::Ollama;

pub const CONCURRENT_WORKERS: usize = 8;

#[derive(Parser, Debug)]
#[command(version, about, long_about = None)]
struct Args {
    /// Number of threads
    #[arg(short = 't', long)]
    threads: Option<usize>,
}

#[tokio::main]
async fn main() -> Result<()> {
    dotenvy::dotenv()?;

    let args = Args::parse();

    let tasks_limit = if let Some(threads) = args.threads {
        threads
    } else {
        CONCURRENT_WORKERS
    };

    let ollama_host = env::var("OLLAMA_HOST").expect("Ollama host not configured.");
    let ollama = Ollama::new(format!("http://{}", ollama_host), 11434);

    println!("Working...");
    let mut futs = FuturesUnordered::new();
    for i in 1..1_000_000 {
        let chunk = format!(
            r#"Breaking News: Economy Sees Slight Uptick in {} Quarter
        According to recent reports, the economy has seen a slight increase in growth over the past few months. Experts attribute this upswing to a combination of factors, including increased consumer spending and a boost in small business confidence.
        The news comes as a welcome relief after a tumultuous year that saw market fluctuations and economic uncertainty. "This is a promising sign for the economy," said Dr. Jane Smith, a leading economist at Harvard University. "We're seeing a renewed sense of optimism among consumers and businesses alike."
        In related news, the housing market has also seen a surge in recent months. With interest rates at an all-time low, many Americans are taking advantage of the opportunity to purchase or refinance their homes.
        However, not everyone is convinced that this upswing will last. "We need to be cautious," said Senator John Doe, a leading voice on economic policy. "While this news is certainly welcome, we can't ignore the underlying issues that still plague our economy."
        As the year draws to a close, economists and policymakers alike will be watching closely to see if this trend continues. Will it be enough to propel us towards a full recovery? Only time will tell.
        In other news, the {} annual conference on economic development is set to take place next month in Washington D.C. The event will bring together leading experts from around the world to discuss the latest trends and strategies for promoting economic growth."#,
            i, i
        );

        let fut = ollama_worker(chunk, &ollama);

        futs.push(fut);

        if futs.len() == tasks_limit {
            let _ = futs.next().await;
        }
    }

    // wait for the rest
    while let Some(()) = futs.next().await {}

    Ok(())
}

pub async fn ollama_worker(chunk: String, ollama: &Ollama) {
    let res = ollama
        .generate_embeddings("all-minilm:l6-v2".to_string(), chunk.clone(), None)
        .await;

    match res {
        Ok(_embeddings) => {
            //println!("Generated embeddings for: {}", &chunk);
        }
        Err(err) => {
            eprintln!(
                "Error fetching embeddings from Ollama: {}\nChunk: {}",
                err, &chunk
            );
        }
    }
}

Cargo.toml:

[package]
name = "ollama-test"
version = "0.1.0"
edition = "2021"

[dependencies]
anyhow = "1"
tokio = { version = "1", features = ["full", "rt-multi-thread"] }
futures = "0"
clap = { version = "4", features = ["derive"] }
dotenvy = "0"
ollama-rs = { version = "0", features = ["rustls", "tokio", "tokio-stream"] }

.env file:

OLLAMA_HOST="localhost"

The Ollama docker-compose.yml:

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    restart: always
    privileged: true
    ports:
      - '11434:11434'
    volumes:
      - ollama-data:/root/.ollama
      - /dev:/dev
    environment:
      - OLLAMA_KEEP_ALIVE=-1
      - OLLAMA_MAX_QUEUE=1024
      - OLLAMA_NUM_PARALLEL=20
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  ollama-ui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: ollama-ui
    restart: always
    depends_on:
      - ollama
    ports:
      - '8080:8080'
    volumes:
      - ollama-ui-data:/app/backend/data
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
volumes:
  ollama-data:
  ollama-ui-data:

So here's how I can reproduce the issue with staggering reliability..

  1. Run the code from this post to keep ollama preoccupied.
  2. Pick one of the following "interesting" payloads as a metaphorical wrench in the metaphorical gears:
 debt 

or

and harvard business review • september 2003

or even

over the brokerage industry. 8
  1. Open another terminal and run the following curl:
curl http://localhost:11434/api/embeddings -d '{
  "model": "all-minilm:l6-v2",
  "prompt": "over the brokerage industry. 8"
}'
  1. Re-send few times if it doesn't work the first time.
  2. ???
  3. PROFIT
GGML_ASSERT: /go/src/github.com/ollama/ollama/llm/llama.cpp/llama.cpp:11010: seq_id < n_tokens && "seq_id cannot be larger than n_tokens with pooling_type == MEAN"

My theory is that whenever something untokenizeable comes in it breaks the llm runner and causes a crash.

This happens only when OLLAMA_NUM_PARALLEL is greater than 1.

N.B.!
My initial post was regarding a different error that I encountered when OLLAMA_NUM_PARALLEL was 1. I am yet to find a reliable way to reproduce that one as well...

P.S.: Further testing seems to reproduce the initially reported issue when OLLAMA_NUM_PARALLEL = 1. Seems like different symptoms of the same thing. When parallelism is enabled it appears to somewhat act like a partial fix...

<!-- gh-comment-id:2123612768 --> @iganev commented on GitHub (May 21, 2024): > Ok, here's a reproducible example. I asked llama3 for exactly 256 words placeholder text sounding like a news article with a "{}" placeholder to be replaced by a number. Made a mini-version of what I am actually doing with Ollama and let it blast 1 million requests, 8 at a time in parallel. First error occurs sometime in the first few dozens of requests. One time was 63rd, then 41st request. > > Here's the exact code I am using: > > ```rust > use std::env; > > use anyhow::Result; > use clap::Parser; > use futures::{stream::FuturesUnordered, StreamExt}; > use ollama_rs::Ollama; > > pub const CONCURRENT_WORKERS: usize = 8; > > #[derive(Parser, Debug)] > #[command(version, about, long_about = None)] > struct Args { > /// Number of threads > #[arg(short = 't', long)] > threads: Option<usize>, > } > > #[tokio::main] > async fn main() -> Result<()> { > dotenvy::dotenv()?; > > let args = Args::parse(); > > let tasks_limit = if let Some(threads) = args.threads { > threads > } else { > CONCURRENT_WORKERS > }; > > let ollama_host = env::var("OLLAMA_HOST").expect("Ollama host not configured."); > let ollama = Ollama::new(format!("http://{}", ollama_host), 11434); > > println!("Working..."); > let mut futs = FuturesUnordered::new(); > for i in 1..1_000_000 { > let chunk = format!( > r#"Breaking News: Economy Sees Slight Uptick in {} Quarter > According to recent reports, the economy has seen a slight increase in growth over the past few months. Experts attribute this upswing to a combination of factors, including increased consumer spending and a boost in small business confidence. > The news comes as a welcome relief after a tumultuous year that saw market fluctuations and economic uncertainty. "This is a promising sign for the economy," said Dr. Jane Smith, a leading economist at Harvard University. "We're seeing a renewed sense of optimism among consumers and businesses alike." > In related news, the housing market has also seen a surge in recent months. With interest rates at an all-time low, many Americans are taking advantage of the opportunity to purchase or refinance their homes. > However, not everyone is convinced that this upswing will last. "We need to be cautious," said Senator John Doe, a leading voice on economic policy. "While this news is certainly welcome, we can't ignore the underlying issues that still plague our economy." > As the year draws to a close, economists and policymakers alike will be watching closely to see if this trend continues. Will it be enough to propel us towards a full recovery? Only time will tell. > In other news, the {} annual conference on economic development is set to take place next month in Washington D.C. The event will bring together leading experts from around the world to discuss the latest trends and strategies for promoting economic growth."#, > i, i > ); > > let fut = ollama_worker(chunk, &ollama); > > futs.push(fut); > > if futs.len() == tasks_limit { > let _ = futs.next().await; > } > } > > // wait for the rest > while let Some(()) = futs.next().await {} > > Ok(()) > } > > pub async fn ollama_worker(chunk: String, ollama: &Ollama) { > let res = ollama > .generate_embeddings("all-minilm:l6-v2".to_string(), chunk.clone(), None) > .await; > > match res { > Ok(_embeddings) => { > //println!("Generated embeddings for: {}", &chunk); > } > Err(err) => { > eprintln!( > "Error fetching embeddings from Ollama: {}\nChunk: {}", > err, &chunk > ); > } > } > } > ``` > > Cargo.toml: > > ```toml > [package] > name = "ollama-test" > version = "0.1.0" > edition = "2021" > > [dependencies] > anyhow = "1" > tokio = { version = "1", features = ["full", "rt-multi-thread"] } > futures = "0" > clap = { version = "4", features = ["derive"] } > dotenvy = "0" > ollama-rs = { version = "0", features = ["rustls", "tokio", "tokio-stream"] } > ``` > > .env file: > > ``` > OLLAMA_HOST="localhost" > ``` > > The Ollama docker-compose.yml: > > ``` > services: > ollama: > image: ollama/ollama > container_name: ollama > restart: always > privileged: true > ports: > - '11434:11434' > volumes: > - ollama-data:/root/.ollama > - /dev:/dev > environment: > - OLLAMA_KEEP_ALIVE=-1 > - OLLAMA_MAX_QUEUE=1024 > - OLLAMA_NUM_PARALLEL=20 > deploy: > resources: > reservations: > devices: > - driver: nvidia > count: all > capabilities: [gpu] > > ollama-ui: > image: ghcr.io/open-webui/open-webui:main > container_name: ollama-ui > restart: always > depends_on: > - ollama > ports: > - '8080:8080' > volumes: > - ollama-ui-data:/app/backend/data > environment: > - OLLAMA_BASE_URL=http://ollama:11434 > volumes: > ollama-data: > ollama-ui-data: > ``` So here's how I can reproduce the issue with staggering reliability.. 1. Run the code from this post to keep ollama preoccupied. 2. Pick one of the following "interesting" payloads as a metaphorical wrench in the metaphorical gears: ```  debt  ``` or ``` and harvard business review • september 2003 ``` or even ``` over the brokerage industry. 8 ``` 3. Open another terminal and run the following curl: ```bash curl http://localhost:11434/api/embeddings -d '{ "model": "all-minilm:l6-v2", "prompt": "over the brokerage industry. 8" }' ``` 4. Re-send few times if it doesn't work the first time. 5. ??? 6. PROFIT ``` GGML_ASSERT: /go/src/github.com/ollama/ollama/llm/llama.cpp/llama.cpp:11010: seq_id < n_tokens && "seq_id cannot be larger than n_tokens with pooling_type == MEAN" ``` My theory is that whenever something untokenizeable comes in it breaks the llm runner and causes a crash. This happens only when `OLLAMA_NUM_PARALLEL` is greater than 1. N.B.! My initial post was regarding a different error that I encountered when `OLLAMA_NUM_PARALLEL` was 1. I am yet to find a reliable way to reproduce that one as well... P.S.: Further testing seems to reproduce the initially reported issue when `OLLAMA_NUM_PARALLEL` = `1`. Seems like different symptoms of the same thing. When parallelism is enabled it appears to somewhat act like a partial fix...
GiteaMirror commented 2026-04-22 07:01:49 -05:00
Author
Owner
Copy Link

@mruniverse8 commented on GitHub (May 22, 2024):

I have the same issue, ollama-model service fails every 20-30 minutes

<!-- gh-comment-id:2124173210 --> @mruniverse8 commented on GitHub (May 22, 2024): I have the same issue, ollama-model service fails every 20-30 minutes
GiteaMirror commented 2026-04-22 07:01:49 -05:00
Author
Owner
Copy Link

@VecherVhatuX commented on GitHub (May 22, 2024):

Yo, bro, thanks for the issue. Got a serious beef with that Ollama service also. Tried to get some fancy script running to make the model spit out smart stuff, but it’s just sittin’ there like a lazy bum. Can’t figure out what’s trippin’ it up, and now my science experiments are in the toilet. Can you fix this mess? By the way, your service is the bomb, much love!

<!-- gh-comment-id:2124199077 --> @VecherVhatuX commented on GitHub (May 22, 2024): Yo, bro, thanks for the issue. Got a serious beef with that Ollama service also. Tried to get some fancy script running to make the model spit out smart stuff, but it’s just sittin’ there like a lazy bum. Can’t figure out what’s trippin’ it up, and now my science experiments are in the toilet. Can you fix this mess? By the way, your service is the bomb, much love!
GiteaMirror commented 2026-04-22 07:01:50 -05:00
Author
Owner
Copy Link

@ciekawy commented on GitHub (May 22, 2024):

I am getting same error when using just llama.cpp server --embeddings with -np > 1 (number of slots for process requests) but only with some models

<!-- gh-comment-id:2125118939 --> @ciekawy commented on GitHub (May 22, 2024): I am getting same error when using just llama.cpp server --embeddings with -np > 1 (number of slots for process requests) but only with some models
GiteaMirror commented 2026-04-22 07:01:51 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Jun 2, 2024):

Hi all, looking into this – sorry you're seeing Ollama hang.

<!-- gh-comment-id:2143982676 --> @jmorganca commented on GitHub (Jun 2, 2024): Hi all, looking into this – sorry you're seeing Ollama hang.
GiteaMirror commented 2026-04-22 07:01:51 -05:00
Author
Owner
Copy Link

@travisgu commented on GitHub (Jun 6, 2024):

Similar issue, Ollama failed to serve embedding every 5 minutes:
image

<!-- gh-comment-id:2151215949 --> @travisgu commented on GitHub (Jun 6, 2024): Similar issue, Ollama failed to serve embedding every 5 minutes: <img width="1252" alt="image" src="https://github.com/ollama/ollama/assets/10804665/b9865f8b-36b4-40aa-8b0d-7cc0bc65eb24">
GiteaMirror commented 2026-04-22 07:01:52 -05:00
Author
Owner
Copy Link

@mfriedman-pr commented on GitHub (Jun 11, 2024):

I also seem to have this issue in 0.1.42. After a number of sequential generate calls, the server will not respond. Subsequent calls will return this error: ollama Response Error: no slots available after 10 retries

<!-- gh-comment-id:2161426834 --> @mfriedman-pr commented on GitHub (Jun 11, 2024): I also seem to have this issue in 0.1.42. After a number of sequential generate calls, the server will not respond. Subsequent calls will return this error: `ollama Response Error: no slots available after 10 retries`
GiteaMirror commented 2026-04-22 07:01:52 -05:00
Author
Owner
Copy Link

@kojinglick-ctec commented on GitHub (Jun 13, 2024):

I've noticed it only happens with OLLAMA_NUM_PARALLEL > 3. Using NVIDIA GeForce RTX 4090 on WSL2.

<!-- gh-comment-id:2166776890 --> @kojinglick-ctec commented on GitHub (Jun 13, 2024): I've noticed it only happens with `OLLAMA_NUM_PARALLEL` > 3. Using NVIDIA GeForce RTX 4090 on WSL2.
GiteaMirror commented 2026-04-22 07:01:53 -05:00
Author
Owner
Copy Link

@advay-pakhale commented on GitHub (Jun 18, 2024):

I've noticed it only happens with OLLAMA_NUM_PARALLEL > 3. Using NVIDIA GeForce RTX 4090 on WSL2.

Haven't tested the exact value of OLLAMA_NUM_PARALLEL that causes inference to fail, but it definitely has something to do with it. Not setting the value fixes things.

<!-- gh-comment-id:2175540671 --> @advay-pakhale commented on GitHub (Jun 18, 2024): > I've noticed it only happens with `OLLAMA_NUM_PARALLEL` > 3. Using NVIDIA GeForce RTX 4090 on WSL2. Haven't tested the exact value of `OLLAMA_NUM_PARALLEL` that causes inference to fail, but it definitely has something to do with it. Not setting the value fixes things.
GiteaMirror commented 2026-04-22 07:01:54 -05:00
Author
Owner
Copy Link

@iganev commented on GitHub (Jun 19, 2024):

I've noticed it only happens with OLLAMA_NUM_PARALLEL > 3. Using NVIDIA GeForce RTX 4090 on WSL2.

Haven't tested the exact value of OLLAMA_NUM_PARALLEL that causes inference to fail, but it definitely has something to do with it. Not setting the value fixes things.

I find this not to be the case. It just errors out in a slightly different and less recoverable way.

<!-- gh-comment-id:2178505069 --> @iganev commented on GitHub (Jun 19, 2024): > > I've noticed it only happens with `OLLAMA_NUM_PARALLEL` > 3. Using NVIDIA GeForce RTX 4090 on WSL2. > > Haven't tested the exact value of `OLLAMA_NUM_PARALLEL` that causes inference to fail, but it definitely has something to do with it. Not setting the value fixes things. I find this not to be the case. It just errors out in a slightly different and less recoverable way.
GiteaMirror commented 2026-04-22 07:01:56 -05:00
Author
Owner
Copy Link

@jmccrosky commented on GitHub (Jun 20, 2024):

I'm also getting this I think. After some unpredictable number of llava calls with an image and prompt, the server will seemingly hang until I drop the connection. Subsequent calls give ollama._types.ResponseError: no slots available after 10 retries.

[GIN] 2024/06/20 - 14:10:37 | 200 | 317.913792ms | 127.0.0.1 | POST "/api/chat"
[GIN] 2024/06/20 - 14:10:40 | 200 | 3.044791542s | 127.0.0.1 | POST "/api/chat"
[GIN] 2024/06/20 - 14:10:41 | 200 | 325.550292ms | 127.0.0.1 | POST "/api/chat"
[GIN] 2024/06/20 - 14:10:45 | 200 | 4.189981541s | 127.0.0.1 | POST "/api/chat"
[GIN] 2024/06/20 - 14:10:46 | 200 | 373.940917ms | 127.0.0.1 | POST "/api/chat"
[GIN] 2024/06/20 - 14:10:52 | 200 | 5.615178041s | 127.0.0.1 | POST "/api/chat"
[GIN] 2024/06/20 - 14:10:52 | 200 | 418.198541ms | 127.0.0.1 | POST "/api/chat"
[GIN] 2024/06/20 - 14:14:57 | 500 | 4m4s | 127.0.0.1 | POST "/api/chat"
[GIN] 2024/06/20 - 14:18:14 | 500 | 76.112916ms | 127.0.0.1 | POST "/api/chat"

<!-- gh-comment-id:2180438839 --> @jmccrosky commented on GitHub (Jun 20, 2024): I'm also getting this I think. After some unpredictable number of llava calls with an image and prompt, the server will seemingly hang until I drop the connection. Subsequent calls give `ollama._types.ResponseError: no slots available after 10 retries`. [GIN] 2024/06/20 - 14:10:37 | 200 | 317.913792ms | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:40 | 200 | 3.044791542s | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:41 | 200 | 325.550292ms | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:45 | 200 | 4.189981541s | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:46 | 200 | 373.940917ms | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:52 | 200 | 5.615178041s | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:52 | 200 | 418.198541ms | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:14:57 | 500 | 4m4s | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:18:14 | 500 | 76.112916ms | 127.0.0.1 | POST "/api/chat"
GiteaMirror commented 2026-04-22 07:01:58 -05:00
Author
Owner
Copy Link

@LoopControl commented on GitHub (Jun 21, 2024):

I'm also getting this no-slots-available error after a 15-20 minutes as well.

I'm running Ollama directly on the Ubuntu host and parallel requests in ollama has been disabled to try to make it more stable but it still happens. After it hangs, it throws 500s when trying to use the API.

Restarting the API server manually every 15ish minutes makes it unusable for long-running jobs.

<!-- gh-comment-id:2182969415 --> @LoopControl commented on GitHub (Jun 21, 2024): I'm also getting this no-slots-available error after a 15-20 minutes as well. I'm running Ollama directly on the Ubuntu host and parallel requests in ollama has been disabled to try to make it more stable but it still happens. After it hangs, it throws 500s when trying to use the API. Restarting the API server manually every 15ish minutes makes it unusable for long-running jobs.
GiteaMirror commented 2026-04-22 07:01:59 -05:00
Author
Owner
Copy Link

@SebastianGode commented on GitHub (Jun 25, 2024):

I've got same issue when tryaing to generate embeddings with multiple threads running on an Nvidia T4.

ollama  | GGML_ASSERT: /go/src/github.com/ollama/ollama/llm/llama.cpp/llama.cpp:12095: seq_id < n_tokens && "seq_id cannot be larger than n_tokens with pooling_type == CLS"
ollama  | [GIN] 2024/06/25 - 14:34:41 | 200 |   967.02334ms |      172.30.0.3 | POST     "/api/embeddings"
ollama  | time=2024-06-25T14:34:41.450Z level=INFO source=routes.go:400 msg="embedding generation failed: do embedding request: Post \"http://127.0.0.1:43813/embedding\": EOF"
ollama  | time=2024-06-25T14:34:41.450Z level=INFO source=routes.go:400 msg="embedding generation failed: health resp: Get \"http://127.0.0.1:43813/health\": EOF"
ollama  | [GIN] 2024/06/25 - 14:34:41 | 500 |  1.233321395s |      172.30.0.3 | POST     "/api/embeddings"
ollama  | [GIN] 2024/06/25 - 14:34:41 | 500 |  1.103395244s |      172.30.0.3 | POST     "/api/embeddings"
ollama  | time=2024-06-25T14:34:46.724Z level=WARN source=sched.go:575 msg="gpu VRAM usage didn't recover within timeout" seconds=5.24813948 model=/root/.ollama/models/blobs/sha256-819c2adf5ce6df2b6bd2ae4ca90d2a69f060afeb438d0c171db57daa02e39c3d
ollama  | time=2024-06-25T14:34:46.975Z level=WARN source=sched.go:575 msg="gpu VRAM usage didn't recover within timeout" seconds=5.498807909 model=/root/.ollama/models/blobs/sha256-819c2adf5ce6df2b6bd2ae4ca90d2a69f060afeb438d0c171db57daa02e39c3d
ollama  | time=2024-06-25T14:34:47.226Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=25 layers.offload=25 layers.split="" memory.available="[14.5 GiB]" memory.required.full="1.2 GiB" memory.required.partial="1.2 GiB" memory.required.kv="30.0 MiB" memory.required.allocations="[1.2 GiB]" memory.weights.total="607.2 MiB" memory.weights.repeating="547.6 MiB" memory.weights.nonrepeating="59.6 MiB" memory.graph.full="80.0 MiB" memory.graph.partial="80.0 MiB"
ollama  | time=2024-06-25T14:34:47.227Z level=INFO source=server.go:359 msg="starting llama server" cmd="/tmp/ollama2786287991/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-819c2adf5ce6df2b6bd2ae4ca90d2a69f060afeb438d0c171db57daa02e39c3d --ctx-size 5120 --batch-size 512 --embedding --log-disable --n-gpu-layers 25 --parallel 10 --port 36791"
ollama  | time=2024-06-25T14:34:47.227Z level=INFO source=sched.go:382 msg="loaded runners" count=1
ollama  | time=2024-06-25T14:34:47.227Z level=INFO source=server.go:547 msg="waiting for llama runner to start responding"
ollama  | time=2024-06-25T14:34:47.227Z level=INFO source=server.go:585 msg="waiting for server to become available" status="llm server error"
ollama  | INFO [main] build info | build=1 commit="7c26775" tid="139639105748992" timestamp=1719326087
ollama  | INFO [main] system info | n_threads=4 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="139639105748992" timestamp=1719326087 total_threads=8
ollama  | INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="12" port="36791" tid="139639105748992" timestamp=1719326087
ollama  | llama_model_loader: loaded meta data with 23 key-value pairs and 389 tensors from /root/.ollama/models/blobs/sha256-819c2adf5ce6df2b6bd2ae4ca90d2a69f060afeb438d0c171db57daa02e39c3d (version GGUF V3 (latest))
ollama  | llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
ollama  | llama_model_loader: - kv   0:                       general.architecture str              = bert
ollama  | llama_model_loader: - kv   1:                               general.name str              = mxbai-embed-large-v1
ollama  | llama_model_loader: - kv   2:                           bert.block_count u32              = 24
ollama  | llama_model_loader: - kv   3:                        bert.context_length u32              = 512
ollama  | llama_model_loader: - kv   4:                      bert.embedding_length u32              = 1024
ollama  | llama_model_loader: - kv   5:                   bert.feed_forward_length u32              = 4096
ollama  | llama_model_loader: - kv   6:                  bert.attention.head_count u32              = 16
ollama  | llama_model_loader: - kv   7:          bert.attention.layer_norm_epsilon f32              = 0.000000
ollama  | llama_model_loader: - kv   8:                          general.file_type u32              = 1
ollama  | llama_model_loader: - kv   9:                      bert.attention.causal bool             = false
ollama  | llama_model_loader: - kv  10:                          bert.pooling_type u32              = 2
ollama  | llama_model_loader: - kv  11:            tokenizer.ggml.token_type_count u32              = 2
ollama  | llama_model_loader: - kv  12:                tokenizer.ggml.bos_token_id u32              = 101
ollama  | llama_model_loader: - kv  13:                tokenizer.ggml.eos_token_id u32              = 102
ollama  | llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = bert
ollama  | llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,30522]   = ["[PAD]", "[unused0]", "[unused1]", "...
ollama  | llama_model_loader: - kv  16:                      tokenizer.ggml.scores arr[f32,30522]   = [-1000.000000, -1000.000000, -1000.00...
ollama  | llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,30522]   = [3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
ollama  | llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 100
ollama  | llama_model_loader: - kv  19:          tokenizer.ggml.seperator_token_id u32              = 102
ollama  | llama_model_loader: - kv  20:            tokenizer.ggml.padding_token_id u32              = 0
ollama  | llama_model_loader: - kv  21:                tokenizer.ggml.cls_token_id u32              = 101
ollama  | llama_model_loader: - kv  22:               tokenizer.ggml.mask_token_id u32              = 103
<!-- gh-comment-id:2189138155 --> @SebastianGode commented on GitHub (Jun 25, 2024): I've got same issue when tryaing to generate embeddings with multiple threads running on an Nvidia T4. ``` ollama | GGML_ASSERT: /go/src/github.com/ollama/ollama/llm/llama.cpp/llama.cpp:12095: seq_id < n_tokens && "seq_id cannot be larger than n_tokens with pooling_type == CLS" ollama | [GIN] 2024/06/25 - 14:34:41 | 200 | 967.02334ms | 172.30.0.3 | POST "/api/embeddings" ollama | time=2024-06-25T14:34:41.450Z level=INFO source=routes.go:400 msg="embedding generation failed: do embedding request: Post \"http://127.0.0.1:43813/embedding\": EOF" ollama | time=2024-06-25T14:34:41.450Z level=INFO source=routes.go:400 msg="embedding generation failed: health resp: Get \"http://127.0.0.1:43813/health\": EOF" ollama | [GIN] 2024/06/25 - 14:34:41 | 500 | 1.233321395s | 172.30.0.3 | POST "/api/embeddings" ollama | [GIN] 2024/06/25 - 14:34:41 | 500 | 1.103395244s | 172.30.0.3 | POST "/api/embeddings" ollama | time=2024-06-25T14:34:46.724Z level=WARN source=sched.go:575 msg="gpu VRAM usage didn't recover within timeout" seconds=5.24813948 model=/root/.ollama/models/blobs/sha256-819c2adf5ce6df2b6bd2ae4ca90d2a69f060afeb438d0c171db57daa02e39c3d ollama | time=2024-06-25T14:34:46.975Z level=WARN source=sched.go:575 msg="gpu VRAM usage didn't recover within timeout" seconds=5.498807909 model=/root/.ollama/models/blobs/sha256-819c2adf5ce6df2b6bd2ae4ca90d2a69f060afeb438d0c171db57daa02e39c3d ollama | time=2024-06-25T14:34:47.226Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=25 layers.offload=25 layers.split="" memory.available="[14.5 GiB]" memory.required.full="1.2 GiB" memory.required.partial="1.2 GiB" memory.required.kv="30.0 MiB" memory.required.allocations="[1.2 GiB]" memory.weights.total="607.2 MiB" memory.weights.repeating="547.6 MiB" memory.weights.nonrepeating="59.6 MiB" memory.graph.full="80.0 MiB" memory.graph.partial="80.0 MiB" ollama | time=2024-06-25T14:34:47.227Z level=INFO source=server.go:359 msg="starting llama server" cmd="/tmp/ollama2786287991/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-819c2adf5ce6df2b6bd2ae4ca90d2a69f060afeb438d0c171db57daa02e39c3d --ctx-size 5120 --batch-size 512 --embedding --log-disable --n-gpu-layers 25 --parallel 10 --port 36791" ollama | time=2024-06-25T14:34:47.227Z level=INFO source=sched.go:382 msg="loaded runners" count=1 ollama | time=2024-06-25T14:34:47.227Z level=INFO source=server.go:547 msg="waiting for llama runner to start responding" ollama | time=2024-06-25T14:34:47.227Z level=INFO source=server.go:585 msg="waiting for server to become available" status="llm server error" ollama | INFO [main] build info | build=1 commit="7c26775" tid="139639105748992" timestamp=1719326087 ollama | INFO [main] system info | n_threads=4 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="139639105748992" timestamp=1719326087 total_threads=8 ollama | INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="12" port="36791" tid="139639105748992" timestamp=1719326087 ollama | llama_model_loader: loaded meta data with 23 key-value pairs and 389 tensors from /root/.ollama/models/blobs/sha256-819c2adf5ce6df2b6bd2ae4ca90d2a69f060afeb438d0c171db57daa02e39c3d (version GGUF V3 (latest)) ollama | llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. ollama | llama_model_loader: - kv 0: general.architecture str = bert ollama | llama_model_loader: - kv 1: general.name str = mxbai-embed-large-v1 ollama | llama_model_loader: - kv 2: bert.block_count u32 = 24 ollama | llama_model_loader: - kv 3: bert.context_length u32 = 512 ollama | llama_model_loader: - kv 4: bert.embedding_length u32 = 1024 ollama | llama_model_loader: - kv 5: bert.feed_forward_length u32 = 4096 ollama | llama_model_loader: - kv 6: bert.attention.head_count u32 = 16 ollama | llama_model_loader: - kv 7: bert.attention.layer_norm_epsilon f32 = 0.000000 ollama | llama_model_loader: - kv 8: general.file_type u32 = 1 ollama | llama_model_loader: - kv 9: bert.attention.causal bool = false ollama | llama_model_loader: - kv 10: bert.pooling_type u32 = 2 ollama | llama_model_loader: - kv 11: tokenizer.ggml.token_type_count u32 = 2 ollama | llama_model_loader: - kv 12: tokenizer.ggml.bos_token_id u32 = 101 ollama | llama_model_loader: - kv 13: tokenizer.ggml.eos_token_id u32 = 102 ollama | llama_model_loader: - kv 14: tokenizer.ggml.model str = bert ollama | llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,30522] = ["[PAD]", "[unused0]", "[unused1]", "... ollama | llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,30522] = [-1000.000000, -1000.000000, -1000.00... ollama | llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,30522] = [3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... ollama | llama_model_loader: - kv 18: tokenizer.ggml.unknown_token_id u32 = 100 ollama | llama_model_loader: - kv 19: tokenizer.ggml.seperator_token_id u32 = 102 ollama | llama_model_loader: - kv 20: tokenizer.ggml.padding_token_id u32 = 0 ollama | llama_model_loader: - kv 21: tokenizer.ggml.cls_token_id u32 = 101 ollama | llama_model_loader: - kv 22: tokenizer.ggml.mask_token_id u32 = 103 ```
GiteaMirror commented 2026-04-22 07:02:00 -05:00
Author
Owner
Copy Link

@qdaioc commented on GitHub (Jul 15, 2024):

Same issue, any fix? thanks!

<!-- gh-comment-id:2228991481 --> @qdaioc commented on GitHub (Jul 15, 2024): Same issue, any fix? thanks!
GiteaMirror commented 2026-04-22 07:02:01 -05:00
Author
Owner
Copy Link

@hybra commented on GitHub (Jul 15, 2024):

I just had the same issue "no slots available after 10 retries", I don't know if this could be related or not, but the Ollama menubar icon was showing that there was an update available "Restart to update" - is maybe the server locked somehow when an update is pending?
Mac version 0.1.48 updated to 0.2.5

<!-- gh-comment-id:2229126894 --> @hybra commented on GitHub (Jul 15, 2024): I just had the same issue "no slots available after 10 retries", I don't know if this could be related or not, but the Ollama menubar icon was showing that there was an update available "Restart to update" - is maybe the server locked somehow when an update is pending? Mac version 0.1.48 updated to 0.2.5
GiteaMirror commented 2026-04-22 07:02:03 -05:00
Author
Owner
Copy Link

@iganev commented on GitHub (Jul 17, 2024):

I just had the same issue "no slots available after 10 retries", I don't know if this could be related or not, but the Ollama menubar icon was showing that there was an update available "Restart to update" - is maybe the server locked somehow when an update is pending? Mac version 0.1.48 updated to 0.2.5

Unfortunately, that's not the case. There's (imho) an issue with the underlying runner that ollama uses (namely llama.cpp) that isn't resolved and doesn't get handled properly when it occurs.

<!-- gh-comment-id:2233840316 --> @iganev commented on GitHub (Jul 17, 2024): > I just had the same issue "no slots available after 10 retries", I don't know if this could be related or not, but the Ollama menubar icon was showing that there was an update available "Restart to update" - is maybe the server locked somehow when an update is pending? Mac version 0.1.48 updated to 0.2.5 Unfortunately, that's not the case. There's (imho) an issue with the underlying runner that ollama uses (namely `llama.cpp`) that isn't resolved and doesn't get handled properly when it occurs.
GiteaMirror commented 2026-04-22 07:02:04 -05:00
Author
Owner
Copy Link

@Morphus1 commented on GitHub (Aug 11, 2024):

yup, think youre right. Not running ollama, just the java wrapper for llama.cpp:

GGML_ASSERT(seq_id < n_tokens && "seq_id cannot be larger than n_tokens with pooling_type == MEAN") failed

with latest update Sorry "GIT_TAG b3534"

rebuilt wrapper with latest:

GGML_ASSERT(seq_id < n_tokens && "seq_id cannot be larger than n_tokens with pooling_type == CLS") failed
GGML_ASSERT(seq_id < n_tokens && "seq_id cannot be larger than n_tokens with pooling_type == MEAN") failed

<!-- gh-comment-id:2282822992 --> @Morphus1 commented on GitHub (Aug 11, 2024): yup, think youre right. Not running ollama, just the java wrapper for llama.cpp: : GGML_ASSERT(seq_id < n_tokens && "seq_id cannot be larger than n_tokens with pooling_type == MEAN") failed # ~with latest update~ Sorry "GIT_TAG b3534" rebuilt wrapper with latest: GGML_ASSERT(seq_id < n_tokens && "seq_id cannot be larger than n_tokens with pooling_type == CLS") failed GGML_ASSERT(seq_id < n_tokens && "seq_id cannot be larger than n_tokens with pooling_type == MEAN") failed
GiteaMirror commented 2026-04-22 07:02:05 -05:00
Author
Owner
Copy Link

@figaroserg1 commented on GitHub (Sep 3, 2024):

Getting this error with llava model after just a few requests: no slots available after 10 retries

2024-09-03 11:44:39,890 - INFO - HTTP Request: POST http://127.0.0.1:11434/api/chat "HTTP/1.1 500 Internal Server Error"
2024-09-03 11:44:39,890 - DEBUG - receive_response_body.started request=<Request [b'POST']>
2024-09-03 11:44:39,890 - DEBUG - receive_response_body.complete
2024-09-03 11:44:39,890 - DEBUG - response_closed.started
2024-09-03 11:44:39,890 - DEBUG - response_closed.complete
2024-09-03 11:44:39,896 - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False
2024-09-03 11:44:39,897 - ERROR - Error processing file: Ready\August 22 Gen1\test (108).jpg - no slots available after 10 retries
2024-09-03 11:44:39,898 - DEBUG - load_verify_locations cafile='C:\Users\serg\AppData\Local\pypoetry\Cache\virtualenvs\autotagger-E-BNvv9j-py3.11\Lib\site-packages\certifi\cacert.pem'
2024-09-03 11:44:40,462 - DEBUG - send_request_headers.started request=<Request [b'POST']>
2024-09-03 11:44:40,463 - DEBUG - send_request_headers.complete
2024-09-03 11:44:40,463 - DEBUG - send_request_body.started request=<Request [b'POST']>
2024-09-03 11:44:40,476 - DEBUG - send_request_body.complete
2024-09-03 11:44:40,476 - DEBUG - receive_response_headers.started request=<Request [b'POST']>
2024-09-03 11:44:41,225 - DEBUG - receive_response_headers.complete return_value=(b'HTTP/1.1', 500, b'Internal Server Error', [(b'Content-Type', b'application/json; charset=utf-8'), (b'Date', b'Tue, 03 Sep 2024 08:44:41 GMT'), (b'Content-Length', b'47')])

<!-- gh-comment-id:2325988327 --> @figaroserg1 commented on GitHub (Sep 3, 2024): Getting this error with **llava** model after just a few requests: **no slots available after 10 retries** 2024-09-03 11:44:39,890 - INFO - HTTP Request: POST http://127.0.0.1:11434/api/chat "HTTP/1.1 500 Internal Server Error" 2024-09-03 11:44:39,890 - DEBUG - receive_response_body.started request=<Request [b'POST']> 2024-09-03 11:44:39,890 - DEBUG - receive_response_body.complete 2024-09-03 11:44:39,890 - DEBUG - response_closed.started 2024-09-03 11:44:39,890 - DEBUG - response_closed.complete 2024-09-03 11:44:39,896 - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False 2024-09-03 11:44:39,897 - ERROR - Error processing file: Ready\August 22 Gen1\test (108).jpg - no slots available after 10 retries 2024-09-03 11:44:39,898 - DEBUG - load_verify_locations cafile='C:\\Users\\serg\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\autotagger-E-BNvv9j-py3.11\\Lib\\site-packages\\certifi\\cacert.pem' 2024-09-03 11:44:40,462 - DEBUG - send_request_headers.started request=<Request [b'POST']> 2024-09-03 11:44:40,463 - DEBUG - send_request_headers.complete 2024-09-03 11:44:40,463 - DEBUG - send_request_body.started request=<Request [b'POST']> 2024-09-03 11:44:40,476 - DEBUG - send_request_body.complete 2024-09-03 11:44:40,476 - DEBUG - receive_response_headers.started request=<Request [b'POST']> 2024-09-03 11:44:41,225 - DEBUG - receive_response_headers.complete return_value=(b'HTTP/1.1', 500, b'Internal Server Error', [(b'Content-Type', b'application/json; charset=utf-8'), (b'Date', b'Tue, 03 Sep 2024 08:44:41 GMT'), (b'Content-Length', b'47')])
GiteaMirror commented 2026-04-22 07:02:06 -05:00
Author
Owner
Copy Link

@mjspeck commented on GitHub (Sep 3, 2024):

This is also a major issue for my team. We're running Ollama on an Ubuntu server with two H100s. We try to call Llava 1.6 with a large text prompt an an image and experience the exact same thing. We also do NOT have OLLAMA_NUM_PARALLEL set.

I'm also getting this I think. After some unpredictable number of llava calls with an image and prompt, the server will seemingly hang until I drop the connection. Subsequent calls give ollama._types.ResponseError: no slots available after 10 retries.

[GIN] 2024/06/20 - 14:10:37 | 200 | 317.913792ms | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:40 | 200 | 3.044791542s | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:41 | 200 | 325.550292ms | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:45 | 200 | 4.189981541s | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:46 | 200 | 373.940917ms | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:52 | 200 | 5.615178041s | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:52 | 200 | 418.198541ms | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:14:57 | 500 | 4m4s | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:18:14 | 500 | 76.112916ms | 127.0.0.1 | POST "/api/chat"

From testing a bit, this issue seems to come from longer prompts. Shortening our prompt made the error go away.

<!-- gh-comment-id:2326911341 --> @mjspeck commented on GitHub (Sep 3, 2024): This is also a major issue for my team. We're running Ollama on an Ubuntu server with two H100s. We try to call Llava 1.6 with a large text prompt an an image and experience the exact same thing. We also do NOT have `OLLAMA_NUM_PARALLEL` set. > I'm also getting this I think. After some unpredictable number of llava calls with an image and prompt, the server will seemingly hang until I drop the connection. Subsequent calls give `ollama._types.ResponseError: no slots available after 10 retries`. > > [GIN] 2024/06/20 - 14:10:37 | 200 | 317.913792ms | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:40 | 200 | 3.044791542s | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:41 | 200 | 325.550292ms | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:45 | 200 | 4.189981541s | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:46 | 200 | 373.940917ms | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:52 | 200 | 5.615178041s | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:10:52 | 200 | 418.198541ms | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:14:57 | 500 | 4m4s | 127.0.0.1 | POST "/api/chat" [GIN] 2024/06/20 - 14:18:14 | 500 | 76.112916ms | 127.0.0.1 | POST "/api/chat" From testing a bit, this issue seems to come from longer prompts. Shortening our prompt made the error go away.
GiteaMirror commented 2026-04-22 07:02:06 -05:00
Author
Owner
Copy Link

@liimiing commented on GitHub (Sep 13, 2024):

same error, how to resolved it?
Added image 'E:\At_Work\2a94e678-6fed-44e7-84c2-b69d92b43f33.jpg'
Error: no slots available after 10 retries

<!-- gh-comment-id:2348187251 --> @liimiing commented on GitHub (Sep 13, 2024): same error, how to resolved it? Added image 'E:\At_Work\2a94e678-6fed-44e7-84c2-b69d92b43f33.jpg' Error: no slots available after 10 retries
GiteaMirror commented 2026-04-22 07:02:07 -05:00
Author
Owner
Copy Link

@iganev commented on GitHub (Sep 13, 2024):

same error, how to resolved it? Added image 'E:\At_Work\2a94e678-6fed-44e7-84c2-b69d92b43f33.jpg' Error: no slots available after 10 retries

You can partially mitigate the issue by enabling parallelism. It will still occur, but it will self-recover and start serving requests again in few seconds.

No permanent solution so far.

<!-- gh-comment-id:2349306541 --> @iganev commented on GitHub (Sep 13, 2024): > same error, how to resolved it? Added image 'E:\At_Work\2a94e678-6fed-44e7-84c2-b69d92b43f33.jpg' Error: no slots available after 10 retries You can partially mitigate the issue by enabling parallelism. It will still occur, but it will self-recover and start serving requests again in few seconds. No permanent solution so far.
GiteaMirror commented 2026-04-22 07:02:08 -05:00
Author
Owner
Copy Link

@HTK-Tech commented on GitHub (Sep 30, 2024):

My workaround ATM when I get this error is to make a request with only the keep_alive parameter set to 0 to unload the model, wait one second, and then resume making requests normally.

await ollama.chat({
        model: 'llava:34b',
        keep_alive: 0,
    });
<!-- gh-comment-id:2381842956 --> @HTK-Tech commented on GitHub (Sep 30, 2024): My workaround ATM when I get this error is to make a request with only the keep_alive parameter set to 0 to unload the model, wait one second, and then resume making requests normally. ``` await ollama.chat({ model: 'llava:34b', keep_alive: 0, }); ```
GiteaMirror commented 2026-04-22 07:02:09 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Oct 23, 2024):

Please give the new 0.4.0 RC a try. We've rewritten the caching (and slot) model for processing requests in the new Go server, so this issue is most likely resolved.

https://github.com/ollama/ollama/releases

<!-- gh-comment-id:2433473029 --> @dhiltgen commented on GitHub (Oct 23, 2024): Please give the new 0.4.0 RC a try. We've rewritten the caching (and slot) model for processing requests in the new Go server, so this issue is most likely resolved. https://github.com/ollama/ollama/releases
GiteaMirror commented 2026-04-22 07:02:09 -05:00
Author
Owner
Copy Link

@giladrom commented on GitHub (Oct 27, 2024):

I can confirm 0.4.0-rc5 does not fix the issue for me. It still times out after 1m40s for most workloads, which I am not sure how to resolve. Where is the 1m40s timeout coming from?

source=llama-server.go:821 msg="Failed to acquire semaphore" error="context canceled"
[GIN] 2024/10/27 - 08:40:55 | 500 |         1m40s |  XXX.XXX.XXX.XXX  | POST     "/api/embed"
source=.:0 msg="http: superfluous response.WriteHeader call from main.(*Server).embeddings (runner.go:743)"
<!-- gh-comment-id:2439910382 --> @giladrom commented on GitHub (Oct 27, 2024): I can confirm `0.4.0-rc5` does not fix the issue for me. It still times out after 1m40s for most workloads, which I am not sure how to resolve. Where is the 1m40s timeout coming from? ``` source=llama-server.go:821 msg="Failed to acquire semaphore" error="context canceled" [GIN] 2024/10/27 - 08:40:55 | 500 | 1m40s | XXX.XXX.XXX.XXX | POST "/api/embed" source=.:0 msg="http: superfluous response.WriteHeader call from main.(*Server).embeddings (runner.go:743)" ```
GiteaMirror commented 2026-04-22 07:02:10 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Oct 28, 2024):

@giladrom looking at the code, I believe your client is timing out and closing the connection before the server is able to process the request. It sounds like you may have a 100s timeout in your client code, or maybe client libraries you're using. Do you believe the system is hung when this happens? Can you try to bump up the timeout on your side and see if it is able to make forward progress or if the system is truly hung. The semaphore in question manages the concurrent requests to the model, so depending on what it got wired up with (4 or 1 depending on VRAM, unless you set OLLAMA_NUM_PARALLEL to something else) it will prevent more requests from being processing in parallel until prior requests complete.

<!-- gh-comment-id:2441989510 --> @dhiltgen commented on GitHub (Oct 28, 2024): @giladrom looking at the code, I believe your client is timing out and closing the connection before the server is able to process the request. It sounds like you may have a 100s timeout in your client code, or maybe client libraries you're using. Do you believe the system is hung when this happens? Can you try to bump up the timeout on your side and see if it is able to make forward progress or if the system is truly hung. The semaphore in question manages the concurrent requests to the model, so depending on what it got wired up with (4 or 1 depending on VRAM, unless you set OLLAMA_NUM_PARALLEL to something else) it will prevent more requests from being processing in parallel until prior requests complete.
GiteaMirror commented 2026-04-22 07:02:11 -05:00
Author
Owner
Copy Link

@giladrom commented on GitHub (Oct 28, 2024):

@dhiltgen That was my conclusion too - I tried looking for 100s timeouts in my client code but couldn't find anything. I'm using LangChain, so that's probably the culprit. For now, I resolved this by using a different Embedding Model vs. Chat model (llama3.2 works fine for chat, and using nomic for embeddings, which is super fast)

<!-- gh-comment-id:2442104890 --> @giladrom commented on GitHub (Oct 28, 2024): @dhiltgen That was my conclusion too - I tried looking for 100s timeouts in my client code but couldn't find anything. I'm using LangChain, so that's probably the culprit. For now, I resolved this by using a different Embedding Model vs. Chat model (llama3.2 works fine for chat, and using nomic for embeddings, which is super fast)
GiteaMirror commented 2026-04-22 07:02:12 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Oct 28, 2024):

@giladrom so just to confirm, avoiding slower models that trigger a LangChain default client-side timeout, you're able to keep running for a long period of time without hangs, is that correct? Has 0.4.0 solved the hangs for you?

<!-- gh-comment-id:2442833687 --> @dhiltgen commented on GitHub (Oct 28, 2024): @giladrom so just to confirm, avoiding slower models that trigger a LangChain default client-side timeout, you're able to keep running for a long period of time without hangs, is that correct? Has 0.4.0 solved the hangs for you?
GiteaMirror commented 2026-04-22 07:02:13 -05:00
Author
Owner
Copy Link

@giladrom commented on GitHub (Oct 30, 2024):

@dhiltgen Correct. 0.3.6 (the docker image default) works fine with smaller embedding models, taking about 12s to complete a job that times out with llama3.2, as does 0.4.0. Trying to use llama3.2 for embeddings times out for both versions.

<!-- gh-comment-id:2446323641 --> @giladrom commented on GitHub (Oct 30, 2024): @dhiltgen Correct. 0.3.6 (the docker image default) works fine with smaller embedding models, taking about 12s to complete a job that times out with llama3.2, as does 0.4.0. Trying to use llama3.2 for embeddings times out for both versions.
GiteaMirror commented 2026-04-22 07:02:13 -05:00
Author
Owner
Copy Link

@KIC commented on GitHub (Oct 31, 2024):

Interestingly, I ended up here having the same issue using plain ollama server. However, I realized I only have this problem with one specific model like hhao/openbmb-minicpm-llama3-v-2_5 although it has worked just fine so far. After I delete the model and download it again the problem seems to be gone - at least for the moment.

<!-- gh-comment-id:2449370664 --> @KIC commented on GitHub (Oct 31, 2024): Interestingly, I ended up here having the same issue using plain ollama server. However, I realized I only have this problem with one specific model like `hhao/openbmb-minicpm-llama3-v-2_5` although it has worked just fine so far. After I delete the model and download it again the problem seems to be gone - at least for the moment.
GiteaMirror commented 2026-04-22 07:02:14 -05:00
Author
Owner
Copy Link

@jashanj0tsingh commented on GitHub (Nov 3, 2024):

@giladrom so just to confirm, avoiding slower models that trigger a LangChain default client-side timeout, you're able to keep running for a long period of time without hangs, is that correct? Has 0.4.0 solved the hangs for you?

Pardon my partial understanding of @giladrom 's use case, but I tried regular old curl request considering client timeout issue with LangChain and tried to load a model different than the one loaded already and not responding (well more on this later)**, but a simple curl request once ollama was in that state took approximately 10 mins to answer why the sky is blue, for me, neither docker, or building from source made any difference (even with go runner), its heartbreaking being unable to use such a great tool.

**With OLLAMA_DEBUG=1 I noticed that the server didn't even manage to log the POST request on a FOREVER loaded model. Once my LangChain application got stuck waiting for ollama req/resp cycle.

Unfortunately, for me at the moment the only resolution is a reboot of the VM, and then 20 mins of sweet glory until it is time to reboot again. I have started considering vLLM at this point because there is no way this would fly in production.

Note: I am on a fully loaded VM on vCenter ( ubuntu 22.04 ), cuda 12, and 2 x 16GB Nvidia A16-16Q, 64 GB RAM

<!-- gh-comment-id:2453441003 --> @jashanj0tsingh commented on GitHub (Nov 3, 2024): > @giladrom so just to confirm, avoiding slower models that trigger a LangChain default client-side timeout, you're able to keep running for a long period of time without hangs, is that correct? Has 0.4.0 solved the hangs for you? Pardon my partial understanding of @giladrom 's use case, but I tried regular old curl request considering client timeout issue with LangChain and tried to load a model different than the one loaded already and not responding (well more on this later)**, but a simple curl request once `ollama` was in that state took approximately 10 mins to answer why the sky is blue, for me, neither docker, or building from source made any difference (even with go runner), its heartbreaking being unable to use such a great tool. **With OLLAMA_DEBUG=1 I noticed that the server didn't even manage to log the POST request on a FOREVER loaded model. Once my LangChain application got stuck waiting for ollama req/resp cycle. Unfortunately, for me at the moment the only resolution is a reboot of the VM, and then 20 mins of sweet glory until it is time to reboot again. I have started considering vLLM at this point because there is no way this would fly in production. Note: I am on a fully loaded VM on vCenter ( ubuntu 22.04 ), cuda 12, and 2 x 16GB Nvidia A16-16Q, 64 GB RAM
GiteaMirror commented 2026-04-22 07:02:15 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Nov 3, 2024):

@jashanj0tsingh can you clarify your scenario a bit? It sounds like you are exercising 2 different models, one with LangChain as the client, one with curl. Do both of these models fit 100% in the GPU(s) or is the Ollama scheduler waiting those 10m for the LangChain client connections to close so that it can unload that model and load the secondary model requested by curl?

Can you explain a little more about what the LangChain client is doing? Is it making multiple requests in parallel, and if so, what limits does it have? Is it calling generate/chat or embedding APIs? Have you changed any scheduler settings in Ollama? Embedding models currently run a single processing thread, and our default queue size is 512 client requests, so if the LangChain client is launch hundreds of parallel requests and those in turn are taking seconds or more to process, perhaps the system isn't actually stuck, just backlogged?

Are you able to observe other metrics on the system when it gets into this state? CPU load, memory load, disk I/O, GPU load, etc.

I might be misunderstanding your scenario, but feels more like a heavy load and working through a backlog and not a "hang after 10-15 minutes" scenario, where Ollama becomes completely wedged and has to be killed to recover.

<!-- gh-comment-id:2453589264 --> @dhiltgen commented on GitHub (Nov 3, 2024): @jashanj0tsingh can you clarify your scenario a bit? It sounds like you are exercising 2 different models, one with LangChain as the client, one with curl. Do both of these models fit 100% in the GPU(s) or is the Ollama scheduler waiting those 10m for the LangChain client connections to close so that it can unload that model and load the secondary model requested by curl? Can you explain a little more about what the LangChain client is doing? Is it making multiple requests in parallel, and if so, what limits does it have? Is it calling generate/chat or embedding APIs? Have you changed any scheduler settings in Ollama? Embedding models currently run a single processing thread, and our default queue size is 512 client requests, so if the LangChain client is launch hundreds of parallel requests and those in turn are taking seconds or more to process, perhaps the system isn't actually stuck, just backlogged? Are you able to observe other metrics on the system when it gets into this state? CPU load, memory load, disk I/O, GPU load, etc. I might be misunderstanding your scenario, but feels more like a heavy load and working through a backlog and not a "hang after 10-15 minutes" scenario, where Ollama becomes completely wedged and has to be killed to recover.
GiteaMirror commented 2026-04-22 07:02:16 -05:00
Author
Owner
Copy Link

@jashanj0tsingh commented on GitHub (Nov 3, 2024):

@dhiltgen thank you for your prompt reply, really appreciate you working on a weekend, yes sure, I'd be happy to explain the scenario, I will try to answer all of your questions,

Context: I building a rather simple REST API server using FastAPI that wraps around LangGraph and Ollama to provide chat/generate endpoints with additional features such as chat history persistence etc.

My Issue: After a couple of requests via the REST API the model seems unreachable, and if the request manages to reach the model somehow, it responds extremely sluggishly, imagine a streaming response with 1 word per two seconds. Experienced with or without LangGraph, consistent across binary installation, manual build ( with and without go runner ), and docker images 3.14 and 0.4.0-rc5.

My resolution: A reboot of the VM, nothing else works, killing the service, killing the container, restarting the binary, none of these work. If I don't reboot, the model will take forever to accept the request and respond extremely sluggishly, talked a bit more below.

Now, I am using smaller models like Llama 3.1 [7B] and Llama 3.2 [3B], and yes they fit with adequate room in my 2x 16 GB GPUs, also, my use case doesn't necessarily requires two separate models, that was just a test as I actively follow this thread and saw your recent conversation and since Llama 3.1 was already in that state, I tried a curl request instead with Llama 3.2, but the result was the same, firstly it took forever for the request to land at the server and then the stream response was 1 words per 2 seconds.

Embeddings are pre-created, just using a simple retriever to query the embeddings instead for a RAG like use-case. I am not doing multiple requests rather sequential conversation back and forth with one user at a time. I have not changed any scheduler settings, everything default, recently started using keep_alive=-1 to keep the model loaded forever but no luck.

htop shows nothing crazy, nvidia-smi seems fine, with no memory overheads, its something I have been experiencing from quite some time, and I am not sure if its related or not but it does stop working after 10-15 minutes, max 20 minutes, and that's it.

With OLLAMA_DEBUG=1 nothing suspicious jumps out in the logs as well.

Edit: I can verify if vCenter has anything to do with it but I am running a fairly small setup on an adequate hardware with enough room to spare. I am planning to verify this by running the model with vLLM. If vLLM works fine it has something to do with Ollama, if not then I may have a different issue.

Edit2: Its just single, sequential request/response cycles with one model that behave like mentioned above.

<!-- gh-comment-id:2453603697 --> @jashanj0tsingh commented on GitHub (Nov 3, 2024): @dhiltgen thank you for your prompt reply, really appreciate you working on a weekend, yes sure, I'd be happy to explain the scenario, I will try to answer all of your questions, Context: I building a rather simple REST API server using FastAPI that wraps around `LangGraph` and `Ollama` to provide chat/generate endpoints with additional features such as chat history persistence etc. My Issue: After a couple of requests via the REST API the model seems unreachable, and if the request manages to reach the model somehow, it responds extremely sluggishly, imagine a streaming response with 1 word per two seconds. Experienced with or without LangGraph, consistent across binary installation, manual build ( with and without go runner ), and docker images 3.14 and 0.4.0-rc5. My resolution: A reboot of the VM, nothing else works, killing the service, killing the container, restarting the binary, none of these work. If I don't reboot, the model will take forever to accept the request and respond extremely sluggishly, talked a bit more below. Now, I am using smaller models like Llama 3.1 [7B] and Llama 3.2 [3B], and yes they fit with adequate room in my 2x 16 GB GPUs, also, my use case doesn't necessarily requires two separate models, that was just a test as I actively follow this thread and saw your recent conversation and since Llama 3.1 was already in that state, I tried a curl request instead with Llama 3.2, but the result was the same, firstly it took forever for the request to land at the server and then the stream response was 1 words per 2 seconds. Embeddings are pre-created, just using a simple retriever to query the embeddings instead for a RAG like use-case. I am not doing multiple requests rather sequential conversation back and forth with one user at a time. I have not changed any scheduler settings, everything default, recently started using `keep_alive=-1` to keep the model loaded forever but no luck. `htop` shows nothing crazy, `nvidia-smi` seems fine, with no memory overheads, its something I have been experiencing from quite some time, and I am not sure if its related or not but it does stop working after 10-15 minutes, max 20 minutes, and that's it. With OLLAMA_DEBUG=1 nothing suspicious jumps out in the logs as well. Edit: I can verify if vCenter has anything to do with it but I am running a fairly small setup on an adequate hardware with enough room to spare. I am planning to verify this by running the model with `vLLM`. If `vLLM` works fine it has something to do with `Ollama`, if not then I may have a different issue. Edit2: Its just single, sequential request/response cycles with one model that behave like mentioned above.
GiteaMirror commented 2026-04-22 07:02:17 -05:00
Author
Owner
Copy Link

@giladrom commented on GitHub (Nov 4, 2024):

@dhiltgen It seems there's a huge difference in performance between 0.3.x and 0.4.x after all - I just retested 0.4.0-rc6 this morning and all files I've tried (ranging from a few kilobytes to 60MB PDFs) completed embedding without timing out, whereas 0.3.6 would time out for anything larger than a few megabytes.

<!-- gh-comment-id:2454062269 --> @giladrom commented on GitHub (Nov 4, 2024): @dhiltgen It seems there's a huge difference in performance between 0.3.x and 0.4.x after all - I just retested `0.4.0-rc6` this morning and all files I've tried (ranging from a few kilobytes to 60MB PDFs) completed embedding without timing out, whereas 0.3.6 would time out for anything larger than a few megabytes.
GiteaMirror commented 2026-04-22 07:02:17 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Nov 4, 2024):

@jashanj0tsingh

A reboot of the VM, nothing else works, killing the service, killing the container, restarting the binary, none of these work. If I don't reboot, the model will take forever to accept the request and respond extremely sluggishly, talked a bit more below.

This is interesting. Just to confirm, terminating ollama and any/all subprocesses (ollama_llama_server) does not clear it up? You mentioned you're running in a VM in vSphere, so this is ESXi passing through the GPU, correct? It seems to imply this fault may be a driver bug, or maybe a virtualization bug that somehow Ollama is triggering based on our usage patterns of the GPU. Is it possible for you to run your scenario on the bare metal to see if you see the same problem occur without virtualization involved? It may also be useful to run a dmesg -w in the VM to see if any interesting warnings/errors are being reported by the NVIDIA drivers when things slow down. For the Ollama server process, setting CUDA_ERROR_LEVEL=50 may yield more warnings/errors from the cuda libraries.

<!-- gh-comment-id:2455316626 --> @dhiltgen commented on GitHub (Nov 4, 2024): @jashanj0tsingh > A reboot of the VM, nothing else works, killing the service, killing the container, restarting the binary, none of these work. If I don't reboot, the model will take forever to accept the request and respond extremely sluggishly, talked a bit more below. This is interesting. Just to confirm, terminating ollama and any/all subprocesses (ollama_llama_server) does not clear it up? You mentioned you're running in a VM in vSphere, so this is ESXi passing through the GPU, correct? It seems to imply this fault may be a driver bug, or maybe a virtualization bug that somehow Ollama is triggering based on our usage patterns of the GPU. Is it possible for you to run your scenario on the bare metal to see if you see the same problem occur without virtualization involved? It may also be useful to run a `dmesg -w` in the VM to see if any interesting warnings/errors are being reported by the NVIDIA drivers when things slow down. For the Ollama server process, setting `CUDA_ERROR_LEVEL=50` may yield more warnings/errors from the cuda libraries.
GiteaMirror commented 2026-04-22 07:02:18 -05:00
Author
Owner
Copy Link

@iganev commented on GitHub (Nov 5, 2024):

My observation on ubuntu 22.04 nvidia 550.127.05 cuda 12.4 using nvidia-docker with ollama 0.3.14 is that it seems to behave much more stable than before.
I need to find time to do more in-depth testing, but so far I cant reproduce the issue.

<!-- gh-comment-id:2458428482 --> @iganev commented on GitHub (Nov 5, 2024): My observation on ubuntu 22.04 nvidia 550.127.05 cuda 12.4 using nvidia-docker with ollama 0.3.14 is that it seems to behave much more stable than before. I need to find time to do more in-depth testing, but so far I cant reproduce the issue.
GiteaMirror commented 2026-04-22 07:02:19 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Nov 6, 2024):

@iganev can you give 0.4.0 a try and see if it resolves the hang after ~15m?

<!-- gh-comment-id:2460376254 --> @dhiltgen commented on GitHub (Nov 6, 2024): @iganev can you give 0.4.0 a try and see if it resolves the hang after ~15m?
GiteaMirror commented 2026-04-22 07:02:19 -05:00
Author
Owner
Copy Link

@jashanj0tsingh commented on GitHub (Nov 7, 2024):

@dhiltgen Wow, that was on point, it indeed was virtualization related, there was an issue that was disrupting the license token ping for the hardware, causing the virtual hardware itself to be sluggish and non-responsive, I accidentally stumbled upon this just because the fired a curl and left it and saw the next morning it landed on ollama, anyways, I am equally relieved and embarrassed, but ollama is off the hook, I just was looking in the wrong direction, apologies for the inconvenience it may have caused.

<!-- gh-comment-id:2462391871 --> @jashanj0tsingh commented on GitHub (Nov 7, 2024): @dhiltgen Wow, that was on point, it indeed was virtualization related, there was an issue that was disrupting the license token ping for the hardware, causing the virtual hardware itself to be sluggish and non-responsive, I accidentally stumbled upon this just because the fired a curl and left it and saw the next morning it landed on `ollama`, anyways, I am equally relieved and embarrassed, but `ollama` is off the hook, I just was looking in the wrong direction, apologies for the inconvenience it may have caused.
GiteaMirror commented 2026-04-22 07:02:20 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Nov 7, 2024):

I'm going to go ahead and close this one out now. If anyone does see the hang reappear with 0.4.0 let us know and I'll reopen the issue to investigate further.

<!-- gh-comment-id:2462787222 --> @dhiltgen commented on GitHub (Nov 7, 2024): I'm going to go ahead and close this one out now. If anyone does see the hang reappear with 0.4.0 let us know and I'll reopen the issue to investigate further.
GiteaMirror commented 2026-04-22 07:02:21 -05:00
Author
Owner
Copy Link

@JTHesse commented on GitHub (Nov 8, 2024):

I'm seeing a similar behavior for 0.4.0, but with a 503 error -> #7573

<!-- gh-comment-id:2464822184 --> @JTHesse commented on GitHub (Nov 8, 2024): I'm seeing a similar behavior for 0.4.0, but with a 503 error -> #7573
GiteaMirror commented 2026-04-22 07:02:23 -05:00
Author
Owner
Copy Link

@gslin1224 commented on GitHub (Jan 16, 2025):

my version is 0.4.1 , Still Have the same Issue

<!-- gh-comment-id:2596361648 --> @gslin1224 commented on GitHub (Jan 16, 2025): my version is 0.4.1 , Still Have the same Issue
GiteaMirror commented 2026-04-22 07:02:23 -05:00
Author
Owner
Copy Link

@jessegross commented on GitHub (Jan 17, 2025):

@gslin1224 Please try it with 0.4.4 or later as there were some additional fixes needed.

<!-- gh-comment-id:2599355797 --> @jessegross commented on GitHub (Jan 17, 2025): @gslin1224 Please try it with 0.4.4 or later as there were some additional fixes needed.
GiteaMirror commented 2026-04-22 07:02:24 -05:00
Author
Owner
Copy Link

@Chleba commented on GitHub (Mar 13, 2025):

I'm still experiencing same Issue as well using vision models for labeling images. After every 30-60. calls ollama will freeze and needs to be restarted.
Using ollama v0.5.7

<!-- gh-comment-id:2722155158 --> @Chleba commented on GitHub (Mar 13, 2025): I'm still experiencing same Issue as well using vision models for labeling images. After every 30-60. calls ollama will freeze and needs to be restarted. Using ollama v0.5.7
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug needs more info
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#28610
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?