github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-11-12 03:46:14 -06:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #9325] Embedding failed after some requests #68142

New Issue
Open
opened 2026-05-04 12:37:47 -05:00 by GiteaMirror · 14 comments
GiteaMirror commented 2026-05-04 12:37:47 -05:00
Owner
Copy Link
Copy Source

Originally created by @QichangZheng on GitHub (Feb 25, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9325

What is the issue?

Ollama version: 0.5.4
GPU: H20
Embedding will fail after several hundreds requests. VRAM 1GB taken out of 96GB.

Relevant log output

time=2025-02-25T11:45:36.531+08:00 level=DEBUG source=cache.go:104 msg="loading cache slot" id=0 cache=1069 prompt=1109 used=0 remaining=1109
[GIN] 2025/02/25 - 11:45:36 | 200 |   21.645947ms |       127.0.0.1 | POST     "/api/embeddings"
time=2025-02-25T11:45:36.551+08:00 level=DEBUG source=sched.go:407 msg="context for request finished"
time=2025-02-25T11:45:36.551+08:00 level=DEBUG source=sched.go:339 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/autodl-tmp/ollama/blobs/sha256-970aa74c0a90ef7482477cf803618e776e173c007bf957f635f1015bfcfef0e6 duration=2h0m0s
time=2025-02-25T11:45:36.551+08:00 level=DEBUG source=sched.go:357 msg="after processing request finished event" modelPath=/root/autodl-tmp/ollama/blobs/sha256-970aa74c0a90ef7482477cf803618e776e173c007bf957f635f1015bfcfef0e6 refCount=0

OS

Linux

GPU

No response

CPU

No response

Ollama version

No response

Originally created by @QichangZheng on GitHub (Feb 25, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9325 ### What is the issue? Ollama version: 0.5.4 GPU: H20 Embedding will fail after several hundreds requests. VRAM 1GB taken out of 96GB. ### Relevant log output ```shell time=2025-02-25T11:45:36.531+08:00 level=DEBUG source=cache.go:104 msg="loading cache slot" id=0 cache=1069 prompt=1109 used=0 remaining=1109 [GIN] 2025/02/25 - 11:45:36 | 200 | 21.645947ms | 127.0.0.1 | POST "/api/embeddings" time=2025-02-25T11:45:36.551+08:00 level=DEBUG source=sched.go:407 msg="context for request finished" time=2025-02-25T11:45:36.551+08:00 level=DEBUG source=sched.go:339 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/autodl-tmp/ollama/blobs/sha256-970aa74c0a90ef7482477cf803618e776e173c007bf957f635f1015bfcfef0e6 duration=2h0m0s time=2025-02-25T11:45:36.551+08:00 level=DEBUG source=sched.go:357 msg="after processing request finished event" modelPath=/root/autodl-tmp/ollama/blobs/sha256-970aa74c0a90ef7482477cf803618e776e173c007bf957f635f1015bfcfef0e6 refCount=0 ``` ### OS Linux ### GPU _No response_ ### CPU _No response_ ### Ollama version _No response_
GiteaMirror added the bug label 2026-05-04 12:37:47 -05:00
GiteaMirror commented 2026-05-04 12:37:51 -05:00
Author
Owner
Copy Link
Copy Source

@RicardoOM-AC commented on GitHub (Jul 24, 2025):

Did this got resolved? Having the same issue, running fine, VRAM is not filling, but after some thousand embeddings, it fails, and do not recover until restart the server.

Ollama.zip

<!-- gh-comment-id:3114885415 --> @RicardoOM-AC commented on GitHub (Jul 24, 2025): Did this got resolved? Having the same issue, running fine, VRAM is not filling, but after some thousand embeddings, it fails, and do not recover until restart the server. [Ollama.zip](https://github.com/user-attachments/files/21419003/Ollama.zip)
GiteaMirror commented 2026-05-04 12:37:54 -05:00
Author
Owner
Copy Link
Copy Source

@icariantk commented on GitHub (Aug 13, 2025):

Ollama version=0.11.4 Still failing after some thousand embeddings.

<!-- gh-comment-id:3186048194 --> @icariantk commented on GitHub (Aug 13, 2025): Ollama version=0.11.4 Still failing after some thousand embeddings.
GiteaMirror commented 2026-05-04 12:37:57 -05:00
Author
Owner
Copy Link
Copy Source

@icariantk commented on GitHub (Aug 26, 2025):

Ollama version=0.11.6 Still failing after some thousand entries

Image

Killing this process allows me to keep working the series

Image Image
<!-- gh-comment-id:3225180304 --> @icariantk commented on GitHub (Aug 26, 2025): Ollama version=0.11.6 Still failing after some thousand entries <img width="691" height="523" alt="Image" src="https://github.com/user-attachments/assets/8d7dd365-30ce-464f-bec3-764bb415f37d" /> Killing this process allows me to keep working the series <img width="1221" height="771" alt="Image" src="https://github.com/user-attachments/assets/d4599601-83a7-48fc-8e19-d0bec78f5245" /> <img width="875" height="562" alt="Image" src="https://github.com/user-attachments/assets/608969f3-ca06-44bc-9f14-45e1e6a6654c" />
GiteaMirror commented 2026-05-04 12:37:59 -05:00
Author
Owner
Copy Link
Copy Source

@rick-github commented on GitHub (Aug 26, 2025):

Server logs may help in debugging.

<!-- gh-comment-id:3225255570 --> @rick-github commented on GitHub (Aug 26, 2025): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) may help in debugging.
GiteaMirror commented 2026-05-04 12:38:03 -05:00
Author
Owner
Copy Link
Copy Source

@icariantk commented on GitHub (Aug 26, 2025):

There is a log 3 comments above, but no problem, the process is still running and need to kill that ollama instances each ~700 embeddings, so i will be here for a long time.

server - copia.zip

<!-- gh-comment-id:3225295396 --> @icariantk commented on GitHub (Aug 26, 2025): There is a log 3 comments above, but no problem, the process is still running and need to kill that ollama instances each ~700 embeddings, so i will be here for a long time. [server - copia.zip](https://github.com/user-attachments/files/21993522/server.-.copia.zip)
GiteaMirror commented 2026-05-04 12:38:06 -05:00
Author
Owner
Copy Link
Copy Source

@rick-github commented on GitHub (Aug 26, 2025):

/api/embeddings is deprecated, the new endpoint is /api/embed.

[GIN] 2025/08/26 - 12:34:35 | 200 |     45.7631ms |             ::1 | POST     "/api/embeddings"
[GIN] 2025/08/26 - 12:35:36 | 500 |          1m0s |             ::1 | POST     "/api/embeddings"

Things were going fine until this call took too long and the client timed out. While the client is waiting for the response, what is the GPU doing? What does nvidia-smi show?

<!-- gh-comment-id:3225331614 --> @rick-github commented on GitHub (Aug 26, 2025): `/api/embeddings` is [deprecated](https://github.com/ollama/ollama/blob/main/docs/api.md#generate-embedding), the new endpoint is [`/api/embed`](https://github.com/ollama/ollama/blob/main/docs/api.md#generate-embeddings). ``` [GIN] 2025/08/26 - 12:34:35 | 200 | 45.7631ms | ::1 | POST "/api/embeddings" [GIN] 2025/08/26 - 12:35:36 | 500 | 1m0s | ::1 | POST "/api/embeddings" ``` Things were going fine until this call took too long and the client timed out. While the client is waiting for the response, what is the GPU doing? What does `nvidia-smi` show?
GiteaMirror commented 2026-05-04 12:38:07 -05:00
Author
Owner
Copy Link
Copy Source

@icariantk commented on GitHub (Aug 26, 2025):

This is the current state, the process has halted, i will restart the process after commenting:

info.txt

<!-- gh-comment-id:3225337970 --> @icariantk commented on GitHub (Aug 26, 2025): This is the current state, the process has halted, i will restart the process after commenting: [info.txt](https://github.com/user-attachments/files/21993703/info.txt)
GiteaMirror commented 2026-05-04 12:38:09 -05:00
Author
Owner
Copy Link
Copy Source

@icariantk commented on GitHub (Aug 26, 2025):

GPU Is not stressed at all

Image
<!-- gh-comment-id:3225342125 --> @icariantk commented on GitHub (Aug 26, 2025): GPU Is not stressed at all <img width="1913" height="1031" alt="Image" src="https://github.com/user-attachments/assets/b4730879-32d3-4faf-9150-5344c65f51ad" />
GiteaMirror commented 2026-05-04 12:38:10 -05:00
Author
Owner
Copy Link
Copy Source

@rick-github commented on GitHub (Aug 26, 2025):

It's not a matter of stress, I'm trying to determine the failure point. If the GPU is busy, it means the runner received the embed request but for some reason is stuck in a processing loop. If the GPU is not busy (which seems to be the case based on the nvidia-smi output), the runner is wedged in some other op. It received the embed request:

time=2025-08-26T12:35:36.220-06:00 level=WARN source=runner.go:127 msg="truncating input prompt" limit=512 prompt=642 keep=1 new=512

but then either didn't fire up a GPU kernel, or did and finished processing but didn't return the response to the client. So now we have some idea of where to look.

<!-- gh-comment-id:3225368706 --> @rick-github commented on GitHub (Aug 26, 2025): It's not a matter of stress, I'm trying to determine the failure point. If the GPU is busy, it means the runner received the embed request but for some reason is stuck in a processing loop. If the GPU is not busy (which seems to be the case based on the `nvidia-smi` output), the runner is wedged in some other op. It received the embed request: ``` time=2025-08-26T12:35:36.220-06:00 level=WARN source=runner.go:127 msg="truncating input prompt" limit=512 prompt=642 keep=1 new=512 ``` but then either didn't fire up a GPU kernel, or did and finished processing but didn't return the response to the client. So now we have some idea of where to look.
GiteaMirror commented 2026-05-04 12:38:11 -05:00
Author
Owner
Copy Link
Copy Source

@icariantk commented on GitHub (Aug 26, 2025):

Let me know if can help, i will need to keep the process running, so i will be here for a while.

<!-- gh-comment-id:3225381424 --> @icariantk commented on GitHub (Aug 26, 2025): Let me know if can help, i will need to keep the process running, so i will be here for a while.
GiteaMirror commented 2026-05-04 12:38:12 -05:00
Author
Owner
Copy Link
Copy Source

@icariantk commented on GitHub (Aug 26, 2025):

Memory in the process looks like its not growing, but the processing keeps steady in the "locked" state:

Image
<!-- gh-comment-id:3225417547 --> @icariantk commented on GitHub (Aug 26, 2025): Memory in the process looks like its not growing, but the processing keeps steady in the "locked" state: <img width="1219" height="762" alt="Image" src="https://github.com/user-attachments/assets/50be5e90-7dfb-4145-8787-8f98f8105421" />
GiteaMirror commented 2026-05-04 12:38:13 -05:00
Author
Owner
Copy Link
Copy Source

@icariantk commented on GitHub (Aug 26, 2025):

I wrote a small PowerShell script to automatically terminate the ollama.exe process every ~15 minutes. That interval matched the recurrence of the error I was seeing, and since running this script the issue has stabilized for longer compared to manually finalizing the task.

Here’s the script I’m using:

while ($true) {
    # Use WMI/CIM to get ProcessId and ExecutablePath for precise filtering
    $targets = Get-CimInstance Win32_Process -Filter "Name = 'ollama.exe'"

    foreach ($p in $targets) {
        try {
            # Log what will be killed (PID + path) for traceability
            Write-Host "Killing PID $($p.ProcessId) at $($p.ExecutablePath)"
            Stop-Process -Id $p.ProcessId -Force -ErrorAction Stop
        }
        catch {
            Write-Warning "Failed to kill PID $($p.ProcessId): $($_.Exception.Message)"
        }
    }

    Start-Sleep -Seconds 900 
}

After the first kill, it’s been working as expected for about 20 minutes now. So far it has force-terminated two processes, and the system has remained stable. Not sure if this helps others facing the same issue, but this approach seems more reliable than just finalizing the task manually.

It took 34 minutes, instead of the 10 minutes it was taking to fail again.

<!-- gh-comment-id:3226079774 --> @icariantk commented on GitHub (Aug 26, 2025): I wrote a small PowerShell script to automatically terminate the ollama.exe process every ~15 minutes. That interval matched the recurrence of the error I was seeing, and since running this script the issue has stabilized for longer compared to manually finalizing the task. Here’s the script I’m using: ``` while ($true) { # Use WMI/CIM to get ProcessId and ExecutablePath for precise filtering $targets = Get-CimInstance Win32_Process -Filter "Name = 'ollama.exe'" foreach ($p in $targets) { try { # Log what will be killed (PID + path) for traceability Write-Host "Killing PID $($p.ProcessId) at $($p.ExecutablePath)" Stop-Process -Id $p.ProcessId -Force -ErrorAction Stop } catch { Write-Warning "Failed to kill PID $($p.ProcessId): $($_.Exception.Message)" } } Start-Sleep -Seconds 900 } ``` After the first kill, it’s been working as expected for about 20 minutes now. So far it has force-terminated two processes, and the system has remained stable. Not sure if this helps others facing the same issue, but this approach seems more reliable than just finalizing the task manually. It took 34 minutes, instead of the 10 minutes it was taking to fail again.
GiteaMirror commented 2026-05-04 12:38:13 -05:00
Author
Owner
Copy Link
Copy Source

@icariantk commented on GitHub (Aug 29, 2025):

Just to be sure, It happen in windows server, linux server is fine:

Image
<!-- gh-comment-id:3238264334 --> @icariantk commented on GitHub (Aug 29, 2025): Just to be sure, It happen in windows server, linux server is fine: <img width="792" height="648" alt="Image" src="https://github.com/user-attachments/assets/fed8b4f3-8c47-4023-bc02-334c29029d1d" />
GiteaMirror commented 2026-05-04 12:38:14 -05:00
Author
Owner
Copy Link
Copy Source

@icariantk commented on GitHub (Sep 3, 2025):

Linux was running Only on CPU tho.

<!-- gh-comment-id:3247510640 --> @icariantk commented on GitHub (Sep 3, 2025): Linux was running Only on CPU tho.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
GiteaMirror ninjasurge (Kaehlen Rog)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#68142