github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #9140] Using MI50 multi-GPU to run DeepSeek-R1-Distill-Qwen-32B, the output results are all garbled. #31705

New Issue
Open
opened 2026-04-22 12:25:12 -05:00 by GiteaMirror · 1 comment
GiteaMirror commented 2026-04-22 12:25:12 -05:00
Owner
Copy Link

Originally created by @bbb4aaa on GitHub (Feb 15, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9140

What is the issue?

env:
multi mi50 gpus with rocm 6.2.4

bug:
The 32B model, loaded across two GPUs, generated entirely garbled responses to all queries. However, the 14B model, running on a single GPU, provided clear and coherent answers without any garbled text.

chat:

introduced , test improve -引进 super test - performance test improvement place test super - Test et identical testa - - entirely打了 , test test test then test Test test test super - results et hind 
similar et performance overall test choose et completely渔 test test test extra test , results overall,, identical adjust , test引进 received identical performance - - test equal test test test introduced 
test渔 improvement , compared et compared test superGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG

>>> hello
 .GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG

>>> 324324
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG

>>> 
>>> 35325325
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG

>>> Send a message (/? for help)

Relevant log output

2月 11 20:47:52 debian ollama[99009]: 2025/02/11 20:47:52 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PR>
2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.009+08:00 level=INFO source=images.go:432 msg="total blobs: 0"
2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.009+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
2月 11 20:47:52 debian ollama[99009]:  - using env:        export GIN_MODE=release
2月 11 20:47:52 debian ollama[99009]:  - using code:        gin.SetMode(gin.ReleaseMode)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST   /api/pull                 --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST   /api/generate             --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST   /api/chat                 --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST   /api/embed                --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST   /api/embeddings           --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST   /api/create               --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST   /api/push                 --> github.com/ollama/ollama/server.(*Server).PushHandler-fm (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST   /api/copy                 --> github.com/ollama/ollama/server.(*Server).CopyHandler-fm (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] DELETE /api/delete               --> github.com/ollama/ollama/server.(*Server).DeleteHandler-fm (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST   /api/show                 --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST   /api/blobs/:digest        --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] HEAD   /api/blobs/:digest        --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] GET    /api/ps                   --> github.com/ollama/ollama/server.(*Server).PsHandler-fm (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST   /v1/chat/completions      --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST   /v1/completions           --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (6 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST   /v1/embeddings            --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (6 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] GET    /v1/models                --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (6 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] GET    /v1/models/:model         --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (6 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] GET    /                         --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] GET    /api/tags                 --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] GET    /api/version              --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] HEAD   /                         --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] HEAD   /api/tags                 --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
2月 11 20:47:52 debian ollama[99009]: [GIN-debug] HEAD   /api/version              --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.010+08:00 level=INFO source=routes.go:1238 msg="Listening on 127.0.0.1:11434 (version 0.5.7)"
2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.010+08:00 level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cuda_v11_avx cuda_v12_avx rocm_avx cpu cpu_avx cpu_avx2]"
2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.010+08:00 level=INFO source=gpu.go:226 msg="looking for compatible GPUs"
2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.038+08:00 level=WARN source=amd_linux.go:61 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version f>
2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.041+08:00 level=INFO source=amd_linux.go:388 msg="amdgpu is supported" gpu=GPU-fd6a30617326b1e1 gpu_type=gfx906
2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.041+08:00 level=INFO source=amd_linux.go:388 msg="amdgpu is supported" gpu=GPU-3cc4394173497dfb gpu_type=gfx906
2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.041+08:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-fd6a30617326b1e1 library=rocm variant="" compute=gfx906 driver=0.0 name=1002>
2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.041+08:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-3cc4394173497dfb library=rocm variant="" compute=gfx906 driver=0.0 name=1002>

OS

Linux

GPU

AMD

CPU

Intel

Ollama version

0.5.11

Originally created by @bbb4aaa on GitHub (Feb 15, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9140 ### What is the issue? env: multi mi50 gpus with rocm 6.2.4 bug: The 32B model, loaded across two GPUs, generated entirely garbled responses to all queries. However, the 14B model, running on a single GPU, provided clear and coherent answers without any garbled text. chat: ``` introduced , test improve -引进 super test - performance test improvement place test super - Test et identical testa - - entirely打了 , test test test then test Test test test super - results et hind similar et performance overall test choose et completely渔 test test test extra test , results overall,, identical adjust , test引进 received identical performance - - test equal test test test introduced test渔 improvement , compared et compared test superGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG >>> hello .GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG >>> 324324 GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG >>> >>> 35325325 GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG >>> Send a message (/? for help) ``` ### Relevant log output ```shell 2月 11 20:47:52 debian ollama[99009]: 2025/02/11 20:47:52 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PR> 2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.009+08:00 level=INFO source=images.go:432 msg="total blobs: 0" 2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.009+08:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0" 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached. 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production. 2月 11 20:47:52 debian ollama[99009]: - using env: export GIN_MODE=release 2月 11 20:47:52 debian ollama[99009]: - using code: gin.SetMode(gin.ReleaseMode) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST /api/pull --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST /api/generate --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST /api/chat --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST /api/embed --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST /api/embeddings --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST /api/create --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST /api/push --> github.com/ollama/ollama/server.(*Server).PushHandler-fm (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST /api/copy --> github.com/ollama/ollama/server.(*Server).CopyHandler-fm (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] DELETE /api/delete --> github.com/ollama/ollama/server.(*Server).DeleteHandler-fm (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST /api/show --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] HEAD /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] GET /api/ps --> github.com/ollama/ollama/server.(*Server).PsHandler-fm (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST /v1/chat/completions --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST /v1/completions --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (6 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] POST /v1/embeddings --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (6 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] GET /v1/models --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (6 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] GET /v1/models/:model --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (6 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] GET / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] GET /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] GET /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] HEAD / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] HEAD /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers) 2月 11 20:47:52 debian ollama[99009]: [GIN-debug] HEAD /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers) 2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.010+08:00 level=INFO source=routes.go:1238 msg="Listening on 127.0.0.1:11434 (version 0.5.7)" 2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.010+08:00 level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cuda_v11_avx cuda_v12_avx rocm_avx cpu cpu_avx cpu_avx2]" 2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.010+08:00 level=INFO source=gpu.go:226 msg="looking for compatible GPUs" 2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.038+08:00 level=WARN source=amd_linux.go:61 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version f> 2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.041+08:00 level=INFO source=amd_linux.go:388 msg="amdgpu is supported" gpu=GPU-fd6a30617326b1e1 gpu_type=gfx906 2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.041+08:00 level=INFO source=amd_linux.go:388 msg="amdgpu is supported" gpu=GPU-3cc4394173497dfb gpu_type=gfx906 2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.041+08:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-fd6a30617326b1e1 library=rocm variant="" compute=gfx906 driver=0.0 name=1002> 2月 11 20:47:52 debian ollama[99009]: time=2025-02-11T20:47:52.041+08:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-3cc4394173497dfb library=rocm variant="" compute=gfx906 driver=0.0 name=1002> ``` ### OS Linux ### GPU AMD ### CPU Intel ### Ollama version 0.5.11
GiteaMirror added the bug label 2026-04-22 12:25:12 -05:00
GiteaMirror commented 2026-04-22 12:25:13 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 15, 2025):

Sounds a lot like #8843: multi-gpu+rocm=bad output.

<!-- gh-comment-id:2661030124 --> @rick-github commented on GitHub (Feb 15, 2025): Sounds a lot like #8843: multi-gpu+rocm=bad output.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#31705
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?