github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-17 05:36:16 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #15975] mistral-medium-3.5 - Produces nonsense outputs #87861

New Issue
Closed
opened 2026-05-10 06:27:34 -05:00 by GiteaMirror · 19 comments
GiteaMirror commented 2026-05-10 06:27:34 -05:00
Owner
Copy Link

Originally created by @Notbici on GitHub (May 5, 2026).
Original GitHub issue: https://github.com/ollama/ollama/issues/15975

Originally assigned to: @pdevine on GitHub.

What is the issue?

Hi,

I'm running mistral-medium-3.5 on two RTX 6000 Pro Blackwell cards, when I execute mistral-medium-3.5 via Ollama it just produces nonsense, is this normal?

Image

I've not touched any settings so the replication was ollama pulling the model and then trying to chat with it.

Relevant log output

time=2026-05-04T22:22:40.276-08:00 level=INFO source=routes.go:1782 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GGML_VK_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:0 OLLAMA_DEBUG:INFO OLLAMA_DEBUG_LOG_REQUESTS:false OLLAMA_EDITOR: OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://x.x.x.x:11434 OLLAMA_KEEP_ALIVE:1h0m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:2 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NO_CLOUD:false OLLAMA_NUM_PARALLEL:2 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false OLLAMA_VULKAN:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2026-05-04T22:22:40.276-08:00 level=INFO source=routes.go:1784 msg="Ollama cloud disabled: false"
time=2026-05-04T22:22:40.277-08:00 level=INFO source=images.go:517 msg="total blobs: 17"
time=2026-05-04T22:22:40.277-08:00 level=INFO source=images.go:524 msg="total unused blobs removed: 0"
time=2026-05-04T22:22:40.277-08:00 level=INFO source=routes.go:1847 msg="Listening on x.x.x.x:11434 (version 0.22.1)"
time=2026-05-04T22:22:40.278-08:00 level=INFO source=runner.go:67 msg="discovering available GPUs..."
time=2026-05-04T22:22:40.278-08:00 level=INFO source=server.go:444 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 39131"
time=2026-05-04T22:22:40.627-08:00 level=INFO source=model_recommendations.go:179 msg="model recommendations cache sleep scheduled" wait=4h3m27.097566603s consecutive_failures=0
time=2026-05-04T22:22:40.826-08:00 level=INFO source=server.go:444 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 45915"
time=2026-05-04T22:22:41.233-08:00 level=INFO source=runner.go:106 msg="experimental Vulkan support disabled.  To enable, set OLLAMA_VULKAN=1"
time=2026-05-04T22:22:41.233-08:00 level=INFO source=server.go:444 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 43089"
time=2026-05-04T22:22:41.233-08:00 level=INFO source=server.go:444 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 37561"
time=2026-05-04T22:22:41.233-08:00 level=INFO source=server.go:444 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 44395"
time=2026-05-04T22:22:41.233-08:00 level=INFO source=server.go:444 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 42687"

time=2026-05-04T22:22:41.552-08:00 level=INFO source=types.go:42 msg="inference compute" id=GPU-ae83d484-92ff-7cf4-997d-451d2e09088c filter_id="" library=CUDA compute=12.0 name=CUDA0 description="NVIDIA RTX PRO 6000 Blackwell Server Edition" libdirs=ollama,cuda_v13 driver=13.2 pci_id=0000:41:00.0 type=discrete total="95.6 GiB" available="95.0 GiB"
time=2026-05-04T22:22:41.553-08:00 level=INFO source=types.go:42 msg="inference compute" id=GPU-aa456d0a-bd98-e1c3-17ca-c94406852129 filter_id="" library=CUDA compute=12.0 name=CUDA1 description="NVIDIA RTX PRO 6000 Blackwell Server Edition" libdirs=ollama,cuda_v13 driver=13.2 pci_id=0000:61:00.0 type=discrete total="95.6 GiB" available="94.4 GiB"
time=2026-05-04T22:22:41.553-08:00 level=INFO source=routes.go:1897 msg="vram-based default context" total_vram="191.2 GiB" default_num_ctx=262144







time=2026-05-04T22:22:57.990-08:00 level=INFO source=server.go:444 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 34987"
time=2026-05-04T22:22:58.469-08:00 level=WARN source=cpu_linux.go:130 msg="failed to parse CPU allowed micro secs" error="strconv.ParseInt: parsing \"max\": invalid syntax"
time=2026-05-04T22:22:58.553-08:00 level=INFO source=server.go:259 msg="enabling flash attention"
time=2026-05-04T22:22:58.553-08:00 level=INFO source=server.go:444 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --model /root/.ollama/models/blobs/sha256-f68f582c8d0c0c6a2597331e51d141be71262ee3397bfba97cb03b695bd31499 --port 36683"
time=2026-05-04T22:22:58.554-08:00 level=INFO source=sched.go:484 msg="system memory" total="251.5 GiB" free="156.6 GiB" free_swap="0 B"
time=2026-05-04T22:22:58.554-08:00 level=INFO source=sched.go:491 msg="gpu memory" id=GPU-ae83d484-92ff-7cf4-997d-451d2e09088c library=CUDA available="94.5 GiB" free="95.0 GiB" minimum="457.0 MiB" overhead="0 B"
time=2026-05-04T22:22:58.554-08:00 level=INFO source=sched.go:491 msg="gpu memory" id=GPU-aa456d0a-bd98-e1c3-17ca-c94406852129 library=CUDA available="94.0 GiB" free="94.4 GiB" minimum="457.0 MiB" overhead="0 B"
time=2026-05-04T22:22:58.554-08:00 level=INFO source=server.go:771 msg="loading model" "model layers"=89 requested=-1
time=2026-05-04T22:22:58.565-08:00 level=INFO source=runner.go:1417 msg="starting ollama engine"
time=2026-05-04T22:22:58.566-08:00 level=INFO source=runner.go:1452 msg="Server listening on 127.0.0.1:36683"
time=2026-05-04T22:22:58.575-08:00 level=INFO source=runner.go:1290 msg=load request="{Operation:fit LoraPath:[] Parallel:2 BatchSize:512 FlashAttention:Enabled KvSize:524288 KvCacheType: NumThreads:16 GPULayers:89[ID:GPU-ae83d484-92ff-7cf4-997d-451d2e09088c Layers:89(0..88)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2026-05-04T22:22:58.606-08:00 level=INFO source=ggml.go:136 msg="" architecture=mistral3 file_type=Q4_K_M name="" description="" num_tensors=1233 num_key_values=51
load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA RTX PRO 6000 Blackwell Server Edition, compute capability 12.0, VMM: yes, ID: GPU-ae83d484-92ff-7cf4-997d-451d2e09088c
  Device 1: NVIDIA RTX PRO 6000 Blackwell Server Edition, compute capability 12.0, VMM: yes, ID: GPU-aa456d0a-bd98-e1c3-17ca-c94406852129
load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v13/libggml-cuda.so
time=2026-05-04T22:22:58.845-08:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=750,800,860,870,890,900,1000,1030,1100,1200,1210 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=750,800,860,870,890,900,1000,1030,1100,1200,1210 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
time=2026-05-04T22:23:00.223-08:00 level=INFO source=runner.go:1290 msg=load request="{Operation:fit LoraPath:[] Parallel:2 BatchSize:512 FlashAttention:Enabled KvSize:524288 KvCacheType: NumThreads:16 GPULayers:66[ID:GPU-ae83d484-92ff-7cf4-997d-451d2e09088c Layers:33(22..54) ID:GPU-aa456d0a-bd98-e1c3-17ca-c94406852129 Layers:33(55..87)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2026-05-04T22:23:01.391-08:00 level=INFO source=runner.go:1290 msg=load request="{Operation:fit LoraPath:[] Parallel:2 BatchSize:512 FlashAttention:Enabled KvSize:524288 KvCacheType: NumThreads:16 GPULayers:65[ID:GPU-ae83d484-92ff-7cf4-997d-451d2e09088c Layers:32(23..54) ID:GPU-aa456d0a-bd98-e1c3-17ca-c94406852129 Layers:33(55..87)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2026-05-04T22:23:02.413-08:00 level=INFO source=runner.go:1290 msg=load request="{Operation:alloc LoraPath:[] Parallel:2 BatchSize:512 FlashAttention:Enabled KvSize:524288 KvCacheType: NumThreads:16 GPULayers:65[ID:GPU-ae83d484-92ff-7cf4-997d-451d2e09088c Layers:32(23..54) ID:GPU-aa456d0a-bd98-e1c3-17ca-c94406852129 Layers:33(55..87)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2026-05-04T22:23:19.478-08:00 level=INFO source=runner.go:1290 msg=load request="{Operation:commit LoraPath:[] Parallel:2 BatchSize:512 FlashAttention:Enabled KvSize:524288 KvCacheType: NumThreads:16 GPULayers:65[ID:GPU-ae83d484-92ff-7cf4-997d-451d2e09088c Layers:32(23..54) ID:GPU-aa456d0a-bd98-e1c3-17ca-c94406852129 Layers:33(55..87)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2026-05-04T22:23:19.478-08:00 level=INFO source=ggml.go:482 msg="offloading 65 repeating layers to GPU"
time=2026-05-04T22:23:19.478-08:00 level=INFO source=ggml.go:486 msg="offloading output layer to CPU"
time=2026-05-04T22:23:19.478-08:00 level=INFO source=ggml.go:494 msg="offloaded 65/89 layers to GPU"
time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:240 msg="model weights" device=CUDA0 size="24.1 GiB"
time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:240 msg="model weights" device=CUDA1 size="25.6 GiB"
time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:245 msg="model weights" device=CPU size="25.1 GiB"
time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:251 msg="kv cache" device=CUDA0 size="64.0 GiB"
time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:251 msg="kv cache" device=CUDA1 size="66.0 GiB"
time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:256 msg="kv cache" device=CPU size="46.0 GiB"
time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:262 msg="compute graph" device=CUDA0 size="3.8 GiB"
time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:262 msg="compute graph" device=CUDA1 size="826.0 MiB"
time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:267 msg="compute graph" device=CPU size="307.2 MiB"
time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:272 msg="total memory" size="255.6 GiB"
time=2026-05-04T22:23:19.478-08:00 level=INFO source=sched.go:561 msg="loaded runners" count=1
time=2026-05-04T22:23:19.478-08:00 level=INFO source=server.go:1364 msg="waiting for llama runner to start responding"
time=2026-05-04T22:23:19.479-08:00 level=INFO source=server.go:1398 msg="waiting for server to become available" status="llm server loading model"
time=2026-05-04T22:23:27.251-08:00 level=INFO source=server.go:1402 msg="llama runner started in 28.70 seconds"

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

0.22.1

Originally created by @Notbici on GitHub (May 5, 2026). Original GitHub issue: https://github.com/ollama/ollama/issues/15975 Originally assigned to: @pdevine on GitHub. ### What is the issue? Hi, I'm running mistral-medium-3.5 on two RTX 6000 Pro Blackwell cards, when I execute mistral-medium-3.5 via Ollama it just produces nonsense, is this normal? <img width="758" height="186" alt="Image" src="https://github.com/user-attachments/assets/64fc9259-6a44-40da-8ce6-e38622a60fc6" /> I've not touched any settings so the replication was ollama pulling the model and then trying to chat with it. ### Relevant log output ```shell time=2026-05-04T22:22:40.276-08:00 level=INFO source=routes.go:1782 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GGML_VK_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:0 OLLAMA_DEBUG:INFO OLLAMA_DEBUG_LOG_REQUESTS:false OLLAMA_EDITOR: OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://x.x.x.x:11434 OLLAMA_KEEP_ALIVE:1h0m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:2 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NO_CLOUD:false OLLAMA_NUM_PARALLEL:2 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false OLLAMA_VULKAN:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2026-05-04T22:22:40.276-08:00 level=INFO source=routes.go:1784 msg="Ollama cloud disabled: false" time=2026-05-04T22:22:40.277-08:00 level=INFO source=images.go:517 msg="total blobs: 17" time=2026-05-04T22:22:40.277-08:00 level=INFO source=images.go:524 msg="total unused blobs removed: 0" time=2026-05-04T22:22:40.277-08:00 level=INFO source=routes.go:1847 msg="Listening on x.x.x.x:11434 (version 0.22.1)" time=2026-05-04T22:22:40.278-08:00 level=INFO source=runner.go:67 msg="discovering available GPUs..." time=2026-05-04T22:22:40.278-08:00 level=INFO source=server.go:444 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 39131" time=2026-05-04T22:22:40.627-08:00 level=INFO source=model_recommendations.go:179 msg="model recommendations cache sleep scheduled" wait=4h3m27.097566603s consecutive_failures=0 time=2026-05-04T22:22:40.826-08:00 level=INFO source=server.go:444 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 45915" time=2026-05-04T22:22:41.233-08:00 level=INFO source=runner.go:106 msg="experimental Vulkan support disabled. To enable, set OLLAMA_VULKAN=1" time=2026-05-04T22:22:41.233-08:00 level=INFO source=server.go:444 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 43089" time=2026-05-04T22:22:41.233-08:00 level=INFO source=server.go:444 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 37561" time=2026-05-04T22:22:41.233-08:00 level=INFO source=server.go:444 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 44395" time=2026-05-04T22:22:41.233-08:00 level=INFO source=server.go:444 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 42687" time=2026-05-04T22:22:41.552-08:00 level=INFO source=types.go:42 msg="inference compute" id=GPU-ae83d484-92ff-7cf4-997d-451d2e09088c filter_id="" library=CUDA compute=12.0 name=CUDA0 description="NVIDIA RTX PRO 6000 Blackwell Server Edition" libdirs=ollama,cuda_v13 driver=13.2 pci_id=0000:41:00.0 type=discrete total="95.6 GiB" available="95.0 GiB" time=2026-05-04T22:22:41.553-08:00 level=INFO source=types.go:42 msg="inference compute" id=GPU-aa456d0a-bd98-e1c3-17ca-c94406852129 filter_id="" library=CUDA compute=12.0 name=CUDA1 description="NVIDIA RTX PRO 6000 Blackwell Server Edition" libdirs=ollama,cuda_v13 driver=13.2 pci_id=0000:61:00.0 type=discrete total="95.6 GiB" available="94.4 GiB" time=2026-05-04T22:22:41.553-08:00 level=INFO source=routes.go:1897 msg="vram-based default context" total_vram="191.2 GiB" default_num_ctx=262144 time=2026-05-04T22:22:57.990-08:00 level=INFO source=server.go:444 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 34987" time=2026-05-04T22:22:58.469-08:00 level=WARN source=cpu_linux.go:130 msg="failed to parse CPU allowed micro secs" error="strconv.ParseInt: parsing \"max\": invalid syntax" time=2026-05-04T22:22:58.553-08:00 level=INFO source=server.go:259 msg="enabling flash attention" time=2026-05-04T22:22:58.553-08:00 level=INFO source=server.go:444 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --model /root/.ollama/models/blobs/sha256-f68f582c8d0c0c6a2597331e51d141be71262ee3397bfba97cb03b695bd31499 --port 36683" time=2026-05-04T22:22:58.554-08:00 level=INFO source=sched.go:484 msg="system memory" total="251.5 GiB" free="156.6 GiB" free_swap="0 B" time=2026-05-04T22:22:58.554-08:00 level=INFO source=sched.go:491 msg="gpu memory" id=GPU-ae83d484-92ff-7cf4-997d-451d2e09088c library=CUDA available="94.5 GiB" free="95.0 GiB" minimum="457.0 MiB" overhead="0 B" time=2026-05-04T22:22:58.554-08:00 level=INFO source=sched.go:491 msg="gpu memory" id=GPU-aa456d0a-bd98-e1c3-17ca-c94406852129 library=CUDA available="94.0 GiB" free="94.4 GiB" minimum="457.0 MiB" overhead="0 B" time=2026-05-04T22:22:58.554-08:00 level=INFO source=server.go:771 msg="loading model" "model layers"=89 requested=-1 time=2026-05-04T22:22:58.565-08:00 level=INFO source=runner.go:1417 msg="starting ollama engine" time=2026-05-04T22:22:58.566-08:00 level=INFO source=runner.go:1452 msg="Server listening on 127.0.0.1:36683" time=2026-05-04T22:22:58.575-08:00 level=INFO source=runner.go:1290 msg=load request="{Operation:fit LoraPath:[] Parallel:2 BatchSize:512 FlashAttention:Enabled KvSize:524288 KvCacheType: NumThreads:16 GPULayers:89[ID:GPU-ae83d484-92ff-7cf4-997d-451d2e09088c Layers:89(0..88)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2026-05-04T22:22:58.606-08:00 level=INFO source=ggml.go:136 msg="" architecture=mistral3 file_type=Q4_K_M name="" description="" num_tensors=1233 num_key_values=51 load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 2 CUDA devices: Device 0: NVIDIA RTX PRO 6000 Blackwell Server Edition, compute capability 12.0, VMM: yes, ID: GPU-ae83d484-92ff-7cf4-997d-451d2e09088c Device 1: NVIDIA RTX PRO 6000 Blackwell Server Edition, compute capability 12.0, VMM: yes, ID: GPU-aa456d0a-bd98-e1c3-17ca-c94406852129 load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v13/libggml-cuda.so time=2026-05-04T22:22:58.845-08:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=750,800,860,870,890,900,1000,1030,1100,1200,1210 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=750,800,860,870,890,900,1000,1030,1100,1200,1210 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) time=2026-05-04T22:23:00.223-08:00 level=INFO source=runner.go:1290 msg=load request="{Operation:fit LoraPath:[] Parallel:2 BatchSize:512 FlashAttention:Enabled KvSize:524288 KvCacheType: NumThreads:16 GPULayers:66[ID:GPU-ae83d484-92ff-7cf4-997d-451d2e09088c Layers:33(22..54) ID:GPU-aa456d0a-bd98-e1c3-17ca-c94406852129 Layers:33(55..87)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2026-05-04T22:23:01.391-08:00 level=INFO source=runner.go:1290 msg=load request="{Operation:fit LoraPath:[] Parallel:2 BatchSize:512 FlashAttention:Enabled KvSize:524288 KvCacheType: NumThreads:16 GPULayers:65[ID:GPU-ae83d484-92ff-7cf4-997d-451d2e09088c Layers:32(23..54) ID:GPU-aa456d0a-bd98-e1c3-17ca-c94406852129 Layers:33(55..87)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2026-05-04T22:23:02.413-08:00 level=INFO source=runner.go:1290 msg=load request="{Operation:alloc LoraPath:[] Parallel:2 BatchSize:512 FlashAttention:Enabled KvSize:524288 KvCacheType: NumThreads:16 GPULayers:65[ID:GPU-ae83d484-92ff-7cf4-997d-451d2e09088c Layers:32(23..54) ID:GPU-aa456d0a-bd98-e1c3-17ca-c94406852129 Layers:33(55..87)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2026-05-04T22:23:19.478-08:00 level=INFO source=runner.go:1290 msg=load request="{Operation:commit LoraPath:[] Parallel:2 BatchSize:512 FlashAttention:Enabled KvSize:524288 KvCacheType: NumThreads:16 GPULayers:65[ID:GPU-ae83d484-92ff-7cf4-997d-451d2e09088c Layers:32(23..54) ID:GPU-aa456d0a-bd98-e1c3-17ca-c94406852129 Layers:33(55..87)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2026-05-04T22:23:19.478-08:00 level=INFO source=ggml.go:482 msg="offloading 65 repeating layers to GPU" time=2026-05-04T22:23:19.478-08:00 level=INFO source=ggml.go:486 msg="offloading output layer to CPU" time=2026-05-04T22:23:19.478-08:00 level=INFO source=ggml.go:494 msg="offloaded 65/89 layers to GPU" time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:240 msg="model weights" device=CUDA0 size="24.1 GiB" time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:240 msg="model weights" device=CUDA1 size="25.6 GiB" time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:245 msg="model weights" device=CPU size="25.1 GiB" time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:251 msg="kv cache" device=CUDA0 size="64.0 GiB" time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:251 msg="kv cache" device=CUDA1 size="66.0 GiB" time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:256 msg="kv cache" device=CPU size="46.0 GiB" time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:262 msg="compute graph" device=CUDA0 size="3.8 GiB" time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:262 msg="compute graph" device=CUDA1 size="826.0 MiB" time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:267 msg="compute graph" device=CPU size="307.2 MiB" time=2026-05-04T22:23:19.478-08:00 level=INFO source=device.go:272 msg="total memory" size="255.6 GiB" time=2026-05-04T22:23:19.478-08:00 level=INFO source=sched.go:561 msg="loaded runners" count=1 time=2026-05-04T22:23:19.478-08:00 level=INFO source=server.go:1364 msg="waiting for llama runner to start responding" time=2026-05-04T22:23:19.479-08:00 level=INFO source=server.go:1398 msg="waiting for server to become available" status="llm server loading model" time=2026-05-04T22:23:27.251-08:00 level=INFO source=server.go:1402 msg="llama runner started in 28.70 seconds" ``` ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version 0.22.1
GiteaMirror added the bug label 2026-05-10 06:27:34 -05:00
GiteaMirror commented 2026-05-10 06:27:36 -05:00
Author
Owner
Copy Link

@Notbici commented on GitHub (May 5, 2026):

Also lowering the context to 50k does reduce memory per GPU down, but all it does is speed up the gibberish output.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.58.03              Driver Version: 595.58.03      CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX PRO 6000 Blac...    On  |   00000000:41:00.0 Off |                    0 |
| N/A   51C    P0             88W /  600W |   57913MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA RTX PRO 6000 Blac...    On  |   00000000:61:00.0 Off |                    0 |
| N/A   52C    P0             88W /  600W |   58181MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

Chat:

me: hi

mistral-medium-3.5:128b:
"create_tasks, "parameters: {"properties: {"description: "the ID of the event to delete, "type: "string"}}, "required: ["event_id], "type: "object"}}, "type: "object"}}, "type: "function(), {"function: {"description: "create a new task, "parameters: {"properties: {"description: "The task description, "type: "string"}, "name: {"description: "the task description, "type: "string, "content: {"description: "event title, "type: "string, "required: ["event_id], "type: "object"}}, "type: "function(), {"function: {"description: "new task,
<!-- gh-comment-id:4376973833 --> @Notbici commented on GitHub (May 5, 2026): Also lowering the context to 50k does reduce memory per GPU down, but all it does is speed up the gibberish output. ``` +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 595.58.03 Driver Version: 595.58.03 CUDA Version: 13.2 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA RTX PRO 6000 Blac... On | 00000000:41:00.0 Off | 0 | | N/A 51C P0 88W / 600W | 57913MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA RTX PRO 6000 Blac... On | 00000000:61:00.0 Off | 0 | | N/A 52C P0 88W / 600W | 58181MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ ``` Chat: ``` me: hi mistral-medium-3.5:128b: "create_tasks, "parameters: {"properties: {"description: "the ID of the event to delete, "type: "string"}}, "required: ["event_id], "type: "object"}}, "type: "object"}}, "type: "function(), {"function: {"description: "create a new task, "parameters: {"properties: {"description: "The task description, "type: "string"}, "name: {"description: "the task description, "type: "string, "content: {"description: "event title, "type: "string, "required: ["event_id], "type: "object"}}, "type: "function(), {"function: {"description: "new task, ```
GiteaMirror commented 2026-05-10 06:27:37 -05:00
Author
Owner
Copy Link

@fcorneli commented on GitHub (May 5, 2026):

I'm experiencing the same issue. In OpenCode I get the following gibberish:

hi
> a type = "003e = a list of the type = "The class = "type: "5= h, type: "4 = m and use the same type of the type

When running via llama.cpp, it works for the model: unsloth/Mistral-Medium-3.5-128B-GGUF:Q4_K_XL as expected.

<!-- gh-comment-id:4376995806 --> @fcorneli commented on GitHub (May 5, 2026): I'm experiencing the same issue. In OpenCode I get the following gibberish: ``` hi > a type = "003e = a list of the type = "The class = "type: "5= h, type: "4 = m and use the same type of the type ``` When running via llama.cpp, it works for the model: unsloth/Mistral-Medium-3.5-128B-GGUF:Q4_K_XL as expected.
GiteaMirror commented 2026-05-10 06:27:37 -05:00
Author
Owner
Copy Link

@pereswa commented on GitHub (May 5, 2026):

I can confirm that mistral-medium-3.5:128b returns nearly useless and repetetive texts.
I use it to generate German texts and these are mixed with English words and have bad grammar like generated by an old 4b model (starting from the first prompt).
On the other hand mistral-large:123b produces very useful results on the same system.

Something seems to be wrong with the provided mistral-medium-3.5:128b or its processing by Ollama.

System: Ollama 0.22.1, Windows 10, AMD Ryzen 5 7640HS, 96GB RAM, RTX4060 8GB - yes, those huge models are a pain on this system :D

<!-- gh-comment-id:4379953256 --> @pereswa commented on GitHub (May 5, 2026): I can confirm that mistral-medium-3.5:128b returns nearly useless and repetetive texts. I use it to generate German texts and these are mixed with English words and have bad grammar like generated by an old 4b model (starting from the first prompt). On the other hand mistral-large:123b produces very useful results on the same system. Something seems to be wrong with the provided mistral-medium-3.5:128b or its processing by Ollama. System: Ollama 0.22.1, Windows 10, AMD Ryzen 5 7640HS, 96GB RAM, RTX4060 8GB - yes, those huge models are a pain on this system :D
GiteaMirror commented 2026-05-10 06:27:38 -05:00
Author
Owner
Copy Link

@fcorneli commented on GitHub (May 5, 2026):

Strange thing is, when running it as follows:

ollama run mistral-medium-3.5:128b-q4_K_M "Explain E=mc^2" --verbose

the output makes sense. It's only when trying to use it via agents like OpenCode that the nonsense output takes place.

<!-- gh-comment-id:4380934622 --> @fcorneli commented on GitHub (May 5, 2026): Strange thing is, when running it as follows: ``` ollama run mistral-medium-3.5:128b-q4_K_M "Explain E=mc^2" --verbose ``` the output makes sense. It's only when trying to use it via agents like OpenCode that the nonsense output takes place.
GiteaMirror commented 2026-05-10 06:27:38 -05:00
Author
Owner
Copy Link

@pereswa commented on GitHub (May 5, 2026):

I tested it: Yes, that prompt produces a short useful answer.

I tested a bit with requesting explanations in German - the response deteriorates after some useful paragraphs.
First, I thought it is an issue with internal translation (like to German) but the issue occurs in english too.

Try "Explain E=mc^2 in detail." -> as soon as the model has to create some more text it gets worse with every new line.
It just shows faster when requesting an answer in an non-English language.

<!-- gh-comment-id:4382011004 --> @pereswa commented on GitHub (May 5, 2026): I tested it: Yes, that prompt produces a short useful answer. I tested a bit with requesting explanations in German - the response deteriorates after some useful paragraphs. First, I thought it is an issue with internal translation (like to German) but the issue occurs in english too. Try "Explain E=mc^2 in detail." -> as soon as the model has to create some more text it gets worse with every new line. It just shows faster when requesting an answer in an non-English language.
GiteaMirror commented 2026-05-10 06:27:39 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (May 5, 2026):

I believe there is an issue w/ long context degradation that Mistral uncovered. I'm taking a look to see if there's something obviously wrong/missing in the Ollama implementation.

<!-- gh-comment-id:4382017338 --> @pdevine commented on GitHub (May 5, 2026): I believe there is an issue w/ long context degradation that Mistral uncovered. I'm taking a look to see if there's something obviously wrong/missing in the Ollama implementation.
GiteaMirror commented 2026-05-10 06:27:40 -05:00
Author
Owner
Copy Link

@fcorneli commented on GitHub (May 5, 2026):

Yep, I can confirm that:

ollama run mistral-medium-3.5:128b-q4_K_M "Explain E=mc^2 in detail." --verbose

results in nonsense after a while:

...
+ **Consoration = **E = mc²** is the **total energy** of a proton
+ **consoration = **E = mc²** is the **total energy** of a proton
+ **Consoration = **E = mc²** is the **total energy** of a proton
+ **Consoration = **E = mc²** is the **total energy** of a proton
+ **consoration = **E = mc²** is the **total energy** of a proton
<!-- gh-comment-id:4382055927 --> @fcorneli commented on GitHub (May 5, 2026): Yep, I can confirm that: ``` ollama run mistral-medium-3.5:128b-q4_K_M "Explain E=mc^2 in detail." --verbose ``` results in nonsense after a while: ``` ... + **Consoration = **E = mc²** is the **total energy** of a proton + **consoration = **E = mc²** is the **total energy** of a proton + **Consoration = **E = mc²** is the **total energy** of a proton + **Consoration = **E = mc²** is the **total energy** of a proton + **consoration = **E = mc²** is the **total energy** of a proton ```
GiteaMirror commented 2026-05-10 06:27:41 -05:00
Author
Owner
Copy Link

@pereswa commented on GitHub (May 5, 2026):

Example (System: Ollama 0.22.1, Windows 10, AMD Ryzen 5 7640HS, 96GB RAM, RTX4060 8GB):

Microsoft Windows [Version 10.0.19045.6036]
(c) Microsoft Corporation. Alle Rechte vorbehalten.

C:\Users\Netzuser>ollama run mistral-medium-3.5:128b
>>> Explain E=mc^2 in detail.
The equation **E = mc²** is one of the most famous equations in physics. It is **Einstein's mass-energy equivalence
principle**, formulated by Albert Einstein in 1905. This equation revolutionized our understanding of space, time,
energy, and matter. Below is a detailed breakdown of the equation, its meaning, and its implications.

---

---

### **1. The Equation: E = mc²**
- **E** = Energy (in joules)
- **m** = mass (in kilograms)
- **c** = speed of light in a vacuum (˜ 3 × 108 m/s)
- **c²** = (speed of light)²

---

---

### **2. Meaning of the Equation**
The equation states that **mass and energy are interchangeable**. This means:
- A small amount of mass can be converted into a **tremendous amount of energy, and vice versa.
- If an object is moving, its **relativistic mass increases, and thus its energy increases.

---

---

### **3. Derivation of E = mc²**
Einstein derived the equation using **special relativity** (a cornerstone of modern physics). Here’s a simplified
derivation:

1. Start with **F = ma** (force), where **F** is the four-force.
2. Assume an object of mass **m** is at rest. Work is done to move it, so its kinetic energy is ½mv².
3. Apply a force **F** over a distance **d**. The work done is **Fd**, so **Fd = Fd**.
   - Work = **Fd** = **F** (from step 2).
   - Thus, **F = mc²**.

This derivation relies on two key postulates of special relativity:
1. **Mass-energy equivalence**: The law of conservation of mass-energy.
2. **Speed of light (c) is constant in a vacuum.

---

---

### **4. Implications of E = mc²**
The equation has far-reaching implications across physics, technology, and everyday life. Here are some key examples:

---

#### **Nuclear Physics**
- **Nuclear fission**: The equation explains that mass and energy are interchangeable. This is the foundation of **E =
mc²**, which is used in nuclear physics ( the photoelectric effect.
- **Atomic splitting**: The equation explains that a small amount of mass can be converted into a tremendous amount of
energy ( **E = mc²**. For example, splitting a uranium battery (1 kg into 4.4 jou, the energy released is **E = (1 kg ×
(108 J) = 9 × 10¹6 J.

- **Cosmology**: The equation explains that rest mass can be converted into energy. For example, in a battery, the total
rest mass of the universe can be converted into energy ( **E = mc².

- **Gravravity**: The equation predicts that nothing can travel faster or faster than the speed of light. For example,
in a vacuum, light cannot have mass, so it can’t move.
- **Grav force**: the equation is used to design particle like particle acceler and particle acceler.

- **Quantuclear applications**
- **Nuclear physics**: The equation is used to explain the behavior of light. For example:
  - **Photoelectric effect**: The equation explains that light behaves both particle and wave properties.
- **Compton effect**: the equation is used to explain the wave-particle nature of light.
- **Grav effect**: the equation is used to calculate the rest mass of objects, such as planets, cars, and planets.

- **Energy from mc²**
  - **Kinetic energy of a photon: h? / ? ˜ 9.0856 × 10¹8 J
  - **1 kg × (1.6021766 × 10^15 J

---

---

### **5. Historical Context**
- **Einstein’s thought experiment ( Michel's famous thought experiment.
- **Michel's famous thought experiment ( Michel's famous thought experiment.
- **Michel's famous thought experiment = Michel's famous thought experiment.

- **Michel's famous thought experiment = Michel's famous thought experiment.

- **Einstein's famous thought experiment = **E = mc².

- **1 kg = c² / 3 × 10 m/s ˜ 3.0856 × 10^15 j

- **2 kg = c² / 3 × 10^15 j
- **3 kg = c² / 3 × 10^15 j
- **4 kg = c² / 3 × 10^15 j
- **5 kg = c² / 3 × 10^15 j

- **6 kg = c² / 3 × 10^15 j

- **7 kg = c²
(cancelled)

I liked: "For example, in a battery, the total rest mass of the universe can be converted into energy"

<!-- gh-comment-id:4382124979 --> @pereswa commented on GitHub (May 5, 2026): Example (System: Ollama 0.22.1, Windows 10, AMD Ryzen 5 7640HS, 96GB RAM, RTX4060 8GB): ``` Microsoft Windows [Version 10.0.19045.6036] (c) Microsoft Corporation. Alle Rechte vorbehalten. C:\Users\Netzuser>ollama run mistral-medium-3.5:128b >>> Explain E=mc^2 in detail. The equation **E = mc²** is one of the most famous equations in physics. It is **Einstein's mass-energy equivalence principle**, formulated by Albert Einstein in 1905. This equation revolutionized our understanding of space, time, energy, and matter. Below is a detailed breakdown of the equation, its meaning, and its implications. --- --- ### **1. The Equation: E = mc²** - **E** = Energy (in joules) - **m** = mass (in kilograms) - **c** = speed of light in a vacuum (˜ 3 × 108 m/s) - **c²** = (speed of light)² --- --- ### **2. Meaning of the Equation** The equation states that **mass and energy are interchangeable**. This means: - A small amount of mass can be converted into a **tremendous amount of energy, and vice versa. - If an object is moving, its **relativistic mass increases, and thus its energy increases. --- --- ### **3. Derivation of E = mc²** Einstein derived the equation using **special relativity** (a cornerstone of modern physics). Here’s a simplified derivation: 1. Start with **F = ma** (force), where **F** is the four-force. 2. Assume an object of mass **m** is at rest. Work is done to move it, so its kinetic energy is ½mv². 3. Apply a force **F** over a distance **d**. The work done is **Fd**, so **Fd = Fd**. - Work = **Fd** = **F** (from step 2). - Thus, **F = mc²**. This derivation relies on two key postulates of special relativity: 1. **Mass-energy equivalence**: The law of conservation of mass-energy. 2. **Speed of light (c) is constant in a vacuum. --- --- ### **4. Implications of E = mc²** The equation has far-reaching implications across physics, technology, and everyday life. Here are some key examples: --- #### **Nuclear Physics** - **Nuclear fission**: The equation explains that mass and energy are interchangeable. This is the foundation of **E = mc²**, which is used in nuclear physics ( the photoelectric effect. - **Atomic splitting**: The equation explains that a small amount of mass can be converted into a tremendous amount of energy ( **E = mc²**. For example, splitting a uranium battery (1 kg into 4.4 jou, the energy released is **E = (1 kg × (108 J) = 9 × 10¹6 J. - **Cosmology**: The equation explains that rest mass can be converted into energy. For example, in a battery, the total rest mass of the universe can be converted into energy ( **E = mc². - **Gravravity**: The equation predicts that nothing can travel faster or faster than the speed of light. For example, in a vacuum, light cannot have mass, so it can’t move. - **Grav force**: the equation is used to design particle like particle acceler and particle acceler. - **Quantuclear applications** - **Nuclear physics**: The equation is used to explain the behavior of light. For example: - **Photoelectric effect**: The equation explains that light behaves both particle and wave properties. - **Compton effect**: the equation is used to explain the wave-particle nature of light. - **Grav effect**: the equation is used to calculate the rest mass of objects, such as planets, cars, and planets. - **Energy from mc²** - **Kinetic energy of a photon: h? / ? ˜ 9.0856 × 10¹8 J - **1 kg × (1.6021766 × 10^15 J --- --- ### **5. Historical Context** - **Einstein’s thought experiment ( Michel's famous thought experiment. - **Michel's famous thought experiment ( Michel's famous thought experiment. - **Michel's famous thought experiment = Michel's famous thought experiment. - **Michel's famous thought experiment = Michel's famous thought experiment. - **Einstein's famous thought experiment = **E = mc². - **1 kg = c² / 3 × 10 m/s ˜ 3.0856 × 10^15 j - **2 kg = c² / 3 × 10^15 j - **3 kg = c² / 3 × 10^15 j - **4 kg = c² / 3 × 10^15 j - **5 kg = c² / 3 × 10^15 j - **6 kg = c² / 3 × 10^15 j - **7 kg = c² (cancelled) ``` I liked: "For example, in a battery, the total rest mass of the universe can be converted into energy"
GiteaMirror commented 2026-05-10 06:27:42 -05:00
Author
Owner
Copy Link

@fcorneli commented on GitHub (May 5, 2026):

For example, in a battery, the total rest mass of the universe can be converted into energy

Maybe time to convert from Ollama to direct usage of llama.cpp... 😄

<!-- gh-comment-id:4382217544 --> @fcorneli commented on GitHub (May 5, 2026): > For example, in a battery, the total rest mass of the universe can be converted into energy Maybe time to convert from Ollama to direct usage of llama.cpp... 😄
GiteaMirror commented 2026-05-10 06:27:43 -05:00
Author
Owner
Copy Link

@fcorneli commented on GitHub (May 5, 2026):

I ran my benchmarks again on the latest version of llama.cpp and ollama, and the difference with Ollama is getting bigger and bigger... is it still worth it sticking with Ollama just for its better GPU memory management?

GPU: NVIDIA RTX 6000 PRO
System: AlmaLinux 10

Model llama.cpp tokens/sec Ollama tokens/sec
gpt-oss-120b 269 (+40%) 191
Qwen3-Coder-Next 179 (+45%) 123
Qwen3.5 122b 120 (+16%) 103

We benchmark llama.cpp via:

./build/bin/llama-cli --hf-repo unsloth/gpt-oss-120b-GGUF:Q4_K_M --prompt "Explain E=mc^2" --device CUDA0 --ctx-size 0
./build/bin/llama-cli --hf-repo unsloth/Qwen3-Coder-Next-GGUF:Q8_0 --prompt "Explain E=mc^2" --device CUDA0 --ctx-size 0
./build/bin/llama-cli --hf-repo unsloth/Qwen3.5-122B-A10B-GGUF:Q4_K_M --prompt "Explain E=mc^2" --device CUDA0 --ctx-size 0

We benchmark Ollama as follows:

ollama run gpt-oss:120b "Explain E=mc^2" --verbose
ollama run qwen3-coder-next:q8_0 "Explain E=mc^2" --verbose
ollama run qwen3.5:122b "Explain E=mc^2" --verbose
<!-- gh-comment-id:4382267901 --> @fcorneli commented on GitHub (May 5, 2026): I ran my benchmarks again on the latest version of llama.cpp and ollama, and the difference with Ollama is getting bigger and bigger... is it still worth it sticking with Ollama just for its better GPU memory management? GPU: NVIDIA RTX 6000 PRO System: AlmaLinux 10 | Model | llama.cpp tokens/sec | Ollama tokens/sec | |---------------------|--------|----------| | gpt-oss-120b | 269 (+40%) | 191 | | Qwen3-Coder-Next | 179 (+45%) | 123 | | Qwen3.5 122b | 120 (+16%) | 103 | We benchmark `llama.cpp` via: <pre> ./build/bin/llama-cli --hf-repo unsloth/gpt-oss-120b-GGUF:Q4_K_M --prompt "Explain E=mc^2" --device CUDA0 --ctx-size 0 ./build/bin/llama-cli --hf-repo unsloth/Qwen3-Coder-Next-GGUF:Q8_0 --prompt "Explain E=mc^2" --device CUDA0 --ctx-size 0 ./build/bin/llama-cli --hf-repo unsloth/Qwen3.5-122B-A10B-GGUF:Q4_K_M --prompt "Explain E=mc^2" --device CUDA0 --ctx-size 0 </pre> We benchmark Ollama as follows: <pre> ollama run gpt-oss:120b "Explain E=mc^2" --verbose ollama run qwen3-coder-next:q8_0 "Explain E=mc^2" --verbose ollama run qwen3.5:122b "Explain E=mc^2" --verbose </pre>
GiteaMirror commented 2026-05-10 06:27:45 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (May 5, 2026):

You'll find with the gpt-oss:120b model the llama.cpp models tend to over-quantize a lot of the tensors, which is why you're getting higher tokens/sec. I can't say for Qwen3/Qwen3.5 coder as I'm not sure how they're quantized. That said, the Ollama version of the GGML library is out of date, but we're hoping to get this fixed soon.

<!-- gh-comment-id:4382408267 --> @pdevine commented on GitHub (May 5, 2026): You'll find with the gpt-oss:120b model the llama.cpp models tend to over-quantize a lot of the tensors, which is why you're getting higher tokens/sec. I can't say for Qwen3/Qwen3.5 coder as I'm not sure how they're quantized. That said, the Ollama version of the GGML library is out of date, but we're hoping to get this fixed soon.
GiteaMirror commented 2026-05-10 06:27:47 -05:00
Author
Owner
Copy Link

@fcorneli commented on GitHub (May 5, 2026):

mmm... I've downloaded the gpt-oss model months ago... and still the benchmark results of llama.cpp keep improving... cannot be just quantization here... they are just better in optimization... indeed Ollama urgently has to re-sync against llama.cpp...

<!-- gh-comment-id:4382465094 --> @fcorneli commented on GitHub (May 5, 2026): mmm... I've downloaded the gpt-oss model months ago... and still the benchmark results of llama.cpp keep improving... cannot be just quantization here... they are just better in optimization... indeed Ollama urgently has to re-sync against llama.cpp...
GiteaMirror commented 2026-05-10 06:27:50 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (May 5, 2026):

It looks like there was an incorrect setting in the mistral metadata for the rope parameters (mscale_all_dim was set to 1.0 instead of 0.0). I'm reconverting now to see if that ends up fixing it.

<!-- gh-comment-id:4382632301 --> @pdevine commented on GitHub (May 5, 2026): It looks like there was an incorrect setting in the mistral metadata for the rope parameters (mscale_all_dim was set to 1.0 instead of 0.0). I'm reconverting now to see if that ends up fixing it.
GiteaMirror commented 2026-05-10 06:27:51 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (May 5, 2026):

It looks like this has fixed it in my testing with the q4_K_M model. I have to republish the models though which I'll push to my own repo first and then move into the library. If you want to try it out you can use:

ollama run pdevine/mistral-medium-3.5:128b-q4_K_M

It's going to take another couple hours to re-convert/quantize the Q8_0 and BF16 models.

<!-- gh-comment-id:4383308060 --> @pdevine commented on GitHub (May 5, 2026): It looks like this has fixed it in my testing with the q4_K_M model. I have to republish the models though which I'll push to my own repo first and then move into the library. If you want to try it out you can use: `ollama run pdevine/mistral-medium-3.5:128b-q4_K_M` It's going to take another couple hours to re-convert/quantize the Q8_0 and BF16 models.
GiteaMirror commented 2026-05-10 06:27:52 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (May 5, 2026):

OK. I've refreshed the models in the library Unfortunately you will need to re-pull for this to work (sorry!). The bug was a misconfiguration in the mistral model uploaded to HuggingFace.

I'm going to go ahead and close this, but feel free to keep commenting.

<!-- gh-comment-id:4383871222 --> @pdevine commented on GitHub (May 5, 2026): OK. I've refreshed the models in the [library](https://ollama.com/library/mistral-medium-3.5/tags) Unfortunately you will need to re-pull for this to work (sorry!). The bug was a misconfiguration in the mistral model uploaded to HuggingFace. I'm going to go ahead and close this, but feel free to keep commenting.
GiteaMirror commented 2026-05-10 06:27:52 -05:00
Author
Owner
Copy Link

@fcorneli commented on GitHub (May 6, 2026):

@pdevine Thanks for fixing this so fast!
I can confirm that both mistral-medium-3.5:128b-q4_K_M and mistral-medium-3.5:128b-q8_0 now work within OpenCode as expected using two NVIDIA RTX 6000 PROs.

<!-- gh-comment-id:4385981953 --> @fcorneli commented on GitHub (May 6, 2026): @pdevine Thanks for fixing this so fast! I can confirm that both `mistral-medium-3.5:128b-q4_K_M` and `mistral-medium-3.5:128b-q8_0` now work within OpenCode as expected using two NVIDIA RTX 6000 PROs.
GiteaMirror commented 2026-05-10 06:27:53 -05:00
Author
Owner
Copy Link

@Notbici commented on GitHub (May 6, 2026):

OK. I've refreshed the models in the library Unfortunately you will need to re-pull for this to work (sorry!). The bug was a misconfiguration in the mistral model uploaded to HuggingFace.

I'm going to go ahead and close this, but feel free to keep commenting.

@pdevine would you know if this newly refreshed model supports thinking? I've enabled "reasoning effort" to "high", tried messing with the top_p and temperature and can't seem to get it to reason, even if I literally say "think really hard about the answer", its not reasoning.

<!-- gh-comment-id:4386899992 --> @Notbici commented on GitHub (May 6, 2026): > OK. I've refreshed the models in the [library](https://ollama.com/library/mistral-medium-3.5/tags) Unfortunately you will need to re-pull for this to work (sorry!). The bug was a misconfiguration in the mistral model uploaded to HuggingFace. > > I'm going to go ahead and close this, but feel free to keep commenting. @pdevine would you know if this newly refreshed model supports thinking? I've enabled "reasoning effort" to "high", tried messing with the top_p and temperature and can't seem to get it to reason, even if I literally say "think really hard about the answer", its not reasoning.
GiteaMirror commented 2026-05-10 06:27:53 -05:00
Author
Owner
Copy Link

@fcorneli commented on GitHub (May 7, 2026):

For reference, using llama.cpp you can enable this high reasoning effort via:

./build/bin/llama-server --port 8081 --ctx-size 0 --models-max 1 --offline --models-preset models-preset.ini

with models-preset.ini containing:

version = 1

[unsloth/Mistral-Medium-3.5-128B-GGUF:Q4_K_XL]
temp = 0.7
chat-template-kwargs = {"reasoning_effort":"high"}

[unsloth/Mistral-Medium-3.5-128B-GGUF:Q8_0]
temp = 0.7
chat-template-kwargs = {"reasoning_effort":"high"}
ctx-size = 150000

And indeed, within OpenCode you see the "Thinking: ..." appear.

<!-- gh-comment-id:4395311528 --> @fcorneli commented on GitHub (May 7, 2026): For reference, using llama.cpp you can enable this high reasoning effort via: ```bash ./build/bin/llama-server --port 8081 --ctx-size 0 --models-max 1 --offline --models-preset models-preset.ini ``` with `models-preset.ini` containing: ```ini version = 1 [unsloth/Mistral-Medium-3.5-128B-GGUF:Q4_K_XL] temp = 0.7 chat-template-kwargs = {"reasoning_effort":"high"} [unsloth/Mistral-Medium-3.5-128B-GGUF:Q8_0] temp = 0.7 chat-template-kwargs = {"reasoning_effort":"high"} ctx-size = 150000 ``` And indeed, within OpenCode you see the "Thinking: ..." appear.
GiteaMirror commented 2026-05-10 06:27:54 -05:00
Author
Owner
Copy Link

@fcorneli commented on GitHub (May 7, 2026):

For Ollama, you can also enable this high reasoning effort in OpenCode via:

"mistral-medium-3.5:128b-q4_K_M": {
    "name": "mistral-medium-3.5:128b-q4_K_M",
    "options": {
        "reasoningEffort": "high",
        "temperature": 0.7
    }
},
<!-- gh-comment-id:4395463012 --> @fcorneli commented on GitHub (May 7, 2026): For Ollama, you can also enable this high reasoning effort in OpenCode via: ```json "mistral-medium-3.5:128b-q4_K_M": { "name": "mistral-medium-3.5:128b-q4_K_M", "options": { "reasoningEffort": "high", "temperature": 0.7 } }, ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#87861
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?