github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 16:11:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #5892] Ollama: 500 error on Larger Models #29437

New Issue
Closed
opened 2026-04-22 08:19:21 -05:00 by GiteaMirror · 62 comments
GiteaMirror commented 2026-04-22 08:19:21 -05:00
Owner
Copy Link

Originally created by @nicholhai on GitHub (Jul 23, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/5892

What is the issue?

Whenever I try to run a model greater than the 7b or 8b, I get the following error. HOWEVER, any of the regular ones that are 7b and 8b run just fine.

Ollama: 500, message='Internal Server Error', url=URL('http://localhost:11434/api/chat')

  • Running Ubuntu Server 24.04
  • Running through docker
  • i7 2.1GHz
  • 64GB RAM
  • GeForce RTX 4060 Ti 16GB

Any assistance would be appreciated

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

No response

Originally created by @nicholhai on GitHub (Jul 23, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/5892 ### What is the issue? Whenever I try to run a model greater than the 7b or 8b, I get the following error. HOWEVER, any of the regular ones that are 7b and 8b run just fine. Ollama: 500, message='Internal Server Error', url=URL('http://localhost:11434/api/chat') - Running Ubuntu Server 24.04 - Running through docker - i7 2.1GHz - 64GB RAM - GeForce RTX 4060 Ti 16GB Any assistance would be appreciated ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version _No response_
GiteaMirror added the bug label 2026-04-22 08:19:21 -05:00
GiteaMirror commented 2026-04-22 08:19:23 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 23, 2024):

Server logs may make it easier to diagnose the issue.

<!-- gh-comment-id:2246309295 --> @rick-github commented on GitHub (Jul 23, 2024): Server logs may make it easier to diagnose the issue.
GiteaMirror commented 2026-04-22 08:19:24 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 23, 2024):

Running Ollama through single docker: docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama

However, I tried it with installing ollama as well and then just launching Docker instance with Openweb UI.. same thing

<!-- gh-comment-id:2246310684 --> @nicholhai commented on GitHub (Jul 23, 2024): Running Ollama through single docker: docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama However, I tried it with installing ollama as well and then just launching Docker instance with Openweb UI.. same thing
GiteaMirror commented 2026-04-22 08:19:25 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 23, 2024):

Server logs may make it easier to diagnose the issue.

How would I get this since its a docker container? Thanks in advance

<!-- gh-comment-id:2246312978 --> @nicholhai commented on GitHub (Jul 23, 2024): > Server logs may make it easier to diagnose the issue. How would I get this since its a docker container? Thanks in advance
GiteaMirror commented 2026-04-22 08:19:28 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 23, 2024):

docker logs

<!-- gh-comment-id:2246314500 --> @rick-github commented on GitHub (Jul 23, 2024): `docker logs`
GiteaMirror commented 2026-04-22 08:19:30 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 23, 2024):

Log output below. What's odd is that this is running on 192.168.3.59, yet it references another machine (in the logs below) with a .17 IP that is also running ollama with openweb UI....

INFO [apps.ollama.main] url: http://localhost:11434
time=2024-07-23T21:07:22.397Z level=INFO source=sched.go:738 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 gpu=GPU-02cae464-d29e-31cd-cd63-22666e511c22 parallel=4 available=16604921856 required="8.7 GiB"
time=2024-07-23T21:07:22.397Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=41 layers.offload=41 layers.split="" memory.available="[15.5 GiB]" memory.required.full="8.7 GiB" memory.required.partial="8.7 GiB" memory.required.kv="1.2 GiB" memory.required.allocations="[8.7 GiB]" memory.weights.total="7.0 GiB" memory.weights.repeating="6.5 GiB" memory.weights.nonrepeating="525.0 MiB" memory.graph.full="568.0 MiB" memory.graph.partial="801.0 MiB"
time=2024-07-23T21:07:22.398Z level=INFO source=server.go:375 msg="starting llama server" cmd="/tmp/ollama3862763016/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 41 --parallel 4 --port 40397"
time=2024-07-23T21:07:22.398Z level=INFO source=sched.go:474 msg="loaded runners" count=1
time=2024-07-23T21:07:22.398Z level=INFO source=server.go:563 msg="waiting for llama runner to start responding"
time=2024-07-23T21:07:22.398Z level=INFO source=server.go:604 msg="waiting for server to become available" status="llm server error"
INFO [main] build info | build=1 commit="a8db2a9" tid="123403186212864" timestamp=1721768842
INFO [main] system info | n_threads=8 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 0 | " tid="123403186212864" timestamp=1721768842 total_threads=24
INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="23" port="40397" tid="123403186212864" timestamp=1721768842
llama_model_loader: loaded meta data with 35 key-value pairs and 363 tensors from /root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Mistral Nemo Instruct 2407
llama_model_loader: - kv 3: general.version str = 2407
llama_model_loader: - kv 4: general.finetune str = Instruct
llama_model_loader: - kv 5: general.basename str = Mistral-Nemo
llama_model_loader: - kv 6: general.size_label str = 12B
llama_model_loader: - kv 7: general.license str = apache-2.0
llama_model_loader: - kv 8: general.languages arr[str,9] = ["en", "fr", "de", "es", "it", "pt", ...
llama_model_loader: - kv 9: llama.block_count u32 = 40
llama_model_loader: - kv 10: llama.context_length u32 = 1024000
llama_model_loader: - kv 11: llama.embedding_length u32 = 5120
llama_model_loader: - kv 12: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 13: llama.attention.head_count u32 = 32
llama_model_loader: - kv 14: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 15: llama.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 16: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 17: llama.attention.key_length u32 = 128
llama_model_loader: - kv 18: llama.attention.value_length u32 = 128
llama_model_loader: - kv 19: general.file_type u32 = 2
llama_model_loader: - kv 20: llama.vocab_size u32 = 131072
llama_model_loader: - kv 21: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 22: tokenizer.ggml.add_space_prefix bool = false
llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 24: tokenizer.ggml.pre str = tekken
llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,131072] = ["", "", "", "[INST]", "[...
llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,131072] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,269443] = ["Ġ Ġ", "Ġ t", "e r", "i n", "Ġ �...
llama_model_loader: - kv 28: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 29: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 30: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 32: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 33: tokenizer.chat_template str = {%- if messages[0]['role'] == 'system...
llama_model_loader: - kv 34: general.quantization_version u32 = 2
llama_model_loader: - type f32: 81 tensors
llama_model_loader: - type q4_0: 281 tensors
llama_model_loader: - type q6_K: 1 tensors
llm_load_vocab: missing or unrecognized pre-tokenizer type, using: 'default'
time=2024-07-23T21:07:22.650Z level=INFO source=server.go:604 msg="waiting for server to become available" status="llm server loading model"
llm_load_vocab: special tokens cache size = 1000
llm_load_vocab: token to piece cache size = 0.8498 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 131072
llm_load_print_meta: n_merges = 269443
llm_load_print_meta: vocab_only = 0
llm_load_print_meta: n_ctx_train = 1024000
llm_load_print_meta: n_embd = 5120
llm_load_print_meta: n_layer = 40
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn = 1024000
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 13B
llm_load_print_meta: model ftype = Q4_0
llm_load_print_meta: model params = 12.25 B
llm_load_print_meta: model size = 6.58 GiB (4.61 BPW)
llm_load_print_meta: general.name = Mistral Nemo Instruct 2407
llm_load_print_meta: BOS token = 1 ''
llm_load_print_meta: EOS token = 2 ''
llm_load_print_meta: UNK token = 0 ''
llm_load_print_meta: LF token = 1196 'Ä'
llm_load_print_meta: max token length = 150
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: yes
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 4060 Ti, compute capability 8.9, VMM: yes
llm_load_tensors: ggml ctx size = 0.34 MiB
llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected 5120, 5120, got 5120, 4096, 1, 1
llama_load_model_from_file: exception loading model
terminate called after throwing an instance of 'std::runtime_error'
what(): check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected 5120, 5120, got 5120, 4096, 1, 1
time=2024-07-23T21:07:23.152Z level=ERROR source=sched.go:480 msg="error loading llama server" error="llama runner process has terminated: signal: aborted (core dumped) error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected 5120, 5120, got 5120, 4096, 1, 1\nllama_load_model_from_file: exception loading model"
[GIN] 2024/07/23 - 21:07:23 | 500 | 901.741973ms | 127.0.0.1 | POST "/api/chat"
INFO: 192.168.3.17:53773 - "POST /ollama/api/chat HTTP/1.1" 500 Internal Server Error
ERROR [asyncio] Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7afabc379390>
INFO: 192.168.3.17:53773 - "GET /api/v1/chats/ HTTP/1.1" 200 OK
time=2024-07-23T21:07:28.273Z level=WARN source=sched.go:671 msg="gpu VRAM usage didn't recover within timeout" seconds=5.12139152 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94
time=2024-07-23T21:07:28.523Z level=WARN source=sched.go:671 msg="gpu VRAM usage didn't recover within timeout" seconds=5.371245575 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94
time=2024-07-23T21:07:28.773Z level=WARN source=sched.go:671 msg="gpu VRAM usage didn't recover within timeout" seconds=5.621260718 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94
INFO: 127.0.0.1:46384 - "GET /health HTTP/1.1" 200 OK
INFO: 127.0.0.1:55030 - "GET /health HTTP/1.1" 200 OK
INFO: 127.0.0.1:53856 - "GET /health HTTP/1.1" 200 OK
INFO: 127.0.0.1:46666 - "GET /health HTTP/1.1" 200 OK
INFO: 127.0.0.1:45528 - "GET /health HTTP/1.1" 200 OK
INFO: 127.0.0.1:56908 - "GET /health HTTP/1.1" 200 OK
INFO: 127.0.0.1:47616 - "GET /health HTTP/1.1" 200 OK
INFO [apps.ollama.main] url: http://localhost:11434
time=2024-07-23T21:11:01.686Z level=INFO source=sched.go:738 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 gpu=GPU-02cae464-d29e-31cd-cd63-22666e511c22 parallel=4 available=16604921856 required="8.7 GiB"
time=2024-07-23T21:11:01.686Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=41 layers.offload=41 layers.split="" memory.available="[15.5 GiB]" memory.required.full="8.7 GiB" memory.required.partial="8.7 GiB" memory.required.kv="1.2 GiB" memory.required.allocations="[8.7 GiB]" memory.weights.total="7.0 GiB" memory.weights.repeating="6.5 GiB" memory.weights.nonrepeating="525.0 MiB" memory.graph.full="568.0 MiB" memory.graph.partial="801.0 MiB"
time=2024-07-23T21:11:01.687Z level=INFO source=server.go:375 msg="starting llama server" cmd="/tmp/ollama3862763016/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 41 --parallel 4 --port 42509"
time=2024-07-23T21:11:01.687Z level=INFO source=sched.go:474 msg="loaded runners" count=1
time=2024-07-23T21:11:01.687Z level=INFO source=server.go:563 msg="waiting for llama runner to start responding"
time=2024-07-23T21:11:01.687Z level=INFO source=server.go:604 msg="waiting for server to become available" status="llm server error"
INFO [main] build info | build=1 commit="a8db2a9" tid="126845667917824" timestamp=1721769061
INFO [main] system info | n_threads=8 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 0 | " tid="126845667917824" timestamp=1721769061 total_threads=24
INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="23" port="42509" tid="126845667917824" timestamp=1721769061
llama_model_loader: loaded meta data with 35 key-value pairs and 363 tensors from /root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Mistral Nemo Instruct 2407
llama_model_loader: - kv 3: general.version str = 2407
llama_model_loader: - kv 4: general.finetune str = Instruct
llama_model_loader: - kv 5: general.basename str = Mistral-Nemo
llama_model_loader: - kv 6: general.size_label str = 12B
llama_model_loader: - kv 7: general.license str = apache-2.0
llama_model_loader: - kv 8: general.languages arr[str,9] = ["en", "fr", "de", "es", "it", "pt", ...
llama_model_loader: - kv 9: llama.block_count u32 = 40
llama_model_loader: - kv 10: llama.context_length u32 = 1024000
llama_model_loader: - kv 11: llama.embedding_length u32 = 5120
llama_model_loader: - kv 12: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 13: llama.attention.head_count u32 = 32
llama_model_loader: - kv 14: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 15: llama.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 16: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 17: llama.attention.key_length u32 = 128
llama_model_loader: - kv 18: llama.attention.value_length u32 = 128
llama_model_loader: - kv 19: general.file_type u32 = 2
llama_model_loader: - kv 20: llama.vocab_size u32 = 131072
llama_model_loader: - kv 21: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 22: tokenizer.ggml.add_space_prefix bool = false
llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 24: tokenizer.ggml.pre str = tekken
llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,131072] = ["", "", "", "[INST]", "[...
llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,131072] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,269443] = ["Ġ Ġ", "Ġ t", "e r", "i n", "Ġ �...
llama_model_loader: - kv 28: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 29: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 30: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 32: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 33: tokenizer.chat_template str = {%- if messages[0]['role'] == 'system...
llama_model_loader: - kv 34: general.quantization_version u32 = 2
llama_model_loader: - type f32: 81 tensors
llama_model_loader: - type q4_0: 281 tensors
llama_model_loader: - type q6_K: 1 tensors
llm_load_vocab: missing or unrecognized pre-tokenizer type, using: 'default'
llm_load_vocab: special tokens cache size = 1000
time=2024-07-23T21:11:01.938Z level=INFO source=server.go:604 msg="waiting for server to become available" status="llm server loading model"
llm_load_vocab: token to piece cache size = 0.8498 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 131072
llm_load_print_meta: n_merges = 269443
llm_load_print_meta: vocab_only = 0
llm_load_print_meta: n_ctx_train = 1024000
llm_load_print_meta: n_embd = 5120
llm_load_print_meta: n_layer = 40
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn = 1024000
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 13B
llm_load_print_meta: model ftype = Q4_0
llm_load_print_meta: model params = 12.25 B
llm_load_print_meta: model size = 6.58 GiB (4.61 BPW)
llm_load_print_meta: general.name = Mistral Nemo Instruct 2407
llm_load_print_meta: BOS token = 1 ''
llm_load_print_meta: EOS token = 2 ''
llm_load_print_meta: UNK token = 0 ''
llm_load_print_meta: LF token = 1196 'Ä'
llm_load_print_meta: max token length = 150
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: yes
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 4060 Ti, compute capability 8.9, VMM: yes
llm_load_tensors: ggml ctx size = 0.34 MiB
llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected 5120, 5120, got 5120, 4096, 1, 1
llama_load_model_from_file: exception loading model
terminate called after throwing an instance of 'std::runtime_error'
what(): check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected 5120, 5120, got 5120, 4096, 1, 1
time=2024-07-23T21:11:02.439Z level=ERROR source=sched.go:480 msg="error loading llama server" error="llama runner process has terminated: signal: aborted (core dumped) error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected 5120, 5120, got 5120, 4096, 1, 1\nllama_load_model_from_file: exception loading model"
[GIN] 2024/07/23 - 21:11:02 | 500 | 904.787254ms | 127.0.0.1 | POST "/api/chat"
INFO: 192.168.3.17:53945 - "POST /ollama/api/chat HTTP/1.1" 500 Internal Server Error
INFO: 192.168.3.17:53945 - "GET /api/v1/chats/ HTTP/1.1" 200 OK
time=2024-07-23T21:11:07.570Z level=WARN source=sched.go:671 msg="gpu VRAM usage didn't recover within timeout" seconds=5.13067364 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94
time=2024-07-23T21:11:07.819Z level=WARN source=sched.go:671 msg="gpu VRAM usage didn't recover within timeout" seconds=5.379864756 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94
time=2024-07-23T21:11:08.069Z level=WARN source=sched.go:671 msg="gpu VRAM usage didn't recover within timeout" seconds=5.630042941 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94
['x2BJHVzmH2bs-VVMAAAD']
INFO: 127.0.0.1:39704 - "GET /health HTTP/1.1" 200 OK
ERROR [asyncio] Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7afabbe97190>
INFO: 127.0.0.1:58774 - "GET /health HTTP/1.1" 200 OK

<!-- gh-comment-id:2246328800 --> @nicholhai commented on GitHub (Jul 23, 2024): Log output below. What's odd is that this is running on 192.168.3.59, yet it references another machine (in the logs below) with a .17 IP that is also running ollama with openweb UI.... INFO [apps.ollama.main] url: http://localhost:11434 time=2024-07-23T21:07:22.397Z level=INFO source=sched.go:738 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 gpu=GPU-02cae464-d29e-31cd-cd63-22666e511c22 parallel=4 available=16604921856 required="8.7 GiB" time=2024-07-23T21:07:22.397Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=41 layers.offload=41 layers.split="" memory.available="[15.5 GiB]" memory.required.full="8.7 GiB" memory.required.partial="8.7 GiB" memory.required.kv="1.2 GiB" memory.required.allocations="[8.7 GiB]" memory.weights.total="7.0 GiB" memory.weights.repeating="6.5 GiB" memory.weights.nonrepeating="525.0 MiB" memory.graph.full="568.0 MiB" memory.graph.partial="801.0 MiB" time=2024-07-23T21:07:22.398Z level=INFO source=server.go:375 msg="starting llama server" cmd="/tmp/ollama3862763016/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 41 --parallel 4 --port 40397" time=2024-07-23T21:07:22.398Z level=INFO source=sched.go:474 msg="loaded runners" count=1 time=2024-07-23T21:07:22.398Z level=INFO source=server.go:563 msg="waiting for llama runner to start responding" time=2024-07-23T21:07:22.398Z level=INFO source=server.go:604 msg="waiting for server to become available" status="llm server error" INFO [main] build info | build=1 commit="a8db2a9" tid="123403186212864" timestamp=1721768842 INFO [main] system info | n_threads=8 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 0 | " tid="123403186212864" timestamp=1721768842 total_threads=24 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="23" port="40397" tid="123403186212864" timestamp=1721768842 llama_model_loader: loaded meta data with 35 key-value pairs and 363 tensors from /root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Mistral Nemo Instruct 2407 llama_model_loader: - kv 3: general.version str = 2407 llama_model_loader: - kv 4: general.finetune str = Instruct llama_model_loader: - kv 5: general.basename str = Mistral-Nemo llama_model_loader: - kv 6: general.size_label str = 12B llama_model_loader: - kv 7: general.license str = apache-2.0 llama_model_loader: - kv 8: general.languages arr[str,9] = ["en", "fr", "de", "es", "it", "pt", ... llama_model_loader: - kv 9: llama.block_count u32 = 40 llama_model_loader: - kv 10: llama.context_length u32 = 1024000 llama_model_loader: - kv 11: llama.embedding_length u32 = 5120 llama_model_loader: - kv 12: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 13: llama.attention.head_count u32 = 32 llama_model_loader: - kv 14: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 15: llama.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 16: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 17: llama.attention.key_length u32 = 128 llama_model_loader: - kv 18: llama.attention.value_length u32 = 128 llama_model_loader: - kv 19: general.file_type u32 = 2 llama_model_loader: - kv 20: llama.vocab_size u32 = 131072 llama_model_loader: - kv 21: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 22: tokenizer.ggml.add_space_prefix bool = false llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 24: tokenizer.ggml.pre str = tekken llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,131072] = ["<unk>", "<s>", "</s>", "[INST]", "[... llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,131072] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ... llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,269443] = ["Ġ Ġ", "Ġ t", "e r", "i n", "Ġ �... llama_model_loader: - kv 28: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 29: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 30: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 32: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 33: tokenizer.chat_template str = {%- if messages[0]['role'] == 'system... llama_model_loader: - kv 34: general.quantization_version u32 = 2 llama_model_loader: - type f32: 81 tensors llama_model_loader: - type q4_0: 281 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: missing or unrecognized pre-tokenizer type, using: 'default' time=2024-07-23T21:07:22.650Z level=INFO source=server.go:604 msg="waiting for server to become available" status="llm server loading model" llm_load_vocab: special tokens cache size = 1000 llm_load_vocab: token to piece cache size = 0.8498 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 131072 llm_load_print_meta: n_merges = 269443 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 1024000 llm_load_print_meta: n_embd = 5120 llm_load_print_meta: n_layer = 40 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 1000000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 1024000 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = 13B llm_load_print_meta: model ftype = Q4_0 llm_load_print_meta: model params = 12.25 B llm_load_print_meta: model size = 6.58 GiB (4.61 BPW) llm_load_print_meta: general.name = Mistral Nemo Instruct 2407 llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 1196 'Ä' llm_load_print_meta: max token length = 150 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: yes ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 4060 Ti, compute capability 8.9, VMM: yes llm_load_tensors: ggml ctx size = 0.34 MiB llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected 5120, 5120, got 5120, 4096, 1, 1 llama_load_model_from_file: exception loading model terminate called after throwing an instance of 'std::runtime_error' what(): check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected 5120, 5120, got 5120, 4096, 1, 1 time=2024-07-23T21:07:23.152Z level=ERROR source=sched.go:480 msg="error loading llama server" error="llama runner process has terminated: signal: aborted (core dumped) error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected 5120, 5120, got 5120, 4096, 1, 1\nllama_load_model_from_file: exception loading model" [GIN] 2024/07/23 - 21:07:23 | 500 | 901.741973ms | 127.0.0.1 | POST "/api/chat" INFO: 192.168.3.17:53773 - "POST /ollama/api/chat HTTP/1.1" 500 Internal Server Error ERROR [asyncio] Unclosed client session client_session: <aiohttp.client.ClientSession object at 0x7afabc379390> INFO: 192.168.3.17:53773 - "GET /api/v1/chats/ HTTP/1.1" 200 OK time=2024-07-23T21:07:28.273Z level=WARN source=sched.go:671 msg="gpu VRAM usage didn't recover within timeout" seconds=5.12139152 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 time=2024-07-23T21:07:28.523Z level=WARN source=sched.go:671 msg="gpu VRAM usage didn't recover within timeout" seconds=5.371245575 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 time=2024-07-23T21:07:28.773Z level=WARN source=sched.go:671 msg="gpu VRAM usage didn't recover within timeout" seconds=5.621260718 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 INFO: 127.0.0.1:46384 - "GET /health HTTP/1.1" 200 OK INFO: 127.0.0.1:55030 - "GET /health HTTP/1.1" 200 OK INFO: 127.0.0.1:53856 - "GET /health HTTP/1.1" 200 OK INFO: 127.0.0.1:46666 - "GET /health HTTP/1.1" 200 OK INFO: 127.0.0.1:45528 - "GET /health HTTP/1.1" 200 OK INFO: 127.0.0.1:56908 - "GET /health HTTP/1.1" 200 OK INFO: 127.0.0.1:47616 - "GET /health HTTP/1.1" 200 OK INFO [apps.ollama.main] url: http://localhost:11434 time=2024-07-23T21:11:01.686Z level=INFO source=sched.go:738 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 gpu=GPU-02cae464-d29e-31cd-cd63-22666e511c22 parallel=4 available=16604921856 required="8.7 GiB" time=2024-07-23T21:11:01.686Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=41 layers.offload=41 layers.split="" memory.available="[15.5 GiB]" memory.required.full="8.7 GiB" memory.required.partial="8.7 GiB" memory.required.kv="1.2 GiB" memory.required.allocations="[8.7 GiB]" memory.weights.total="7.0 GiB" memory.weights.repeating="6.5 GiB" memory.weights.nonrepeating="525.0 MiB" memory.graph.full="568.0 MiB" memory.graph.partial="801.0 MiB" time=2024-07-23T21:11:01.687Z level=INFO source=server.go:375 msg="starting llama server" cmd="/tmp/ollama3862763016/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 41 --parallel 4 --port 42509" time=2024-07-23T21:11:01.687Z level=INFO source=sched.go:474 msg="loaded runners" count=1 time=2024-07-23T21:11:01.687Z level=INFO source=server.go:563 msg="waiting for llama runner to start responding" time=2024-07-23T21:11:01.687Z level=INFO source=server.go:604 msg="waiting for server to become available" status="llm server error" INFO [main] build info | build=1 commit="a8db2a9" tid="126845667917824" timestamp=1721769061 INFO [main] system info | n_threads=8 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 0 | " tid="126845667917824" timestamp=1721769061 total_threads=24 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="23" port="42509" tid="126845667917824" timestamp=1721769061 llama_model_loader: loaded meta data with 35 key-value pairs and 363 tensors from /root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Mistral Nemo Instruct 2407 llama_model_loader: - kv 3: general.version str = 2407 llama_model_loader: - kv 4: general.finetune str = Instruct llama_model_loader: - kv 5: general.basename str = Mistral-Nemo llama_model_loader: - kv 6: general.size_label str = 12B llama_model_loader: - kv 7: general.license str = apache-2.0 llama_model_loader: - kv 8: general.languages arr[str,9] = ["en", "fr", "de", "es", "it", "pt", ... llama_model_loader: - kv 9: llama.block_count u32 = 40 llama_model_loader: - kv 10: llama.context_length u32 = 1024000 llama_model_loader: - kv 11: llama.embedding_length u32 = 5120 llama_model_loader: - kv 12: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 13: llama.attention.head_count u32 = 32 llama_model_loader: - kv 14: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 15: llama.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 16: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 17: llama.attention.key_length u32 = 128 llama_model_loader: - kv 18: llama.attention.value_length u32 = 128 llama_model_loader: - kv 19: general.file_type u32 = 2 llama_model_loader: - kv 20: llama.vocab_size u32 = 131072 llama_model_loader: - kv 21: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 22: tokenizer.ggml.add_space_prefix bool = false llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 24: tokenizer.ggml.pre str = tekken llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,131072] = ["<unk>", "<s>", "</s>", "[INST]", "[... llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,131072] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ... llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,269443] = ["Ġ Ġ", "Ġ t", "e r", "i n", "Ġ �... llama_model_loader: - kv 28: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 29: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 30: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 32: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 33: tokenizer.chat_template str = {%- if messages[0]['role'] == 'system... llama_model_loader: - kv 34: general.quantization_version u32 = 2 llama_model_loader: - type f32: 81 tensors llama_model_loader: - type q4_0: 281 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: missing or unrecognized pre-tokenizer type, using: 'default' llm_load_vocab: special tokens cache size = 1000 time=2024-07-23T21:11:01.938Z level=INFO source=server.go:604 msg="waiting for server to become available" status="llm server loading model" llm_load_vocab: token to piece cache size = 0.8498 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 131072 llm_load_print_meta: n_merges = 269443 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 1024000 llm_load_print_meta: n_embd = 5120 llm_load_print_meta: n_layer = 40 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 1000000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 1024000 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = 13B llm_load_print_meta: model ftype = Q4_0 llm_load_print_meta: model params = 12.25 B llm_load_print_meta: model size = 6.58 GiB (4.61 BPW) llm_load_print_meta: general.name = Mistral Nemo Instruct 2407 llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 1196 'Ä' llm_load_print_meta: max token length = 150 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: yes ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 4060 Ti, compute capability 8.9, VMM: yes llm_load_tensors: ggml ctx size = 0.34 MiB llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected 5120, 5120, got 5120, 4096, 1, 1 llama_load_model_from_file: exception loading model terminate called after throwing an instance of 'std::runtime_error' what(): check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected 5120, 5120, got 5120, 4096, 1, 1 time=2024-07-23T21:11:02.439Z level=ERROR source=sched.go:480 msg="error loading llama server" error="llama runner process has terminated: signal: aborted (core dumped) error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected 5120, 5120, got 5120, 4096, 1, 1\nllama_load_model_from_file: exception loading model" [GIN] 2024/07/23 - 21:11:02 | 500 | 904.787254ms | 127.0.0.1 | POST "/api/chat" INFO: 192.168.3.17:53945 - "POST /ollama/api/chat HTTP/1.1" 500 Internal Server Error INFO: 192.168.3.17:53945 - "GET /api/v1/chats/ HTTP/1.1" 200 OK time=2024-07-23T21:11:07.570Z level=WARN source=sched.go:671 msg="gpu VRAM usage didn't recover within timeout" seconds=5.13067364 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 time=2024-07-23T21:11:07.819Z level=WARN source=sched.go:671 msg="gpu VRAM usage didn't recover within timeout" seconds=5.379864756 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 time=2024-07-23T21:11:08.069Z level=WARN source=sched.go:671 msg="gpu VRAM usage didn't recover within timeout" seconds=5.630042941 model=/root/.ollama/models/blobs/sha256-b559938ab7a0392fc9ea9675b82280f2a15669ec3e0e0fc491c9cb0a7681cf94 ['x2BJHVzmH2bs-VVMAAAD'] INFO: 127.0.0.1:39704 - "GET /health HTTP/1.1" 200 OK ERROR [asyncio] Unclosed client session client_session: <aiohttp.client.ClientSession object at 0x7afabbe97190> INFO: 127.0.0.1:58774 - "GET /health HTTP/1.1" 200 OK
GiteaMirror commented 2026-04-22 08:19:31 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 23, 2024):

llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected 5120, 5120, got 5120, 4096, 1, 1

You are trying to run a model which is not supported by your version of ollama. The start of the logs that identifies the version is not included, but it's probably less than 0.2.8. You can try upgrading to the most recent version of ollama, but Mistral-Nemo support has only just been added and so it may not perform as well as it could until all of the bugs are ironed out.

<!-- gh-comment-id:2246376871 --> @rick-github commented on GitHub (Jul 23, 2024): ``` llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected 5120, 5120, got 5120, 4096, 1, 1 ``` You are trying to run a model which is not supported by your version of ollama. The start of the logs that identifies the version is not included, but it's probably less than 0.2.8. You can try upgrading to the most recent version of ollama, but Mistral-Nemo support has only just been added and so it may not perform as well as it could until all of the bugs are ironed out.
GiteaMirror commented 2026-04-22 08:19:32 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 23, 2024):

Thank you very much. I will investigate that

From: frob @.>
Date: Tuesday, July 23, 2024 at 5:51 PM
To: ollama/ollama @.>
Cc: nicholhai @.>, Author @.>
Subject: Re: [ollama/ollama] Ollama: 500 error on Larger Models (Issue #5892)

llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected 5120, 5120, got 5120, 4096, 1, 1

You are trying to run a model which is not supported by your version of ollama. The start of the logs that identifies the version is not included, but it's probably less than 0.2.8. You can try upgrading to the most recent version of ollama, but Mistral-Nemo support has only just been added and so it may not perform as well as it could until all of the bugs are ironed out.

—
Reply to this email directly, view it on GitHubhttps://github.com/ollama/ollama/issues/5892#issuecomment-2246376871, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AW6WDRDGHAVK4MHEEXDSK5TZN3F6HAVCNFSM6AAAAABLLF7FN2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBWGM3TMOBXGE.
You are receiving this because you authored the thread.Message ID: @.***>

<!-- gh-comment-id:2246423833 --> @nicholhai commented on GitHub (Jul 23, 2024): Thank you very much. I will investigate that From: frob ***@***.***> Date: Tuesday, July 23, 2024 at 5:51 PM To: ollama/ollama ***@***.***> Cc: nicholhai ***@***.***>, Author ***@***.***> Subject: Re: [ollama/ollama] Ollama: 500 error on Larger Models (Issue #5892) llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected 5120, 5120, got 5120, 4096, 1, 1 You are trying to run a model which is not supported by your version of ollama. The start of the logs that identifies the version is not included, but it's probably less than 0.2.8. You can try upgrading to the most recent version of ollama, but Mistral-Nemo support has only just been added and so it may not perform as well as it could until all of the bugs are ironed out. — Reply to this email directly, view it on GitHub<https://github.com/ollama/ollama/issues/5892#issuecomment-2246376871>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AW6WDRDGHAVK4MHEEXDSK5TZN3F6HAVCNFSM6AAAAABLLF7FN2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBWGM3TMOBXGE>. You are receiving this because you authored the thread.Message ID: ***@***.***>
GiteaMirror commented 2026-04-22 08:19:32 -05:00
Author
Owner
Copy Link

@Cephra commented on GitHub (Jul 24, 2024):

Can you tell me how you've started the docker container? The latest versions of the underlying images used for the creation of the docker containers will not be pulled automatically. You need to manually issue a command such as this: docker pull ollama/ollama:rocm (I use the rocm version because I run ollama on an AMD card, that might be different for you), then remove the old container and create a new one. At least that's what works for me.

Also about the IP address being different in the logs: That's likely because it's showing the internal docker IP assigned to the container ollama's running in.

<!-- gh-comment-id:2246753525 --> @Cephra commented on GitHub (Jul 24, 2024): Can you tell me how you've started the docker container? The latest versions of the underlying images used for the creation of the docker containers will not be pulled automatically. You need to manually issue a command such as this: `docker pull ollama/ollama:rocm` (I use the rocm version because I run ollama on an AMD card, that might be different for you), then remove the old container and create a new one. At least that's what works for me. Also about the IP address being different in the logs: That's likely because it's showing the internal docker IP assigned to the container ollama's running in.
GiteaMirror commented 2026-04-22 08:19:33 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama

<!-- gh-comment-id:2247525034 --> @nicholhai commented on GitHub (Jul 24, 2024): docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama
GiteaMirror commented 2026-04-22 08:19:34 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 24, 2024):

If you add --pull always it will always pull the newest version when you start.

<!-- gh-comment-id:2247537232 --> @rick-github commented on GitHub (Jul 24, 2024): If you add `--pull always` it will always pull the newest version when you start.
GiteaMirror commented 2026-04-22 08:19:35 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

I added the --pull always at the end:

sudo docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama --pull always

Got the following error:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "--pull": executable file not found in $PATH: unknown

<!-- gh-comment-id:2247592135 --> @nicholhai commented on GitHub (Jul 24, 2024): I added the --pull always at the end: sudo docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama --pull always Got the following error: docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "--pull": executable file not found in $PATH: unknown
GiteaMirror commented 2026-04-22 08:19:37 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 24, 2024):

Put it before the name of the image being pulled:

docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always  --pull always ghcr.io/open-webui/open-webui:ollama
<!-- gh-comment-id:2247596608 --> @rick-github commented on GitHub (Jul 24, 2024): Put it before the name of the image being pulled: ``` docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always --pull always ghcr.io/open-webui/open-webui:ollama ```
GiteaMirror commented 2026-04-22 08:19:38 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

Yes, I realized that.. I did it here:

sudo docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama --pull always -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama

Thank you.

<!-- gh-comment-id:2247599100 --> @nicholhai commented on GitHub (Jul 24, 2024): Yes, I realized that.. I did it here: sudo docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama --pull always -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama Thank you.
GiteaMirror commented 2026-04-22 08:19:38 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

It started. However, the original error still persists.. All 7b, 9b models run just fine. Just as I try using 12b and up, I get the error:

Ollama: 500, message='Internal Server Error', url=URL('http://localhost:11434/api/chat')

I have it running all the large models on my Mac Studio, no issues and I followed the same installation guide from https://docs.openwebui.com/getting-started/

Not sure what the difference is. I have wiped the server and reinstalled this using all the methods on that page, 20 times over

ollama: Pulling from open-webui/open-webui
Digest: sha256:199f1d5e5bf5c6954af376af3738e1ff76aab4987677abd177d01e5b97b6ca2c
Status: Image is up to date for ghcr.io/open-webui/open-webui:ollama
c7aecdd05e00ef66a8fb0b31021e83e4b47a5e062dddbc8ca19376f698543100

<!-- gh-comment-id:2247607306 --> @nicholhai commented on GitHub (Jul 24, 2024): It started. However, the original error still persists.. All 7b, 9b models run just fine. Just as I try using 12b and up, I get the error: **Ollama: 500, message='Internal Server Error', url=URL('http://localhost:11434/api/chat')** I have it running all the large models on my Mac Studio, no issues and I followed the same installation guide from https://docs.openwebui.com/getting-started/ Not sure what the difference is. I have wiped the server and reinstalled this using all the methods on that page, 20 times over ollama: Pulling from open-webui/open-webui Digest: sha256:199f1d5e5bf5c6954af376af3738e1ff76aab4987677abd177d01e5b97b6ca2c Status: Image is up to date for ghcr.io/open-webui/open-webui:ollama c7aecdd05e00ef66a8fb0b31021e83e4b47a5e062dddbc8ca19376f698543100
GiteaMirror commented 2026-04-22 08:19:40 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 24, 2024):

The version of ollama in that container is 0.2.1. You can wait until they update ghcr.io/open-webui/open-webui:ollama or run standalone ghcr.io/open-webui/open-webui with your own ollama instance.

<!-- gh-comment-id:2247616789 --> @rick-github commented on GitHub (Jul 24, 2024): The version of ollama in that container is 0.2.1. You can wait until they update ghcr.io/open-webui/open-webui:ollama or run standalone ghcr.io/open-webui/open-webui with your own ollama instance.
GiteaMirror commented 2026-04-22 08:19:41 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

The confusing part is that it all runs on my Mac Studio.... any ideas why or what is different? Any other suggestions for installing a whole new instance of ollama and openweb UI that will work with Ubuntu server? (or any other OS as I have this machine dedicated to this)

<!-- gh-comment-id:2247623876 --> @nicholhai commented on GitHub (Jul 24, 2024): The confusing part is that it all runs on my Mac Studio.... any ideas why or what is different? Any other suggestions for installing a whole new instance of ollama and openweb UI that **will** work with Ubuntu server? (or any other OS as I have this machine dedicated to this)
GiteaMirror commented 2026-04-22 08:19:41 -05:00
Author
Owner
Copy Link

@Cephra commented on GitHub (Jul 24, 2024):

@nicholhai it looks like the container you're starting is the open-webui one. That is not ollama, but the web UI you use to chat with LLMs. Do you have another container running that is ollama?

<!-- gh-comment-id:2247625456 --> @Cephra commented on GitHub (Jul 24, 2024): @nicholhai it looks like the container you're starting is the open-webui one. That is not ollama, but the web UI you use to chat with LLMs. Do you have another container running that is ollama?
GiteaMirror commented 2026-04-22 08:19:42 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

@nicholhai it looks like the container you're starting is the open-webui one. That is not ollama, but the web UI you use to chat with LLMs. Do you have another container running that is ollama?

That is the "all-in-one" container that will run ollama and web UI according to the instructions on : https://docs.openwebui.com/getting-started/

<!-- gh-comment-id:2247629128 --> @nicholhai commented on GitHub (Jul 24, 2024): > @nicholhai it looks like the container you're starting is the open-webui one. That is not ollama, but the web UI you use to chat with LLMs. Do you have another container running that is ollama? That is the "all-in-one" container that will run ollama and web UI according to the instructions on : **https://docs.openwebui.com/getting-started/**
GiteaMirror commented 2026-04-22 08:19:45 -05:00
Author
Owner
Copy Link

@Cephra commented on GitHub (Jul 24, 2024):

Thanks for the clarification. I wasn't aware that there was such a thing.

<!-- gh-comment-id:2247630537 --> @Cephra commented on GitHub (Jul 24, 2024): Thanks for the clarification. I wasn't aware that there was such a thing.
GiteaMirror commented 2026-04-22 08:19:47 -05:00
Author
Owner
Copy Link

@Cephra commented on GitHub (Jul 24, 2024):

Maybe try running ollama in a separate container then. The official ollama images are updated more frequently.

<!-- gh-comment-id:2247632949 --> @Cephra commented on GitHub (Jul 24, 2024): Maybe try running ollama in a separate container then. The official ollama images are updated more frequently.
GiteaMirror commented 2026-04-22 08:19:47 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

Thanks for the clarification. I wasn't aware that there was such a thing.

No worries. This is the section:

Installing Open WebUI with Bundled Ollama Support
This installation method uses a single container image that bundles Open WebUI with Ollama, allowing for a streamlined setup via a single command. Choose the appropriate command based on your hardware setup:

With GPU Support: Utilize GPU resources by running the following command:

docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama

For CPU Only: If you're not using a GPU, use this command instead:

docker run -d -p 3000:8080 -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama

Both commands facilitate a built-in, hassle-free installation of both Open WebUI and Ollama, ensuring that you can get everything up and running swiftly.

After installation, you can access Open WebUI at http://localhost:3000/.

<!-- gh-comment-id:2247635112 --> @nicholhai commented on GitHub (Jul 24, 2024): > Thanks for the clarification. I wasn't aware that there was such a thing. No worries. This is the section: **Installing Open WebUI with Bundled Ollama Support** This installation method uses a single container image that bundles Open WebUI with Ollama, allowing for a streamlined setup via a single command. Choose the appropriate command based on your hardware setup: **With GPU Support: Utilize GPU resources by running the following command:** _docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama_ **For CPU Only: If you're not using a GPU, use this command instead:** _docker run -d -p 3000:8080 -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama_ Both commands facilitate a built-in, hassle-free installation of both Open WebUI and Ollama, ensuring that you can get everything up and running swiftly. After installation, you can access Open WebUI at http://localhost:3000/.
GiteaMirror commented 2026-04-22 08:19:48 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

Maybe try running ollama in a separate container then. The official ollama images are updated more frequently.

I don't know how to do that, unfortunately. Any help would be appreciated as I am not as skilled in docker containers and how to reference one from another

<!-- gh-comment-id:2247637705 --> @nicholhai commented on GitHub (Jul 24, 2024): > Maybe try running ollama in a separate container then. The official ollama images are updated more frequently. I don't know how to do that, unfortunately. Any help would be appreciated as I am not as skilled in docker containers and how to reference one from another
GiteaMirror commented 2026-04-22 08:19:48 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 24, 2024):

Start the ollama container:

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Start the open-webui container:

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main
<!-- gh-comment-id:2247640794 --> @rick-github commented on GitHub (Jul 24, 2024): Start the ollama container: ``` docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama ``` Start the open-webui container: ``` docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main ```
GiteaMirror commented 2026-04-22 08:19:49 -05:00
Author
Owner
Copy Link

@Cephra commented on GitHub (Jul 24, 2024):

Maybe try running ollama in a separate container then. The official ollama images are updated more frequently.

I don't know how to do that, unfortunately. Any help would be appreciated as I am not as skilled in docker containers and how to reference one from another

Sure. I'd first read this
https://hub.docker.com/r/ollama/ollama
It will explain how to run ollama using docker.

After that you should check the official open-webui docs: https://github.com/open-webui/open-webui?tab=readme-ov-file#installation-with-default-configuration

It explains the steps in order to use the local ollama container with the UI. I use the same setup myself and it's been working fine for a long time now. If you need any help feel free to ask.

<!-- gh-comment-id:2247642981 --> @Cephra commented on GitHub (Jul 24, 2024): > > Maybe try running ollama in a separate container then. The official ollama images are updated more frequently. > > I don't know how to do that, unfortunately. Any help would be appreciated as I am not as skilled in docker containers and how to reference one from another Sure. I'd first read this https://hub.docker.com/r/ollama/ollama It will explain how to run ollama using docker. After that you should check the official open-webui docs: https://github.com/open-webui/open-webui?tab=readme-ov-file#installation-with-default-configuration It explains the steps in order to use the local ollama container with the UI. I use the same setup myself and it's been working fine for a long time now. If you need any help feel free to ask.
GiteaMirror commented 2026-04-22 08:19:50 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 24, 2024):

It's curious that your Mac Studio works. Server logs from that might be illuminating.

<!-- gh-comment-id:2247647347 --> @rick-github commented on GitHub (Jul 24, 2024): It's curious that your Mac Studio works. Server logs from that might be illuminating.
GiteaMirror commented 2026-04-22 08:19:50 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

Start the ollama container:

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Start the open-webui container:

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

Thank you. Trying this now... how would I run this with GPU support? just add --gpus=all ?

<!-- gh-comment-id:2247659264 --> @nicholhai commented on GitHub (Jul 24, 2024): > Start the ollama container: > > ``` > docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama > ``` > > Start the open-webui container: > > ``` > docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main > ``` Thank you. Trying this now... how would I run this with GPU support? just add _--gpus=all_ ?
GiteaMirror commented 2026-04-22 08:19:51 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 24, 2024):

You only need GPU support on the ollama container, open-webui (as far as I know) doesn't do any inference itself.

<!-- gh-comment-id:2247662499 --> @rick-github commented on GitHub (Jul 24, 2024): You only need GPU support on the ollama container, open-webui (as far as I know) doesn't do any inference itself.
GiteaMirror commented 2026-04-22 08:19:51 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

Now a new fresh hell lol. When I try to download a model after starting the above two, I get:

Download Cancelled
Open WebUI: Server Connection Error

<!-- gh-comment-id:2247662526 --> @nicholhai commented on GitHub (Jul 24, 2024): Now a new fresh hell lol. When I try to download a model after starting the above two, I get: **_Download Cancelled Open WebUI: Server Connection Error_**
GiteaMirror commented 2026-04-22 08:19:53 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 24, 2024):

Logs from both containers.

<!-- gh-comment-id:2247664665 --> @rick-github commented on GitHub (Jul 24, 2024): Logs from both containers.
GiteaMirror commented 2026-04-22 08:19:54 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

Openweb Container
client_session: <aiohttp.client.ClientSession object at 0x7104b17a7c50>
INFO: 192.168.3.17:51151 - "POST /ollama/api/pull/0 HTTP/1.1" 500 Internal Server Error
INFO [apps.openai.main] get_all_models()
INFO [apps.ollama.main] get_all_models()
ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused]
INFO: 127.0.0.1:49484 - "GET /health HTTP/1.1" 200 OK

Ollama Container

2024/07/24 11:21:23 routes.go:1100: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
time=2024-07-24T11:21:23.353Z level=INFO source=images.go:784 msg="total blobs: 44"
time=2024-07-24T11:21:23.354Z level=INFO source=images.go:791 msg="total unused blobs removed: 0"
time=2024-07-24T11:21:23.354Z level=INFO source=routes.go:1147 msg="Listening on [::]:11434 (version 0.2.8)"
time=2024-07-24T11:21:23.355Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama1220858021/runners
time=2024-07-24T11:21:26.002Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cuda_v11 rocm_v60102 cpu cpu_avx cpu_avx2]"
time=2024-07-24T11:21:26.002Z level=INFO source=gpu.go:205 msg="looking for compatible GPUs"
time=2024-07-24T11:21:26.239Z level=INFO source=types.go:105 msg="inference compute" id=GPU-02cae464-d29e-31cd-cd63-22666e511c22 library=cuda compute=8.9 driver=12.5 name="NVIDIA GeForce RTX 4060 Ti" total="15.6 GiB" available="15.5 GiB"

<!-- gh-comment-id:2247672773 --> @nicholhai commented on GitHub (Jul 24, 2024): **Openweb Container** client_session: <aiohttp.client.ClientSession object at 0x7104b17a7c50> INFO: 192.168.3.17:51151 - "POST /ollama/api/pull/0 HTTP/1.1" 500 Internal Server Error INFO [apps.openai.main] get_all_models() INFO [apps.ollama.main] get_all_models() ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused] INFO: 127.0.0.1:49484 - "GET /health HTTP/1.1" 200 OK **Ollama Container** 2024/07/24 11:21:23 routes.go:1100: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]" time=2024-07-24T11:21:23.353Z level=INFO source=images.go:784 msg="total blobs: 44" time=2024-07-24T11:21:23.354Z level=INFO source=images.go:791 msg="total unused blobs removed: 0" time=2024-07-24T11:21:23.354Z level=INFO source=routes.go:1147 msg="Listening on [::]:11434 (version 0.2.8)" time=2024-07-24T11:21:23.355Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama1220858021/runners time=2024-07-24T11:21:26.002Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cuda_v11 rocm_v60102 cpu cpu_avx cpu_avx2]" time=2024-07-24T11:21:26.002Z level=INFO source=gpu.go:205 msg="looking for compatible GPUs" time=2024-07-24T11:21:26.239Z level=INFO source=types.go:105 msg="inference compute" id=GPU-02cae464-d29e-31cd-cd63-22666e511c22 library=cuda compute=8.9 driver=12.5 name="NVIDIA GeForce RTX 4060 Ti" total="15.6 GiB" available="15.5 GiB"
GiteaMirror commented 2026-04-22 08:19:54 -05:00
Author
Owner
Copy Link

@Cephra commented on GitHub (Jul 24, 2024):

The webui container tries to access ollama running on localhost. But since ollama is running in a separate container, you will have to explicitly set the hostname of ollama in the webui.

<!-- gh-comment-id:2247711351 --> @Cephra commented on GitHub (Jul 24, 2024): The webui container tries to access ollama running on localhost. But since ollama is running in a separate container, you will have to explicitly set the hostname of ollama in the webui.
GiteaMirror commented 2026-04-22 08:19:56 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

How would I get the hostname for the ollama container?

<!-- gh-comment-id:2247714432 --> @nicholhai commented on GitHub (Jul 24, 2024): How would I get the hostname for the ollama container?
GiteaMirror commented 2026-04-22 08:19:57 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 24, 2024):

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data --name open-webui --restart always --pull always ghcr.io/open-webui/open-webui:main
<!-- gh-comment-id:2247717235 --> @rick-github commented on GitHub (Jul 24, 2024): ``` docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data --name open-webui --restart always --pull always ghcr.io/open-webui/open-webui:main ```
GiteaMirror commented 2026-04-22 08:19:58 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

I stopped both docker containers: sudo docker rm -f $(sudo docker ps -a -q)

Then I started them both again one by one (this time using the new one you just posted): Same error for Download Cancelled..... as above

sudo docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

sudo docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data --name open-webui --restart always --pull always ghcr.io/open-webui/open-webui:main

<!-- gh-comment-id:2247728942 --> @nicholhai commented on GitHub (Jul 24, 2024): I stopped both docker containers: sudo docker rm -f $(sudo docker ps -a -q) Then I started them both again one by one (this time using the new one you just posted): Same error for Download Cancelled..... as above `sudo docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama` `sudo docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data --name open-webui --restart always --pull always ghcr.io/open-webui/open-webui:main`
GiteaMirror commented 2026-04-22 08:20:01 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 24, 2024):

Hmm, worked fine here. Logs from new instances?

<!-- gh-comment-id:2247732739 --> @rick-github commented on GitHub (Jul 24, 2024): Hmm, worked fine here. Logs from new instances?
GiteaMirror commented 2026-04-22 08:20:01 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

2024/07/24 11:54:17 routes.go:1100: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
time=2024-07-24T11:54:17.503Z level=INFO source=images.go:784 msg="total blobs: 44"
time=2024-07-24T11:54:17.503Z level=INFO source=images.go:791 msg="total unused blobs removed: 0"
time=2024-07-24T11:54:17.504Z level=INFO source=routes.go:1147 msg="Listening on [::]:11434 (version 0.2.8)"
time=2024-07-24T11:54:17.505Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama3984924101/runners
time=2024-07-24T11:54:19.630Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 rocm_v60102]"
time=2024-07-24T11:54:19.630Z level=INFO source=gpu.go:205 msg="looking for compatible GPUs"
time=2024-07-24T11:54:19.863Z level=INFO source=types.go:105 msg="inference compute" id=GPU-02cae464-d29e-31cd-cd63-22666e511c22 library=cuda compute=8.9 driver=12.5 name="NVIDIA GeForce RTX 4060 Ti" total="15.6 GiB" available="15.5 GiB

INFO: 127.0.0.1:45242 - "GET /health HTTP/1.1" 200 OK
INFO [apps.openai.main] get_all_models()
INFO [apps.ollama.main] get_all_models()
ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused]
INFO: 127.0.0.1:44652 - "GET /health HTTP/1.1" 200 OK
INFO [apps.openai.main] get_all_models()
INFO [apps.ollama.main] get_all_models()
ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused]
INFO: 127.0.0.1:59950 - "GET /health HTTP/1.1" 200 OK

<!-- gh-comment-id:2247736926 --> @nicholhai commented on GitHub (Jul 24, 2024): 2024/07/24 11:54:17 routes.go:1100: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]" time=2024-07-24T11:54:17.503Z level=INFO source=images.go:784 msg="total blobs: 44" time=2024-07-24T11:54:17.503Z level=INFO source=images.go:791 msg="total unused blobs removed: 0" time=2024-07-24T11:54:17.504Z level=INFO source=routes.go:1147 msg="Listening on [::]:11434 (version 0.2.8)" time=2024-07-24T11:54:17.505Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama3984924101/runners time=2024-07-24T11:54:19.630Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 rocm_v60102]" time=2024-07-24T11:54:19.630Z level=INFO source=gpu.go:205 msg="looking for compatible GPUs" time=2024-07-24T11:54:19.863Z level=INFO source=types.go:105 msg="inference compute" id=GPU-02cae464-d29e-31cd-cd63-22666e511c22 library=cuda compute=8.9 driver=12.5 name="NVIDIA GeForce RTX 4060 Ti" total="15.6 GiB" available="15.5 GiB INFO: 127.0.0.1:45242 - "GET /health HTTP/1.1" 200 OK INFO [apps.openai.main] get_all_models() INFO [apps.ollama.main] get_all_models() ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused] INFO: 127.0.0.1:44652 - "GET /health HTTP/1.1" 200 OK INFO [apps.openai.main] get_all_models() INFO [apps.ollama.main] get_all_models() ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused] INFO: 127.0.0.1:59950 - "GET /health HTTP/1.1" 200 OK
GiteaMirror commented 2026-04-22 08:20:02 -05:00
Author
Owner
Copy Link

@Cephra commented on GitHub (Jul 24, 2024):

Can you try this:

To start ollama: docker run --restart always -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

To start open-webui: docker run -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 --name open-webui --restart always ghcr.io/open-webui/open-webui:main

These are the commands I am using.

NOTE: Make sure you change the image in the ollama command to suit your needs. I saw you have an nvidia card so you'll have to adjust the image in order to get GPU acceleration.

<!-- gh-comment-id:2247742172 --> @Cephra commented on GitHub (Jul 24, 2024): Can you try this: To start ollama: `docker run --restart always -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama` To start open-webui: `docker run -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 --name open-webui --restart always ghcr.io/open-webui/open-webui:main` These are the commands I am using. **NOTE**: Make sure you change the image in the ollama command to suit your needs. I saw you have an nvidia card so you'll have to adjust the image in order to get GPU acceleration.
GiteaMirror commented 2026-04-22 08:20:04 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

INFO: 192.168.3.17:52600 - "GET /ollama/config HTTP/1.1" 200 OK
INFO [apps.openai.main] get_all_models()
INFO [apps.ollama.main] get_all_models()
INFO [apps.openai.main] get_all_models()
INFO [apps.ollama.main] get_all_models()
ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused]
ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused]
INFO [apps.ollama.main] get_all_models()
INFO [apps.ollama.main] get_all_models()
ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused]
ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused]
ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused]

<!-- gh-comment-id:2247743707 --> @nicholhai commented on GitHub (Jul 24, 2024): INFO: 192.168.3.17:52600 - "GET /ollama/config HTTP/1.1" 200 OK INFO [apps.openai.main] get_all_models() INFO [apps.ollama.main] get_all_models() INFO [apps.openai.main] get_all_models() INFO [apps.ollama.main] get_all_models() ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused] ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused] INFO [apps.ollama.main] get_all_models() INFO [apps.ollama.main] get_all_models() ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused] ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused] ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused]
GiteaMirror commented 2026-04-22 08:20:04 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 24, 2024):

Still trying to connect to localhost. Do docker stop open-webui ; docker rm open-webui and then run the docker run command for open-webui.

<!-- gh-comment-id:2247746358 --> @rick-github commented on GitHub (Jul 24, 2024): Still trying to connect to localhost. Do `docker stop open-webui ; docker rm open-webui` and then run the `docker run` command for open-webui.
GiteaMirror commented 2026-04-22 08:20:05 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 24, 2024):

ollama image contains Nvidia support.

<!-- gh-comment-id:2247750462 --> @rick-github commented on GitHub (Jul 24, 2024): ollama image contains Nvidia support.
GiteaMirror commented 2026-04-22 08:20:07 -05:00
Author
Owner
Copy Link

@Cephra commented on GitHub (Jul 24, 2024):

Also something you should try: After removing the containers, issue a docker image prune -a to ensure the old, possibly outdated, images are also deleted.

IIRC simply issuing docker rm -f does not remove the images of the containers.

<!-- gh-comment-id:2247751032 --> @Cephra commented on GitHub (Jul 24, 2024): Also something you should try: After removing the containers, issue a `docker image prune -a` to ensure the old, possibly outdated, images are also deleted. IIRC simply issuing `docker rm -f` does not remove the images of the containers.
GiteaMirror commented 2026-04-22 08:20:08 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

Trying all the above now. Thank you so much. Appreciate all the support

<!-- gh-comment-id:2247755424 --> @nicholhai commented on GitHub (Jul 24, 2024): Trying all the above now. Thank you so much. Appreciate all the support
GiteaMirror commented 2026-04-22 08:20:08 -05:00
Author
Owner
Copy Link

@Cephra commented on GitHub (Jul 24, 2024):

ollama image contains Nvidia support.

Ah, thanks for clarifying! Haven't used an nvidia gpu myself.

<!-- gh-comment-id:2247759340 --> @Cephra commented on GitHub (Jul 24, 2024): > ollama image contains Nvidia support. Ah, thanks for clarifying! Haven't used an nvidia gpu myself.
GiteaMirror commented 2026-04-22 08:20:09 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

Also something you should try: After removing the containers, issue a docker image prune -a to ensure the old, possibly outdated, images are also deleted.

IIRC simply issuing docker rm -f does not remove the images of the containers.

Whoa. that removed a whole whack of containers. Thanks @Cephra

<!-- gh-comment-id:2247768871 --> @nicholhai commented on GitHub (Jul 24, 2024): > Also something you should try: After removing the containers, issue a `docker image prune -a` to ensure the old, possibly outdated, images are also deleted. > > IIRC simply issuing `docker rm -f` does not remove the images of the containers. Whoa. that removed a whole whack of containers. Thanks @Cephra
GiteaMirror commented 2026-04-22 08:20:10 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

sudo docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data --name open-webui --restart always --pull always ghcr.io/open-webui/open-webui:main

  • stopped all docker containers
  • removed them
  • pruned
  • started then both again

INFO: 192.168.3.17:53485 - "GET /ollama/config HTTP/1.1" 200 OK
INFO [apps.openai.main] get_all_models()
INFO [apps.ollama.main] get_all_models()
INFO [apps.openai.main] get_all_models()
INFO [apps.ollama.main] get_all_models()
ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused]
INFO [apps.ollama.main] get_all_models()
ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused]
INFO [apps.ollama.main] get_all_models()
ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused]
ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused]
ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused]
INFO: 192.168.3.17:53455 - "GET /ollama/api/version HTTP/1.1" 500 Internal Server Error
INFO: 192.168.3.17:53485 - "GET /ollama/urls HTTP/1.1" 200 OK

Going to try Cephy's commands now

<!-- gh-comment-id:2247782005 --> @nicholhai commented on GitHub (Jul 24, 2024): > sudo docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data --name open-webui --restart always --pull always ghcr.io/open-webui/open-webui:main - stopped all docker containers - removed them - pruned - started then both again INFO: 192.168.3.17:53485 - "GET /ollama/config HTTP/1.1" 200 OK INFO [apps.openai.main] get_all_models() INFO [apps.ollama.main] get_all_models() INFO [apps.openai.main] get_all_models() INFO [apps.ollama.main] get_all_models() ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused] INFO [apps.ollama.main] get_all_models() ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused] INFO [apps.ollama.main] get_all_models() ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused] ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused] ERROR [apps.ollama.main] Connection error: Cannot connect to host localhost:11434 ssl:default [Connection refused] INFO: 192.168.3.17:53455 - "GET /ollama/api/version HTTP/1.1" 500 Internal Server Error INFO: 192.168.3.17:53485 - "GET /ollama/urls HTTP/1.1" 200 OK _**Going to try Cephy's commands now**_
GiteaMirror commented 2026-04-22 08:20:11 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

Can you try this:

To start ollama: docker run --restart always -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

To start open-webui: docker run -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 --name open-webui --restart always ghcr.io/open-webui/open-webui:main

These are the commands I am using.

NOTE: Make sure you change the image in the ollama command to suit your needs. I saw you have an nvidia card so you'll have to adjust the image in order to get GPU acceleration.

docker: Error response from daemon: error gathering device information while adding custom device "/dev/kfd": no such file or directory.

I am still learning docker ...

<!-- gh-comment-id:2247792231 --> @nicholhai commented on GitHub (Jul 24, 2024): > Can you try this: > > To start ollama: `docker run --restart always -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama` > > To start open-webui: `docker run -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 --name open-webui --restart always ghcr.io/open-webui/open-webui:main` > > These are the commands I am using. > > **NOTE**: Make sure you change the image in the ollama command to suit your needs. I saw you have an nvidia card so you'll have to adjust the image in order to get GPU acceleration. **docker: Error response from daemon: error gathering device information while adding custom device "/dev/kfd": no such file or directory.** I am still learning docker ...
GiteaMirror commented 2026-04-22 08:20:12 -05:00
Author
Owner
Copy Link

@Cephra commented on GitHub (Jul 24, 2024):

Can you try this:
To start ollama: docker run --restart always -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
To start open-webui: docker run -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 --name open-webui --restart always ghcr.io/open-webui/open-webui:main
These are the commands I am using.
NOTE: Make sure you change the image in the ollama command to suit your needs. I saw you have an nvidia card so you'll have to adjust the image in order to get GPU acceleration.

docker: Error response from daemon: error gathering device information while adding custom device "/dev/kfd": no such file or directory.

I am still learning docker ...

Oh okay.. maybe those switches are Linux exclusive. For ollama try this command instead: docker run --restart always --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Update: The two switches are only needed with AMD GPUs. For Nvidia --gpus=all is enough, like @rick-github pointed out.

<!-- gh-comment-id:2247800474 --> @Cephra commented on GitHub (Jul 24, 2024): > > Can you try this: > > To start ollama: `docker run --restart always -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama` > > To start open-webui: `docker run -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 --name open-webui --restart always ghcr.io/open-webui/open-webui:main` > > These are the commands I am using. > > **NOTE**: Make sure you change the image in the ollama command to suit your needs. I saw you have an nvidia card so you'll have to adjust the image in order to get GPU acceleration. > > **docker: Error response from daemon: error gathering device information while adding custom device "/dev/kfd": no such file or directory.** > > I am still learning docker ... Oh okay.. maybe those switches are Linux exclusive. For ollama try this command instead: `docker run --restart always --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama` **Update**: The two switches are only needed with AMD GPUs. For Nvidia `--gpus=all` is enough, like @rick-github pointed out.
GiteaMirror commented 2026-04-22 08:20:13 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

Nope. the above won't even launch the localhost:3000

<!-- gh-comment-id:2247800999 --> @nicholhai commented on GitHub (Jul 24, 2024): Nope. the above won't even launch the localhost:3000
GiteaMirror commented 2026-04-22 08:20:14 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 24, 2024):

You have an nvidia card, --gpus all is all you need.

<!-- gh-comment-id:2247801106 --> @rick-github commented on GitHub (Jul 24, 2024): You have an nvidia card, `--gpus all` is all you need.
GiteaMirror commented 2026-04-22 08:20:16 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 24, 2024):

Maybe there is saved state.

docker stop open-webui
docker rm open-webui
docker volume rm open-webui
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data --name open-webui --restart always --pull always ghcr.io/open-webui/open-webui:main
<!-- gh-comment-id:2247806767 --> @rick-github commented on GitHub (Jul 24, 2024): Maybe there is saved state. ``` docker stop open-webui docker rm open-webui docker volume rm open-webui docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data --name open-webui --restart always --pull always ghcr.io/open-webui/open-webui:main ```
GiteaMirror commented 2026-04-22 08:20:16 -05:00
Author
Owner
Copy Link

@Cephra commented on GitHub (Jul 24, 2024):

I'm pretty sure @rick-github is right! There's probably still a volume with runtime config for open-webui.

<!-- gh-comment-id:2247808310 --> @Cephra commented on GitHub (Jul 24, 2024): I'm pretty sure @rick-github is right! There's probably still a volume with runtime config for open-webui.
GiteaMirror commented 2026-04-22 08:20:17 -05:00
Author
Owner
Copy Link

@Cephra commented on GitHub (Jul 24, 2024):

Nope. the above won't even launch the localhost:3000

You mean the container isn't even started? Do you get any error message or something?

<!-- gh-comment-id:2247811758 --> @Cephra commented on GitHub (Jul 24, 2024): > Nope. the above won't even launch the localhost:3000 You mean the container isn't even started? Do you get any error message or something?
GiteaMirror commented 2026-04-22 08:20:18 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

Nope. the above won't even launch the localhost:3000

You mean the container isn't even started? Do you get any error message or something?

It started. Just went to "Page not found..."

<!-- gh-comment-id:2247840851 --> @nicholhai commented on GitHub (Jul 24, 2024): > > Nope. the above won't even launch the localhost:3000 > > You mean the container isn't even started? Do you get any error message or something? It started. Just went to "Page not found..."
GiteaMirror commented 2026-04-22 08:20:19 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

Maybe there is saved state.

docker stop open-webui
docker rm open-webui
docker volume rm open-webui
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data --name open-webui --restart always --pull always ghcr.io/open-webui/open-webui:main

We have a winner! I think the "docker volume rm open-webui" did the trick. I tried a 12b model and it worked. Now pulling a 70b model to try.

Strangely I had to re-register my admin account HOWEVER, all the previously downloaded models were already there

<!-- gh-comment-id:2247844288 --> @nicholhai commented on GitHub (Jul 24, 2024): > Maybe there is saved state. > > ``` > docker stop open-webui > docker rm open-webui > docker volume rm open-webui > docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data --name open-webui --restart always --pull always ghcr.io/open-webui/open-webui:main > ``` We have a winner! I think the "docker volume rm open-webui" did the trick. I tried a 12b model and it worked. Now pulling a 70b model to try. **Strangely I had to re-register my admin account HOWEVER, all the previously downloaded models were already there**
GiteaMirror commented 2026-04-22 08:20:20 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 24, 2024):

The models are stored in the ollama container, the user info is stored in the open-webui container, which was deleted with the docker volume rm open-webui command.

<!-- gh-comment-id:2247851027 --> @rick-github commented on GitHub (Jul 24, 2024): The models are stored in the ollama container, the user info is stored in the open-webui container, which was deleted with the `docker volume rm open-webui` command.
GiteaMirror commented 2026-04-22 08:20:21 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 24, 2024):

Be aware that a 70b model will not fit on your GPU and ollama will load most of it in RAM and use both GPU and CPU for inference, so it will run pretty slow.

<!-- gh-comment-id:2247879019 --> @rick-github commented on GitHub (Jul 24, 2024): Be aware that a 70b model will not fit on your GPU and ollama will load most of it in RAM and use both GPU and CPU for inference, so it will run pretty slow.
GiteaMirror commented 2026-04-22 08:20:21 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

Ah. the download stopped at 60% of so with EOF error

<!-- gh-comment-id:2247910374 --> @nicholhai commented on GitHub (Jul 24, 2024): Ah. the download stopped at 60% of so with EOF error
GiteaMirror commented 2026-04-22 08:20:22 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

Be aware that a 70b model will not fit on your GPU and ollama will load most of it in RAM and use both GPU and CPU for inference, so it will run pretty slow.

the 70b runs (slow) on my Mac Studio.. what would I need to run the large models? In terms of hardware?

<!-- gh-comment-id:2247913045 --> @nicholhai commented on GitHub (Jul 24, 2024): > Be aware that a 70b model will not fit on your GPU and ollama will load most of it in RAM and use both GPU and CPU for inference, so it will run pretty slow. the 70b runs (slow) on my Mac Studio.. what would I need to run the large models? In terms of hardware?
GiteaMirror commented 2026-04-22 08:20:23 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 24, 2024):

ollama downloads chunks so if you restart the download it should start where the previous download stopped. Make sure you have plenty of disk available in /var/lib/docker.

The size of the model is shown on the ollama model library, eg llama3.1:70b is 40G so you would need 3 RTX 4060 Ti cards or 2 A10s or one A40.

<!-- gh-comment-id:2247939421 --> @rick-github commented on GitHub (Jul 24, 2024): ollama downloads chunks so if you restart the download it should start where the previous download stopped. Make sure you have plenty of disk available in /var/lib/docker. The size of the model is shown on the [ollama model library](https://ollama.com/library/), eg llama3.1:70b is 40G so you would need 3 RTX 4060 Ti cards or 2 A10s or one A40.
GiteaMirror commented 2026-04-22 08:20:23 -05:00
Author
Owner
Copy Link

@Cephra commented on GitHub (Jul 24, 2024):

@nicholhai Can you please close the issue if it's resolved?

<!-- gh-comment-id:2248216388 --> @Cephra commented on GitHub (Jul 24, 2024): @nicholhai Can you please close the issue if it's resolved?
GiteaMirror commented 2026-04-22 08:20:24 -05:00
Author
Owner
Copy Link

@nicholhai commented on GitHub (Jul 24, 2024):

Will do. Thank you for all the assistance.

<!-- gh-comment-id:2248322855 --> @nicholhai commented on GitHub (Jul 24, 2024): Will do. Thank you for all the assistance.
GiteaMirror commented 2026-04-22 08:20:25 -05:00
Author
Owner
Copy Link

@HackHussy commented on GitHub (Aug 19, 2024):

ollama 500, Llama3.1:70b for me it was lack of memory expected 28.8gb actual 8.8gb
Sadge I'll try quantization

<!-- gh-comment-id:2296739310 --> @HackHussy commented on GitHub (Aug 19, 2024): ollama 500, Llama3.1:70b for me it was lack of memory expected 28.8gb actual 8.8gb Sadge I'll try quantization
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#29437
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?