github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #4222] server not responding #2632

New Issue
Closed
opened 2026-04-12 12:58:54 -05:00 by GiteaMirror · 6 comments
GiteaMirror commented 2026-04-12 12:58:54 -05:00
Owner
Copy Link

Originally created by @thomassrour on GitHub (May 7, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/4222

Originally assigned to: @dhiltgen on GitHub.

What is the issue?

Hello,

I have trouble reaching my ollama container. I have tried using the images for 0.1.32 and 0.1.33, as some users reported bugs 0.1.33 but it doesn't work on either. Here is the output of docker logs, when trying mixtral (I have also tried llama3, same result) :

time=2024-05-07T07:33:21.130Z level=INFO source=images.go:817 msg="total blobs: 10"
time=2024-05-07T07:33:21.134Z level=INFO source=images.go:824 msg="total unused blobs removed: 0"
time=2024-05-07T07:33:21.135Z level=INFO source=routes.go:1143 msg="Listening on [::]:11434 (version 0.1.32)"
time=2024-05-07T07:33:21.136Z level=INFO source=payload.go:28 msg="extracting embedded files" dir=/tmp/ollama3873501864/runners
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gz
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gz
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gz
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublas.so.11.gz
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublasLt.so.11.gz
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcudart.so.11.0.gz
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/ollama_llama_server.gz
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/deps.txt.gz
time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/ollama_llama_server.gz
time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu
time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx
time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx2
time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cuda_v11
time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/rocm_v60002
time=2024-05-07T07:33:25.457Z level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cuda_v11 rocm_v60002 cpu cpu_avx cpu_avx2]"
time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:42 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2024-05-07T07:33:25.457Z level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-05-07T07:33:25.457Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*"
time=2024-05-07T07:33:25.457Z level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/tmp/ollama3873501864/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers//libcudart.so /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so* /usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so**]"
time=2024-05-07T07:33:25.462Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0]"
wiring cudart library functions in /tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0
dlsym: cudaSetDevice
dlsym: cudaDeviceSynchronize
dlsym: cudaDeviceReset
dlsym: cudaMemGetInfo
dlsym: cudaGetDeviceCount
dlsym: cudaDeviceGetAttribute
dlsym: cudaDriverGetVersion
CUDA driver version: 12-0
time=2024-05-07T07:33:25.495Z level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart"
time=2024-05-07T07:33:25.495Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
[0] CUDA totalMem 85895020544
[0] CUDA freeMem 79269199872
time=2024-05-07T07:33:25.661Z level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.0"
releasing cudart library
time=2024-05-07T07:33:30.257Z level=DEBUG source=gguf.go:57 msg="model = &llm.gguf{containerGGUF:(llm.containerGGUF)(0xc000426c80), kv:llm.KV{}, tensors:[]llm.Tensor(nil), parameters:0x0}"
time=2024-05-07T07:33:30.522Z level=DEBUG source=gguf.go:193 msg="general.architecture = llama"
time=2024-05-07T07:33:30.528Z level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-05-07T07:33:30.528Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so"
time=2024-05-07T07:33:30.528Z level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/tmp/ollama3873501864/runners/cuda/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers//libcudart.so /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so* /usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so**]"
time=2024-05-07T07:33:30.529Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0]"
wiring cudart library functions in /tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0
dlsym: cudaSetDevice
dlsym: cudaDeviceSynchronize
dlsym: cudaDeviceReset
dlsym: cudaMemGetInfo
dlsym: cudaGetDeviceCount
dlsym: cudaDeviceGetAttribute
dlsym: cudaDriverGetVersion
CUDA driver version: 12-0
time=2024-05-07T07:33:30.530Z level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart"
time=2024-05-07T07:33:30.530Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
[0] CUDA totalMem 85895020544
[0] CUDA freeMem 79269199872
time=2024-05-07T07:33:30.674Z level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.0"
releasing cudart library
time=2024-05-07T07:33:30.719Z level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-05-07T07:33:30.719Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*"
time=2024-05-07T07:33:30.719Z level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/tmp/ollama3873501864/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers//libcudart.so /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so* /usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so**]"
time=2024-05-07T07:33:30.720Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0]"
wiring cudart library functions in /tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0
dlsym: cudaSetDevice
dlsym: cudaDeviceSynchronize
dlsym: cudaDeviceReset
dlsym: cudaMemGetInfo
dlsym: cudaGetDeviceCount
dlsym: cudaDeviceGetAttribute
dlsym: cudaDriverGetVersion
CUDA driver version: 12-0
time=2024-05-07T07:33:30.721Z level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart"
time=2024-05-07T07:33:30.721Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
[0] CUDA totalMem 85895020544
[0] CUDA freeMem 79269199872
time=2024-05-07T07:33:30.869Z level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.0"
releasing cudart library
time=2024-05-07T07:33:30.916Z level=INFO source=server.go:127 msg="offload to gpu" reallayers=33 layers=33 required="26042.6 MiB" used="26042.6 MiB" available="75597.0 MiB" kv="256.0 MiB" fulloffload="184.0 MiB" partialoffload="935.0 MiB"
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx2
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cuda_v11
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/rocm_v60002
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx2
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cuda_v11
time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/rocm_v60002
time=2024-05-07T07:33:30.916Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-05-07T07:33:30.918Z level=DEBUG source=server.go:259 msg="LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/tmp/ollama3873501864/runners/cuda_v11"
time=2024-05-07T07:33:30.918Z level=INFO source=server.go:264 msg="starting llama server" cmd="/tmp/ollama3873501864/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-e9e56e8bb5f0fcd4860675e6837a8f6a94e659f5fa7dce6a1076279336320f2b --ctx-size 2048 --batch-size 512 --embedding --log-format json --n-gpu-layers 33 --verbose --port 33787"
time=2024-05-07T07:33:30.919Z level=INFO source=server.go:389 msg="waiting for llama runner to start responding"
{"function":"server_params_parse","level":"WARN","line":2494,"msg":"server.cpp is not built with verbose logging.","tid":"140005422133248","timestamp":1715067210}
time=2024-05-07T07:33:30.970Z level=DEBUG source=server.go:420 msg="server not yet available" error="health resp: Get "http://127.0.0.1:33787/health": dial tcp 127.0.0.1:33787: connect: connection refused"
{"build":1,"commit":"7593639","function":"main","level":"INFO","line":2819,"msg":"build info","tid":"140005422133248","timestamp":1715067210}
{"function":"main","level":"INFO","line":2822,"msg":"system info","n_threads":4,"n_threads_batch":-1,"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | ","tid":"140005422133248","timestamp":1715067210,"total_threads":8}
llama_model_loader: loaded meta data with 26 key-value pairs and 995 tensors from /root/.ollama/models/blobs/sha256-e9e56e8bb5f0fcd4860675e6837a8f6a94e659f5fa7dce6a1076279336320f2b (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = mistralai
llama_model_loader: - kv 2: llama.context_length u32 = 32768
llama_model_loader: - kv 3: llama.embedding_length u32 = 4096
llama_model_loader: - kv 4: llama.block_count u32 = 32
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 7: llama.attention.head_count u32 = 32
llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 9: llama.expert_count u32 = 8
llama_model_loader: - kv 10: llama.expert_used_count u32 = 2
llama_model_loader: - kv 11: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 12: llama.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 13: general.file_type u32 = 2
llama_model_loader: - kv 14: tokenizer.ggml.model str = llama
llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32000] = ["", "", "", "<0x00>", "<...
llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 18: tokenizer.ggml.merges arr[str,58980] = ["▁ t", "i n", "e r", "▁ a", "h e...
llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 21: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 24: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...
llama_model_loader: - kv 25: general.quantization_version u32 = 2
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type f16: 32 tensors
llama_model_loader: - type q4_0: 833 tensors
llama_model_loader: - type q8_0: 64 tensors
llama_model_loader: - type q6_K: 1 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32000
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 8
llm_load_print_meta: n_expert_used = 2
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 8x7B
llm_load_print_meta: model ftype = Q4_0
llm_load_print_meta: model params = 46.70 B
llm_load_print_meta: model size = 24.62 GiB (4.53 BPW)
llm_load_print_meta: general.name = mistralai
llm_load_print_meta: BOS token = 1 ''
llm_load_print_meta: EOS token = 2 ''
llm_load_print_meta: UNK token = 0 ''
llm_load_print_meta: LF token = 13 '<0x0A>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: yes
ggml_cuda_init: CUDA_USE_TENSOR_CORES: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: GRID A100D-80C, compute capability 8.0, VMM: no
time=2024-05-07T07:33:31.220Z level=DEBUG source=server.go:420 msg="server not yet available" error="server not responding"
llm_load_tensors: ggml ctx size = 0.96 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors: CUDA_Host buffer size = 70.31 MiB
llm_load_tensors: CUDA0 buffer size = 25145.55 MiB
..................................................time=2024-05-07T07:35:23.693Z level=DEBUG source=server.go:420 msg="server not yet available" error="health resp: Get "http://127.0.0.1:33787/health": dial tcp 127.0.0.1:33787: i/o timeout"
time=2024-05-07T07:35:23.894Z level=DEBUG source=server.go:420 msg="server not yet available" error="server not responding"
`

Thanks for your help

OS

Docker

GPU

Nvidia

CPU

No response

Ollama version

0.1.32, 0.1.33

Originally created by @thomassrour on GitHub (May 7, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/4222 Originally assigned to: @dhiltgen on GitHub. ### What is the issue? Hello, I have trouble reaching my ollama container. I have tried using the images for 0.1.32 and 0.1.33, as some users reported bugs 0.1.33 but it doesn't work on either. Here is the output of docker logs, when trying mixtral (I have also tried llama3, same result) : time=2024-05-07T07:33:21.130Z level=INFO source=images.go:817 msg="total blobs: 10" time=2024-05-07T07:33:21.134Z level=INFO source=images.go:824 msg="total unused blobs removed: 0" time=2024-05-07T07:33:21.135Z level=INFO source=routes.go:1143 msg="Listening on [::]:11434 (version 0.1.32)" time=2024-05-07T07:33:21.136Z level=INFO source=payload.go:28 msg="extracting embedded files" dir=/tmp/ollama3873501864/runners time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gz time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gz time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gz time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublas.so.11.gz time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublasLt.so.11.gz time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcudart.so.11.0.gz time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/ollama_llama_server.gz time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/deps.txt.gz time=2024-05-07T07:33:21.136Z level=DEBUG source=payload.go:160 msg=extracting variant=rocm_v60002 file=build/linux/x86_64/rocm_v60002/bin/ollama_llama_server.gz time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx2 time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cuda_v11 time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/rocm_v60002 time=2024-05-07T07:33:25.457Z level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cuda_v11 rocm_v60002 cpu cpu_avx cpu_avx2]" time=2024-05-07T07:33:25.457Z level=DEBUG source=payload.go:42 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY" time=2024-05-07T07:33:25.457Z level=INFO source=gpu.go:121 msg="Detecting GPU type" time=2024-05-07T07:33:25.457Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*" time=2024-05-07T07:33:25.457Z level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/tmp/ollama3873501864/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so* /usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so**]" time=2024-05-07T07:33:25.462Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0]" wiring cudart library functions in /tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0 dlsym: cudaSetDevice dlsym: cudaDeviceSynchronize dlsym: cudaDeviceReset dlsym: cudaMemGetInfo dlsym: cudaGetDeviceCount dlsym: cudaDeviceGetAttribute dlsym: cudaDriverGetVersion CUDA driver version: 12-0 time=2024-05-07T07:33:25.495Z level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart" time=2024-05-07T07:33:25.495Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" [0] CUDA totalMem 85895020544 [0] CUDA freeMem 79269199872 time=2024-05-07T07:33:25.661Z level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.0" releasing cudart library time=2024-05-07T07:33:30.257Z level=DEBUG source=gguf.go:57 msg="model = &llm.gguf{containerGGUF:(*llm.containerGGUF)(0xc000426c80), kv:llm.KV{}, tensors:[]*llm.Tensor(nil), parameters:0x0}" time=2024-05-07T07:33:30.522Z level=DEBUG source=gguf.go:193 msg="general.architecture = llama" time=2024-05-07T07:33:30.528Z level=INFO source=gpu.go:121 msg="Detecting GPU type" time=2024-05-07T07:33:30.528Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*" time=2024-05-07T07:33:30.528Z level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/tmp/ollama3873501864/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so* /usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so**]" time=2024-05-07T07:33:30.529Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0]" wiring cudart library functions in /tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0 dlsym: cudaSetDevice dlsym: cudaDeviceSynchronize dlsym: cudaDeviceReset dlsym: cudaMemGetInfo dlsym: cudaGetDeviceCount dlsym: cudaDeviceGetAttribute dlsym: cudaDriverGetVersion CUDA driver version: 12-0 time=2024-05-07T07:33:30.530Z level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart" time=2024-05-07T07:33:30.530Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" [0] CUDA totalMem 85895020544 [0] CUDA freeMem 79269199872 time=2024-05-07T07:33:30.674Z level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.0" releasing cudart library time=2024-05-07T07:33:30.719Z level=INFO source=gpu.go:121 msg="Detecting GPU type" time=2024-05-07T07:33:30.719Z level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*" time=2024-05-07T07:33:30.719Z level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/tmp/ollama3873501864/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so* /usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so**]" time=2024-05-07T07:33:30.720Z level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [/tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0]" wiring cudart library functions in /tmp/ollama3873501864/runners/cuda_v11/libcudart.so.11.0 dlsym: cudaSetDevice dlsym: cudaDeviceSynchronize dlsym: cudaDeviceReset dlsym: cudaMemGetInfo dlsym: cudaGetDeviceCount dlsym: cudaDeviceGetAttribute dlsym: cudaDriverGetVersion CUDA driver version: 12-0 time=2024-05-07T07:33:30.721Z level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart" time=2024-05-07T07:33:30.721Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" [0] CUDA totalMem 85895020544 [0] CUDA freeMem 79269199872 time=2024-05-07T07:33:30.869Z level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.0" releasing cudart library time=2024-05-07T07:33:30.916Z level=INFO source=server.go:127 msg="offload to gpu" reallayers=33 layers=33 required="26042.6 MiB" used="26042.6 MiB" available="75597.0 MiB" kv="256.0 MiB" fulloffload="184.0 MiB" partialoffload="935.0 MiB" time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx2 time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cuda_v11 time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/rocm_v60002 time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cpu_avx2 time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/cuda_v11 time=2024-05-07T07:33:30.916Z level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3873501864/runners/rocm_v60002 time=2024-05-07T07:33:30.916Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-05-07T07:33:30.918Z level=DEBUG source=server.go:259 msg="LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/tmp/ollama3873501864/runners/cuda_v11" time=2024-05-07T07:33:30.918Z level=INFO source=server.go:264 msg="starting llama server" cmd="/tmp/ollama3873501864/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-e9e56e8bb5f0fcd4860675e6837a8f6a94e659f5fa7dce6a1076279336320f2b --ctx-size 2048 --batch-size 512 --embedding --log-format json --n-gpu-layers 33 --verbose --port 33787" time=2024-05-07T07:33:30.919Z level=INFO source=server.go:389 msg="waiting for llama runner to start responding" {"function":"server_params_parse","level":"WARN","line":2494,"msg":"server.cpp is not built with verbose logging.","tid":"140005422133248","timestamp":1715067210} time=2024-05-07T07:33:30.970Z level=DEBUG source=server.go:420 msg="server not yet available" error="health resp: Get \"http://127.0.0.1:33787/health\": dial tcp 127.0.0.1:33787: connect: connection refused" {"build":1,"commit":"7593639","function":"main","level":"INFO","line":2819,"msg":"build info","tid":"140005422133248","timestamp":1715067210} {"function":"main","level":"INFO","line":2822,"msg":"system info","n_threads":4,"n_threads_batch":-1,"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | ","tid":"140005422133248","timestamp":1715067210,"total_threads":8} llama_model_loader: loaded meta data with 26 key-value pairs and 995 tensors from /root/.ollama/models/blobs/sha256-e9e56e8bb5f0fcd4860675e6837a8f6a94e659f5fa7dce6a1076279336320f2b (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.name str = mistralai llama_model_loader: - kv 2: llama.context_length u32 = 32768 llama_model_loader: - kv 3: llama.embedding_length u32 = 4096 llama_model_loader: - kv 4: llama.block_count u32 = 32 llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 7: llama.attention.head_count u32 = 32 llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 9: llama.expert_count u32 = 8 llama_model_loader: - kv 10: llama.expert_used_count u32 = 2 llama_model_loader: - kv 11: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 12: llama.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 13: general.file_type u32 = 2 llama_model_loader: - kv 14: tokenizer.ggml.model str = llama llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32000] = ["<unk>", "<s>", "</s>", "<0x00>", "<... llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... llama_model_loader: - kv 18: tokenizer.ggml.merges arr[str,58980] = ["▁ t", "i n", "e r", "▁ a", "h e... llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 21: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 24: tokenizer.chat_template str = {{ bos_token }}{% for message in mess... llama_model_loader: - kv 25: general.quantization_version u32 = 2 llama_model_loader: - type f32: 65 tensors llama_model_loader: - type f16: 32 tensors llama_model_loader: - type q4_0: 833 tensors llama_model_loader: - type q8_0: 64 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special tokens definition check successful ( 259/32000 ). llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32000 llm_load_print_meta: n_merges = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: n_expert = 8 llm_load_print_meta: n_expert_used = 2 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 1000000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = 8x7B llm_load_print_meta: model ftype = Q4_0 llm_load_print_meta: model params = 46.70 B llm_load_print_meta: model size = 24.62 GiB (4.53 BPW) llm_load_print_meta: general.name = mistralai llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 13 '<0x0A>' ggml_cuda_init: GGML_CUDA_FORCE_MMQ: yes ggml_cuda_init: CUDA_USE_TENSOR_CORES: no ggml_cuda_init: found 1 CUDA devices: Device 0: GRID A100D-80C, compute capability 8.0, VMM: no time=2024-05-07T07:33:31.220Z level=DEBUG source=server.go:420 msg="server not yet available" error="server not responding" llm_load_tensors: ggml ctx size = 0.96 MiB llm_load_tensors: offloading 32 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 33/33 layers to GPU llm_load_tensors: CUDA_Host buffer size = 70.31 MiB llm_load_tensors: CUDA0 buffer size = 25145.55 MiB ..................................................time=2024-05-07T07:35:23.693Z level=DEBUG source=server.go:420 msg="server not yet available" error="health resp: Get \"http://127.0.0.1:33787/health\": dial tcp 127.0.0.1:33787: i/o timeout" time=2024-05-07T07:35:23.894Z level=DEBUG source=server.go:420 msg="server not yet available" error="server not responding" ` Thanks for your help ### OS Docker ### GPU Nvidia ### CPU _No response_ ### Ollama version 0.1.32, 0.1.33
GiteaMirror added the bugneeds more info labels 2026-04-12 12:58:54 -05:00
GiteaMirror commented 2026-04-12 12:58:55 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (May 8, 2024):

The mixtral model is quite large, and it looks like your GPU has 80G available, so I'm guessing this is a cloud instance. We've seen that cloud instances can have slow I/O and take quite a long time to load larger models. We currently have a 10m timeout set in the code, but have heard some users report it taking longer than 10m to fully load larger models in some cases. Did you see it hit the 10m timeout eventually? You might also want to open another terminal into your instance and check performance metrics.

<!-- gh-comment-id:2101325851 --> @dhiltgen commented on GitHub (May 8, 2024): The mixtral model is quite large, and it looks like your GPU has 80G available, so I'm guessing this is a cloud instance. We've seen that cloud instances can have slow I/O and take quite a long time to load larger models. We currently have a 10m timeout set in the code, but have heard some users report it taking longer than 10m to fully load larger models in some cases. Did you see it hit the 10m timeout eventually? You might also want to open another terminal into your instance and check performance metrics.
GiteaMirror commented 2026-04-12 12:58:55 -05:00
Author
Owner
Copy Link

@thomassrour commented on GitHub (May 10, 2024):

Hello @dhiltgen, thank you for your answer, ollama now detects the GPU but inference is extremely slow (about 2 words in 10 minutes). The setup I have is a A100-80GB GPU virtualized with nvidia GRID, with driver version 525 and CUDA 12. GPU usage is at 0%, but it is technically using it, otherwise it would be NA.

Would you have any guidance on how to proceed ? Thanks again

<!-- gh-comment-id:2104287643 --> @thomassrour commented on GitHub (May 10, 2024): Hello @dhiltgen, thank you for your answer, ollama now detects the GPU but inference is extremely slow (about 2 words in 10 minutes). The setup I have is a A100-80GB GPU virtualized with nvidia GRID, with driver version 525 and CUDA 12. GPU usage is at 0%, but it is technically using it, otherwise it would be NA. Would you have any guidance on how to proceed ? Thanks again
GiteaMirror commented 2026-04-12 12:58:56 -05:00
Author
Owner
Copy Link

@DeusNexus commented on GitHub (May 14, 2024):

I would like to add that it's best to completely disable any swap memory. This has tremendously helped me so far.

<!-- gh-comment-id:2109614594 --> @DeusNexus commented on GitHub (May 14, 2024): I would like to add that it's best to completely disable any swap memory. This has tremendously helped me so far.
GiteaMirror commented 2026-04-12 12:58:56 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (May 16, 2024):

@thomassrour it sounds like it did eventually load, but took a long time to do so. Once it loaded, you didn't see performance you were expecting. Can you share the server log (or check for the line that looks something like this llm_load_tensors: offloaded 33/33 layers to GPU and see how many layers loaded? My suspicion is you are loading a very large model, and it doesn't fully fit within the GPUs VRAM. As a result, it will be partially processing on the CPU, which will slow down inference significantly. You can try running a smaller model that fully fits in the GPU, or you can use a larger VM type so it has more CPU resources allocated.

Also, if you upgrade to 0.1.38 (just shipped) we've added an ollama ps which you can run to see more information about how much of the model is loaded into GPU to avoid having to poke around in the server log.

<!-- gh-comment-id:2113685304 --> @dhiltgen commented on GitHub (May 16, 2024): @thomassrour it sounds like it did eventually load, but took a long time to do so. Once it loaded, you didn't see performance you were expecting. Can you share the server log (or check for the line that looks something like this `llm_load_tensors: offloaded 33/33 layers to GPU` and see how many layers loaded? My suspicion is you are loading a very large model, and it doesn't fully fit within the GPUs VRAM. As a result, it will be partially processing on the CPU, which will slow down inference significantly. You can try running a smaller model that fully fits in the GPU, or you can use a larger VM type so it has more CPU resources allocated. Also, if you upgrade to 0.1.38 (just shipped) we've added an `ollama ps` which you can run to see more information about how much of the model is loaded into GPU to avoid having to poke around in the server log.
GiteaMirror commented 2026-04-12 12:58:56 -05:00
Author
Owner
Copy Link

@DeusNexus commented on GitHub (May 16, 2024):

Actually most users that run into issue of not using enough layers can occur if num_batch size is too small or too high.
Try num_batch=32 or num_batch=64 and leave num_gpu at default or try different values. These two worked best for me.

<!-- gh-comment-id:2113996394 --> @DeusNexus commented on GitHub (May 16, 2024): Actually most users that run into issue of not using enough layers can occur if num_batch size is too small or too high. Try num_batch=32 or num_batch=64 and leave num_gpu at default or try different values. These two worked best for me.
GiteaMirror commented 2026-04-12 12:58:56 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (May 31, 2024):

It sounds like this has probably been resolved. If you're still having unexpected performance, please make sure to upgrade to the latest version, and if that doesn't resolve it, let us know and I'll re-open the issue.

<!-- gh-comment-id:2143011124 --> @dhiltgen commented on GitHub (May 31, 2024): It sounds like this has probably been resolved. If you're still having unexpected performance, please make sure to upgrade to the latest version, and if that doesn't resolve it, let us know and I'll re-open the issue.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug needs more info
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#2632
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?