github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 12:48:05 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #2851] Troubleshooting Dify Connection to Ollama Service: CPU vs. GPU Differences #1736

New Issue
Closed
opened 2026-04-12 11:43:04 -05:00 by GiteaMirror · 14 comments
GiteaMirror commented 2026-04-12 11:43:04 -05:00
Owner
Copy Link

Originally created by @xiaotianfotos on GitHub (Mar 1, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2851

my problem is

i 'm using Dify to connect to ollama service

when using ollama API on Dify,always loaded to CPU memory,

but i tried using
curl http://localhost:11434/api/generate -d '{
"model": "qwen:14b",
"prompt": "Why is the sky blue?"
}'
loaded into GPU

log description
1st run :ollama API via Dify
2nd run : curl

i also tried using OpenAI-API-compatible on Dify,loaded to GPU

log as following
time=2024-03-01T10:34:19.796+08:00 level=INFO source=images.go:710 msg="total blobs: 5"
time=2024-03-01T10:34:19.796+08:00 level=INFO source=images.go:717 msg="total unused blobs removed: 0"
time=2024-03-01T10:34:19.797+08:00 level=INFO source=routes.go:1019 msg="Listening on [::]:11434 (version 0.1.27)"
time=2024-03-01T10:34:19.797+08:00 level=INFO source=payload_common.go:107 msg="Extracting dynamic libraries..."
time=2024-03-01T10:34:22.664+08:00 level=INFO source=payload_common.go:146 msg="Dynamic LLM libraries [rocm_v5 cpu_avx rocm_v6 cpu_avx2 cuda_v11 cpu]"
time=2024-03-01T10:34:22.664+08:00 level=INFO source=gpu.go:94 msg="Detecting GPU type"
time=2024-03-01T10:34:22.664+08:00 level=INFO source=gpu.go:265 msg="Searching for GPU management library libnvidia-ml.so"
time=2024-03-01T10:34:22.666+08:00 level=INFO source=gpu.go:311 msg="Discovered GPU libraries: [/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.535.154.05]"
time=2024-03-01T10:34:22.672+08:00 level=INFO source=gpu.go:99 msg="Nvidia GPU detected"
time=2024-03-01T10:34:22.672+08:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-01T10:34:22.678+08:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.9"
time=2024-03-01T10:34:38.704+08:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-01T10:34:38.704+08:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.9"
time=2024-03-01T10:34:38.704+08:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-01T10:34:38.704+08:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.9"
time=2024-03-01T10:34:38.704+08:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
loading library /tmp/ollama1445230373/cuda_v11/libext_server.so
time=2024-03-01T10:34:38.715+08:00 level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: /tmp/ollama1445230373/cuda_v11/libext_server.so"
time=2024-03-01T10:34:38.715+08:00 level=INFO source=dyn_ext_server.go:150 msg="Initializing llama server"
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: yes
ggml_init_cublas: CUDA_USE_TENSOR_CORES: no
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
llama_model_loader: loaded meta data with 20 key-value pairs and 483 tensors from /home/liyy/.ollama/models/blobs/sha256:de0334402b975e19dd48eb43a13f7534772fb5b4a054447f8f6a861b87ec5799 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen2
llama_model_loader: - kv 1: general.name str = Qwen2-beta-14B-Chat
llama_model_loader: - kv 2: qwen2.block_count u32 = 40
llama_model_loader: - kv 3: qwen2.context_length u32 = 32768
llama_model_loader: - kv 4: qwen2.embedding_length u32 = 5120
llama_model_loader: - kv 5: qwen2.feed_forward_length u32 = 13696
llama_model_loader: - kv 6: qwen2.attention.head_count u32 = 40
llama_model_loader: - kv 7: qwen2.attention.head_count_kv u32 = 40
llama_model_loader: - kv 8: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 9: qwen2.use_parallel_residual bool = true
llama_model_loader: - kv 10: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 11: tokenizer.ggml.tokens arr[str,152064] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 12: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 13: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 14: tokenizer.ggml.eos_token_id u32 = 151643
llama_model_loader: - kv 15: tokenizer.ggml.padding_token_id u32 = 151643
llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 151643
llama_model_loader: - kv 17: tokenizer.chat_template str = {% for message in messages %}{{'<|im_...
llama_model_loader: - kv 18: general.quantization_version u32 = 2
llama_model_loader: - kv 19: general.file_type u32 = 2
llama_model_loader: - type f32: 201 tensors
llama_model_loader: - type q4_0: 281 tensors
llama_model_loader: - type q6_K: 1 tensors
llm_load_vocab: special tokens definition check successful ( 421/152064 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = qwen2
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 152064
llm_load_print_meta: n_merges = 151387
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 5120
llm_load_print_meta: n_head = 40
llm_load_print_meta: n_head_kv = 40
llm_load_print_meta: n_layer = 40
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 1
llm_load_print_meta: n_embd_k_gqa = 5120
llm_load_print_meta: n_embd_v_gqa = 5120
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-06
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff = 13696
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: model type = 13B
llm_load_print_meta: model ftype = Q4_0
llm_load_print_meta: model params = 14.17 B
llm_load_print_meta: model size = 7.61 GiB (4.62 BPW)
llm_load_print_meta: general.name = Qwen2-beta-14B-Chat
llm_load_print_meta: BOS token = 151643 '<|endoftext|>'
llm_load_print_meta: EOS token = 151643 '<|endoftext|>'
llm_load_print_meta: PAD token = 151643 '<|endoftext|>'
llm_load_print_meta: LF token = 148848 'ÄĬ'
llm_load_tensors: ggml ctx size = 0.37 MiB
llm_load_tensors: offloading 1 repeating layers to GPU
llm_load_tensors: offloaded 1/41 layers to GPU
llm_load_tensors: CPU buffer size = 7794.73 MiB
llm_load_tensors: CUDA0 buffer size = 169.20 MiB
.........................................................................................
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CUDA_Host KV buffer size = 1560.00 MiB
llama_kv_cache_init: CUDA0 KV buffer size = 40.00 MiB
llama_new_context_with_model: KV self size = 1600.00 MiB, K (f16): 800.00 MiB, V (f16): 800.00 MiB
llama_new_context_with_model: CUDA_Host input buffer size = 15.02 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 184.01 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 327.00 MiB
llama_new_context_with_model: graph splits (measure): 5
time=2024-03-01T10:34:42.057+08:00 level=INFO source=dyn_ext_server.go:161 msg="Starting llama main loop"
[GIN] 2024/03/01 - 10:35:05 | 200 | 27.784462455s | 172.18.0.3 | POST "/api/chat"
time=2024-03-01T10:35:51.167+08:00 level=INFO source=routes.go:78 msg="changing loaded model"
time=2024-03-01T10:35:53.541+08:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-01T10:35:53.541+08:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.9"
time=2024-03-01T10:35:53.541+08:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-01T10:35:53.541+08:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.9"
time=2024-03-01T10:35:53.541+08:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
loading library /tmp/ollama1445230373/cuda_v11/libext_server.so
time=2024-03-01T10:35:53.541+08:00 level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: /tmp/ollama1445230373/cuda_v11/libext_server.so"
time=2024-03-01T10:35:53.541+08:00 level=INFO source=dyn_ext_server.go:150 msg="Initializing llama server"
llama_model_loader: loaded meta data with 20 key-value pairs and 483 tensors from /home/liyy/.ollama/models/blobs/sha256:de0334402b975e19dd48eb43a13f7534772fb5b4a054447f8f6a861b87ec5799 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen2
llama_model_loader: - kv 1: general.name str = Qwen2-beta-14B-Chat
llama_model_loader: - kv 2: qwen2.block_count u32 = 40
llama_model_loader: - kv 3: qwen2.context_length u32 = 32768
llama_model_loader: - kv 4: qwen2.embedding_length u32 = 5120
llama_model_loader: - kv 5: qwen2.feed_forward_length u32 = 13696
llama_model_loader: - kv 6: qwen2.attention.head_count u32 = 40
llama_model_loader: - kv 7: qwen2.attention.head_count_kv u32 = 40
llama_model_loader: - kv 8: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 9: qwen2.use_parallel_residual bool = true
llama_model_loader: - kv 10: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 11: tokenizer.ggml.tokens arr[str,152064] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 12: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 13: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 14: tokenizer.ggml.eos_token_id u32 = 151643
llama_model_loader: - kv 15: tokenizer.ggml.padding_token_id u32 = 151643
llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 151643
llama_model_loader: - kv 17: tokenizer.chat_template str = {% for message in messages %}{{'<|im_...
llama_model_loader: - kv 18: general.quantization_version u32 = 2
llama_model_loader: - kv 19: general.file_type u32 = 2
llama_model_loader: - type f32: 201 tensors
llama_model_loader: - type q4_0: 281 tensors
llama_model_loader: - type q6_K: 1 tensors
llm_load_vocab: special tokens definition check successful ( 421/152064 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = qwen2
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 152064
llm_load_print_meta: n_merges = 151387
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 5120
llm_load_print_meta: n_head = 40
llm_load_print_meta: n_head_kv = 40
llm_load_print_meta: n_layer = 40
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 1
llm_load_print_meta: n_embd_k_gqa = 5120
llm_load_print_meta: n_embd_v_gqa = 5120
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-06
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff = 13696
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: model type = 13B
llm_load_print_meta: model ftype = Q4_0
llm_load_print_meta: model params = 14.17 B
llm_load_print_meta: model size = 7.61 GiB (4.62 BPW)
llm_load_print_meta: general.name = Qwen2-beta-14B-Chat
llm_load_print_meta: BOS token = 151643 '<|endoftext|>'
llm_load_print_meta: EOS token = 151643 '<|endoftext|>'
llm_load_print_meta: PAD token = 151643 '<|endoftext|>'
llm_load_print_meta: LF token = 148848 'ÄĬ'
llm_load_tensors: ggml ctx size = 0.37 MiB
llm_load_tensors: offloading 40 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 41/41 layers to GPU
llm_load_tensors: CPU buffer size = 417.66 MiB
llm_load_tensors: CUDA0 buffer size = 7377.08 MiB
.........................................................................................
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: yes
ggml_init_cublas: CUDA_USE_TENSOR_CORES: no
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
llama_kv_cache_init: CUDA0 KV buffer size = 1600.00 MiB
llama_new_context_with_model: KV self size = 1600.00 MiB, K (f16): 800.00 MiB, V (f16): 800.00 MiB
llama_new_context_with_model: CUDA_Host input buffer size = 15.02 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 307.00 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 10.00 MiB
llama_new_context_with_model: graph splits (measure): 3
time=2024-03-01T10:35:55.605+08:00 level=INFO source=dyn_ext_server.go:161 msg="Starting llama main loop"
[GIN] 2024/03/01 - 10:35:58 | 200 | 6.918120565s | ::1 | POST "/api/generate"

Originally created by @xiaotianfotos on GitHub (Mar 1, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2851 my problem is i 'm using Dify to connect to ollama service when using ollama API on Dify,always **loaded to CPU memory**, but i tried using curl http://localhost:11434/api/generate -d '{ "model": "qwen:14b", "prompt": "Why is the sky blue?" }' **loaded into GPU** log description 1st run :ollama API via Dify 2nd run : curl i also tried using OpenAI-API-compatible on Dify,**loaded to GPU** log as following time=2024-03-01T10:34:19.796+08:00 level=INFO source=images.go:710 msg="total blobs: 5" time=2024-03-01T10:34:19.796+08:00 level=INFO source=images.go:717 msg="total unused blobs removed: 0" time=2024-03-01T10:34:19.797+08:00 level=INFO source=routes.go:1019 msg="Listening on [::]:11434 (version 0.1.27)" time=2024-03-01T10:34:19.797+08:00 level=INFO source=payload_common.go:107 msg="Extracting dynamic libraries..." time=2024-03-01T10:34:22.664+08:00 level=INFO source=payload_common.go:146 msg="Dynamic LLM libraries [rocm_v5 cpu_avx rocm_v6 cpu_avx2 cuda_v11 cpu]" time=2024-03-01T10:34:22.664+08:00 level=INFO source=gpu.go:94 msg="Detecting GPU type" time=2024-03-01T10:34:22.664+08:00 level=INFO source=gpu.go:265 msg="Searching for GPU management library libnvidia-ml.so" time=2024-03-01T10:34:22.666+08:00 level=INFO source=gpu.go:311 msg="Discovered GPU libraries: [/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.535.154.05]" time=2024-03-01T10:34:22.672+08:00 level=INFO source=gpu.go:99 msg="Nvidia GPU detected" time=2024-03-01T10:34:22.672+08:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-03-01T10:34:22.678+08:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.9" time=2024-03-01T10:34:38.704+08:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-03-01T10:34:38.704+08:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.9" time=2024-03-01T10:34:38.704+08:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-03-01T10:34:38.704+08:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.9" time=2024-03-01T10:34:38.704+08:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" loading library /tmp/ollama1445230373/cuda_v11/libext_server.so time=2024-03-01T10:34:38.715+08:00 level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: /tmp/ollama1445230373/cuda_v11/libext_server.so" time=2024-03-01T10:34:38.715+08:00 level=INFO source=dyn_ext_server.go:150 msg="Initializing llama server" ggml_init_cublas: GGML_CUDA_FORCE_MMQ: yes ggml_init_cublas: CUDA_USE_TENSOR_CORES: no ggml_init_cublas: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes llama_model_loader: loaded meta data with 20 key-value pairs and 483 tensors from /home/liyy/.ollama/models/blobs/sha256:de0334402b975e19dd48eb43a13f7534772fb5b4a054447f8f6a861b87ec5799 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = qwen2 llama_model_loader: - kv 1: general.name str = Qwen2-beta-14B-Chat llama_model_loader: - kv 2: qwen2.block_count u32 = 40 llama_model_loader: - kv 3: qwen2.context_length u32 = 32768 llama_model_loader: - kv 4: qwen2.embedding_length u32 = 5120 llama_model_loader: - kv 5: qwen2.feed_forward_length u32 = 13696 llama_model_loader: - kv 6: qwen2.attention.head_count u32 = 40 llama_model_loader: - kv 7: qwen2.attention.head_count_kv u32 = 40 llama_model_loader: - kv 8: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 9: qwen2.use_parallel_residual bool = true llama_model_loader: - kv 10: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 11: tokenizer.ggml.tokens arr[str,152064] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 12: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 13: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... llama_model_loader: - kv 14: tokenizer.ggml.eos_token_id u32 = 151643 llama_model_loader: - kv 15: tokenizer.ggml.padding_token_id u32 = 151643 llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 151643 llama_model_loader: - kv 17: tokenizer.chat_template str = {% for message in messages %}{{'<|im_... llama_model_loader: - kv 18: general.quantization_version u32 = 2 llama_model_loader: - kv 19: general.file_type u32 = 2 llama_model_loader: - type f32: 201 tensors llama_model_loader: - type q4_0: 281 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special tokens definition check successful ( 421/152064 ). llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = qwen2 llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 152064 llm_load_print_meta: n_merges = 151387 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 5120 llm_load_print_meta: n_head = 40 llm_load_print_meta: n_head_kv = 40 llm_load_print_meta: n_layer = 40 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 1 llm_load_print_meta: n_embd_k_gqa = 5120 llm_load_print_meta: n_embd_v_gqa = 5120 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-06 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: n_ff = 13696 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: model type = 13B llm_load_print_meta: model ftype = Q4_0 llm_load_print_meta: model params = 14.17 B llm_load_print_meta: model size = 7.61 GiB (4.62 BPW) llm_load_print_meta: general.name = Qwen2-beta-14B-Chat llm_load_print_meta: BOS token = 151643 '<|endoftext|>' llm_load_print_meta: EOS token = 151643 '<|endoftext|>' llm_load_print_meta: PAD token = 151643 '<|endoftext|>' llm_load_print_meta: LF token = 148848 'ÄĬ' llm_load_tensors: ggml ctx size = 0.37 MiB llm_load_tensors: offloading 1 repeating layers to GPU llm_load_tensors: offloaded 1/41 layers to GPU llm_load_tensors: CPU buffer size = 7794.73 MiB llm_load_tensors: CUDA0 buffer size = 169.20 MiB ......................................................................................... llama_new_context_with_model: n_ctx = 2048 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: CUDA_Host KV buffer size = 1560.00 MiB llama_kv_cache_init: CUDA0 KV buffer size = 40.00 MiB llama_new_context_with_model: KV self size = 1600.00 MiB, K (f16): 800.00 MiB, V (f16): 800.00 MiB llama_new_context_with_model: CUDA_Host input buffer size = 15.02 MiB llama_new_context_with_model: CUDA0 compute buffer size = 184.01 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 327.00 MiB llama_new_context_with_model: graph splits (measure): 5 time=2024-03-01T10:34:42.057+08:00 level=INFO source=dyn_ext_server.go:161 msg="Starting llama main loop" [GIN] 2024/03/01 - 10:35:05 | 200 | 27.784462455s | 172.18.0.3 | POST "/api/chat" time=2024-03-01T10:35:51.167+08:00 level=INFO source=routes.go:78 msg="changing loaded model" time=2024-03-01T10:35:53.541+08:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-03-01T10:35:53.541+08:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.9" time=2024-03-01T10:35:53.541+08:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-03-01T10:35:53.541+08:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.9" time=2024-03-01T10:35:53.541+08:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" loading library /tmp/ollama1445230373/cuda_v11/libext_server.so time=2024-03-01T10:35:53.541+08:00 level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: /tmp/ollama1445230373/cuda_v11/libext_server.so" time=2024-03-01T10:35:53.541+08:00 level=INFO source=dyn_ext_server.go:150 msg="Initializing llama server" llama_model_loader: loaded meta data with 20 key-value pairs and 483 tensors from /home/liyy/.ollama/models/blobs/sha256:de0334402b975e19dd48eb43a13f7534772fb5b4a054447f8f6a861b87ec5799 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = qwen2 llama_model_loader: - kv 1: general.name str = Qwen2-beta-14B-Chat llama_model_loader: - kv 2: qwen2.block_count u32 = 40 llama_model_loader: - kv 3: qwen2.context_length u32 = 32768 llama_model_loader: - kv 4: qwen2.embedding_length u32 = 5120 llama_model_loader: - kv 5: qwen2.feed_forward_length u32 = 13696 llama_model_loader: - kv 6: qwen2.attention.head_count u32 = 40 llama_model_loader: - kv 7: qwen2.attention.head_count_kv u32 = 40 llama_model_loader: - kv 8: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 9: qwen2.use_parallel_residual bool = true llama_model_loader: - kv 10: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 11: tokenizer.ggml.tokens arr[str,152064] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 12: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 13: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... llama_model_loader: - kv 14: tokenizer.ggml.eos_token_id u32 = 151643 llama_model_loader: - kv 15: tokenizer.ggml.padding_token_id u32 = 151643 llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 151643 llama_model_loader: - kv 17: tokenizer.chat_template str = {% for message in messages %}{{'<|im_... llama_model_loader: - kv 18: general.quantization_version u32 = 2 llama_model_loader: - kv 19: general.file_type u32 = 2 llama_model_loader: - type f32: 201 tensors llama_model_loader: - type q4_0: 281 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special tokens definition check successful ( 421/152064 ). llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = qwen2 llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 152064 llm_load_print_meta: n_merges = 151387 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 5120 llm_load_print_meta: n_head = 40 llm_load_print_meta: n_head_kv = 40 llm_load_print_meta: n_layer = 40 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 1 llm_load_print_meta: n_embd_k_gqa = 5120 llm_load_print_meta: n_embd_v_gqa = 5120 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-06 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: n_ff = 13696 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: model type = 13B llm_load_print_meta: model ftype = Q4_0 llm_load_print_meta: model params = 14.17 B llm_load_print_meta: model size = 7.61 GiB (4.62 BPW) llm_load_print_meta: general.name = Qwen2-beta-14B-Chat llm_load_print_meta: BOS token = 151643 '<|endoftext|>' llm_load_print_meta: EOS token = 151643 '<|endoftext|>' llm_load_print_meta: PAD token = 151643 '<|endoftext|>' llm_load_print_meta: LF token = 148848 'ÄĬ' llm_load_tensors: ggml ctx size = 0.37 MiB llm_load_tensors: offloading 40 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 41/41 layers to GPU llm_load_tensors: CPU buffer size = 417.66 MiB llm_load_tensors: CUDA0 buffer size = 7377.08 MiB ......................................................................................... llama_new_context_with_model: n_ctx = 2048 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 ggml_init_cublas: GGML_CUDA_FORCE_MMQ: yes ggml_init_cublas: CUDA_USE_TENSOR_CORES: no ggml_init_cublas: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes llama_kv_cache_init: CUDA0 KV buffer size = 1600.00 MiB llama_new_context_with_model: KV self size = 1600.00 MiB, K (f16): 800.00 MiB, V (f16): 800.00 MiB llama_new_context_with_model: CUDA_Host input buffer size = 15.02 MiB llama_new_context_with_model: CUDA0 compute buffer size = 307.00 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 10.00 MiB llama_new_context_with_model: graph splits (measure): 3 time=2024-03-01T10:35:55.605+08:00 level=INFO source=dyn_ext_server.go:161 msg="Starting llama main loop" [GIN] 2024/03/01 - 10:35:58 | 200 | 6.918120565s | ::1 | POST "/api/generate"
GiteaMirror added the bug label 2026-04-12 11:43:04 -05:00
GiteaMirror commented 2026-04-12 11:43:06 -05:00
Author
Owner
Copy Link

@remy415 commented on GitHub (Mar 5, 2024):

@xiaotianfotos not sure what's causing the issue, but it would seem like there's not enough CUDA memory free for the backend service to load the 14b model. It may simply be something that requires a computer reboot. You additionally could try using 7b models and see if those work better. I don't think it's a difference between Dify and curl -- if I understand it correctly their format should be nearly identical from the perspective of the backend and should have no impact in how many layers are loaded into the GPU. I could be wrong though, I'm not that familiar with Dify.

<!-- gh-comment-id:1979699425 --> @remy415 commented on GitHub (Mar 5, 2024): @xiaotianfotos not sure what's causing the issue, but it would seem like there's not enough CUDA memory free for the backend service to load the 14b model. It may simply be something that requires a computer reboot. You additionally could try using 7b models and see if those work better. I don't think it's a difference between Dify and curl -- if I understand it correctly their format should be nearly identical from the perspective of the backend and should have no impact in how many layers are loaded into the GPU. I could be wrong though, I'm not that familiar with Dify.
GiteaMirror commented 2026-04-12 11:43:07 -05:00
Author
Owner
Copy Link

@zeyowang commented on GitHub (Mar 6, 2024):

@xiaotianfotos Has your issue been solved? I encountered the same issue.

<!-- gh-comment-id:1980172856 --> @zeyowang commented on GitHub (Mar 6, 2024): @xiaotianfotos Has your issue been solved? I encountered the same issue.
GiteaMirror commented 2026-04-12 11:43:08 -05:00
Author
Owner
Copy Link

@xiaotianfotos commented on GitHub (Mar 6, 2024):

@xiaotianfotos Has your issue been solved? I encountered the same issue.您的问题解决了吗？我遇到了同样的问题。

not yet

<!-- gh-comment-id:1980174326 --> @xiaotianfotos commented on GitHub (Mar 6, 2024): > @xiaotianfotos Has your issue been solved? I encountered the same issue.您的问题解决了吗？我遇到了同样的问题。 not yet
GiteaMirror commented 2026-04-12 11:43:09 -05:00
Author
Owner
Copy Link

@zeyowang commented on GitHub (Mar 6, 2024):

@xiaotianfotos Has your issue been solved? I encountered the same issue.您的问题解决了吗？我遇到了同样的问题。

not yet

Disable the option NUM GPU in dify, and then manually set the ollama model parameter num_gpu

<!-- gh-comment-id:1980314276 --> @zeyowang commented on GitHub (Mar 6, 2024): > > @xiaotianfotos Has your issue been solved? I encountered the same issue.您的问题解决了吗？我遇到了同样的问题。 > > not yet Disable the option NUM GPU in dify, and then manually set the ollama model parameter num_gpu
GiteaMirror commented 2026-04-12 11:43:09 -05:00
Author
Owner
Copy Link

@bsd20107 commented on GitHub (Mar 9, 2024):

@xiaotianfotos Has your issue been solved? I encountered the same issue.您的问题解决了吗？我遇到了同样的问题。

not yet

Disable the option NUM GPU in dify, and then manually set the ollama model parameter num_gpu

Hi there. i am encountering same issue but lost in the method you mentioned. Could you pls guide me how to disable NUM GPU in dify and manally set the ollama model parameter? Thank you very much!

<!-- gh-comment-id:1986734212 --> @bsd20107 commented on GitHub (Mar 9, 2024): > > > @xiaotianfotos Has your issue been solved? I encountered the same issue.您的问题解决了吗？我遇到了同样的问题。 > > > > > > not yet > > Disable the option NUM GPU in dify, and then manually set the ollama model parameter num_gpu Hi there. i am encountering same issue but lost in the method you mentioned. Could you pls guide me how to disable NUM GPU in dify and manally set the ollama model parameter? Thank you very much!
GiteaMirror commented 2026-04-12 11:43:10 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Mar 12, 2024):

Hi folks, sorry you hit an issue. What error/issue are you seeing with Dify?

<!-- gh-comment-id:1989680333 --> @jmorganca commented on GitHub (Mar 12, 2024): Hi folks, sorry you hit an issue. What error/issue are you seeing with Dify?
GiteaMirror commented 2026-04-12 11:43:11 -05:00
Author
Owner
Copy Link

@xiaotianfotos commented on GitHub (Mar 12, 2024):

Hi folks, sorry you hit an issue. What error/issue are you seeing with Dify?

All the problem are based on this scenario: Using Ollama API in Dify

1.Generating content becomes incredibly slow. Upon examining with nvtop, it revealed that the GPU was not totally utilized.

2.Additionally, if a System Prompt is configured in Dify, it will output wrong code.like this:
G222<)=+A6D6)D1<E!3F6(8@>:8BC65093=39)$&$C!.5E0&17<!.<#:H6<1.=B!.G<.!F"+G#E6@F$9,F"$4CAG2<2-),5E3:>A0)$<F@*#;BH.-=!:.89)&66>

All these problems disappear when using Ollama provided OpenAI API within Dify.

<!-- gh-comment-id:1989697452 --> @xiaotianfotos commented on GitHub (Mar 12, 2024): > Hi folks, sorry you hit an issue. What error/issue are you seeing with Dify? All the problem are based on this scenario: Using Ollama API in Dify 1.Generating content becomes incredibly slow. Upon examining with nvtop, it revealed that the GPU was not totally utilized. 2.Additionally, if a System Prompt is configured in Dify, it will output wrong code.like this: G222<)=+A6D6)D1<E!3F6(8@>:8BC65093=39)$&$C!.5E0&17<!.<#:H6<1.=B!.G<.!F"+G#E6@F$9,F"$4CAG2<2-),5E3:>A0)$<F@*#;BH.-=!:.89)&66> All these problems disappear when using Ollama provided OpenAI API within Dify.
GiteaMirror commented 2026-04-12 11:43:12 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Mar 12, 2024):

Thanks! It's possible the Dify integration with Ollama isn't quite right, would it be possible to link me to it? I can help point out what might be going on there. Good to hear the OpenAI compatible API fixes the issue – more to come here. I'll close this for now but would still love to help hunt down why it might not be working right

<!-- gh-comment-id:1990945952 --> @jmorganca commented on GitHub (Mar 12, 2024): Thanks! It's possible the Dify integration with Ollama isn't quite right, would it be possible to link me to it? I can help point out what might be going on there. Good to hear the OpenAI compatible API fixes the issue – more to come here. I'll close this for now but would still love to help hunt down why it might not be working right
GiteaMirror commented 2026-04-12 11:43:13 -05:00
Author
Owner
Copy Link

@liyanshuai2018 commented on GitHub (Apr 22, 2024):

Did not understand the chat. So, is the issue resolved?

<!-- gh-comment-id:2069542377 --> @liyanshuai2018 commented on GitHub (Apr 22, 2024): Did not understand the chat. So, is the issue resolved?
GiteaMirror commented 2026-04-12 11:43:14 -05:00
Author
Owner
Copy Link

@remy415 commented on GitHub (Apr 22, 2024):

@liyanshuai2018 partially fixed. Use openapi compatible API in Dify will work.

<!-- gh-comment-id:2069640157 --> @remy415 commented on GitHub (Apr 22, 2024): @liyanshuai2018 partially fixed. Use `openapi compatible API` in Dify will work.
GiteaMirror commented 2026-04-12 11:43:15 -05:00
Author
Owner
Copy Link

@liyanshuai2018 commented on GitHub (Apr 22, 2024):

@remy415 @xiaotianfotos How to do it, in Dify ? Can you give an example ?

<!-- gh-comment-id:2070103207 --> @liyanshuai2018 commented on GitHub (Apr 22, 2024): @remy415 @xiaotianfotos How to do it, in Dify ? Can you give an example ?
GiteaMirror commented 2026-04-12 11:43:15 -05:00
Author
Owner
Copy Link

@remy415 commented on GitHub (Apr 22, 2024):

@liyanshuai2018 I don't use Dify, your best bet would be to check the Dify documentation, unless someone else is able to share their experience here.

<!-- gh-comment-id:2070125754 --> @remy415 commented on GitHub (Apr 22, 2024): @liyanshuai2018 I don't use Dify, your best bet would be to check the Dify documentation, unless someone else is able to share their experience here.
GiteaMirror commented 2026-04-12 11:43:16 -05:00
Author
Owner
Copy Link

@xiaotianfotos commented on GitHub (Apr 23, 2024):

@remy415 @xiaotianfotos How to do it, in Dify ? Can you give an example ? 在 Dify 中如何做到这一点？你能给个例子吗 ？

in Dify's model provider there is a OpenAI-API-compatible provider
set baseURL:http://your_ollama_server:11434/v1
and the model name showed in ollama list

<!-- gh-comment-id:2071258764 --> @xiaotianfotos commented on GitHub (Apr 23, 2024): > @remy415 @xiaotianfotos How to do it, in Dify ? Can you give an example ? 在 Dify 中如何做到这一点？你能给个例子吗 ？ in Dify's model provider there is a OpenAI-API-compatible provider set baseURL:http://your_ollama_server:11434/v1 and the model name showed in ollama list
GiteaMirror commented 2026-04-12 11:43:17 -05:00
Author
Owner
Copy Link

@liyanshuai2018 commented on GitHub (Apr 23, 2024):

@remy415 @xiaotianfotos How to do it, in Dify ? Can you give an example ? 在 Dify 中如何做到这一点？你能给个例子吗 ？

@xiaotianfotos in Dify's model provider there is a OpenAI-API-compatible provider set baseURL:http://your_ollama_server:11434/v1 and the model name showed in ollama list

What version of Diffy is yours?? v0.57 works fine, but v0.64 can't add models and gives a 502 error. Is this a problem just for me?
image

<!-- gh-comment-id:2072902873 --> @liyanshuai2018 commented on GitHub (Apr 23, 2024): > > @remy415 @xiaotianfotos How to do it, in Dify ? Can you give an example ? 在 Dify 中如何做到这一点？你能给个例子吗 ？ > > @xiaotianfotos in Dify's model provider there is a OpenAI-API-compatible provider set baseURL:http://your_ollama_server:11434/v1 and the model name showed in ollama list What version of Diffy is yours?? v0.57 works fine, but v0.64 can't add models and gives a 502 error. Is this a problem just for me? ![image](https://github.com/ollama/ollama/assets/42138577/c5c54211-a76d-4c89-a8fc-64d891bca03b)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#1736
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?