github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-21 05:01:48 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #9564] parallel request error #6238

New Issue
Closed
opened 2026-04-12 17:39:41 -05:00 by GiteaMirror · 25 comments
GiteaMirror commented 2026-04-12 17:39:41 -05:00
Owner
Copy Link

Originally created by @EdwinMeriaux on GitHub (Mar 7, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9564

What is the issue?

Hello! Sorry if the issue is my misunderstanding: from my understand OLLAMA_NUM_PARALLEL is originally set to 1 or 4 depending on available memory but can be increased if there is extra memory right?

I am operating a RTX 4090 with 24gb of VRAM and if I set OLLAMA_NUM_PARALLEL to 10 it does not run faster or use more of the available memory. Please see the enclosed logs from nvidia-smi. If i run the following code with parallel_num = one or four I see improvement. Above that nothing changes:

import asyncio
import time
import ollama

parallel_num = 4 # Max concurrent requests
total_requests = 20 # Total number of requests

async def describe_image(number):
"""Asynchronous function to send a request to Ollama."""
try:
client = ollama.AsyncClient() # No async with
res = await client.chat(
model="llava",
messages=[{'role': 'user', 'content': 'Describe Fermat's Last Theorem'}]
)
print(f"Completed request {number}")
except Exception as e:
print(f"Error on request {number}: {e}")

async def main():
start_time = time.time() # Start timing

semaphore = asyncio.Semaphore(parallel_num)  # Limit concurrent tasks

async def limited_request(number):
    async with semaphore:
        await describe_image(number)

tasks = [limited_request(i) for i in range(total_requests)]
await asyncio.gather(*tasks)  # Run tasks in parallel

end_time = time.time()  # End timing
print(f"All requests completed in {end_time - start_time:.2f} seconds.")

if name == "main":
asyncio.run(main()) # Run the async function

I launch the script with:

OLLAMA_GPU_MEMORY=20000 OLLAMA_NUM_PARALLEL=2 python3 function_test1.py (where I change the number 2 as needed)

Relevant log output

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.120                Driver Version: 550.120        CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        Off |   00000000:01:00.0  On |                  Off |
| 30%   55C    P2            292W /  450W |    7140MiB /  24564MiB |     93%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      1365      G   /usr/lib/xorg/Xorg                            287MiB |
|    0   N/A  N/A      5273      G   xfwm4                                           8MiB |
|    0   N/A  N/A      6512      G   ...erProcess --variations-seed-version         94MiB |
|    0   N/A  N/A      6979      G   /usr/lib/firefox/firefox                      205MiB |
|    0   N/A  N/A     37498      C   /usr/local/bin/ollama                        6516MiB |
+-----------------------------------------------------------------------------------------+

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

0.5.13

Originally created by @EdwinMeriaux on GitHub (Mar 7, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9564 ### What is the issue? Hello! Sorry if the issue is my misunderstanding: from my understand OLLAMA_NUM_PARALLEL is originally set to 1 or 4 depending on available memory but can be increased if there is extra memory right? I am operating a RTX 4090 with 24gb of VRAM and if I set OLLAMA_NUM_PARALLEL to 10 it does not run faster or use more of the available memory. Please see the enclosed logs from nvidia-smi. If i run the following code with parallel_num = one or four I see improvement. Above that nothing changes: import asyncio import time import ollama parallel_num = 4 # Max concurrent requests total_requests = 20 # Total number of requests async def describe_image(number): """Asynchronous function to send a request to Ollama.""" try: client = ollama.AsyncClient() # No async with res = await client.chat( model="llava", messages=[{'role': 'user', 'content': 'Describe Fermat\'s Last Theorem'}] ) print(f"Completed request {number}") except Exception as e: print(f"Error on request {number}: {e}") async def main(): start_time = time.time() # Start timing semaphore = asyncio.Semaphore(parallel_num) # Limit concurrent tasks async def limited_request(number): async with semaphore: await describe_image(number) tasks = [limited_request(i) for i in range(total_requests)] await asyncio.gather(*tasks) # Run tasks in parallel end_time = time.time() # End timing print(f"All requests completed in {end_time - start_time:.2f} seconds.") if __name__ == "__main__": asyncio.run(main()) # Run the async function I launch the script with: OLLAMA_GPU_MEMORY=20000 OLLAMA_NUM_PARALLEL=2 python3 function_test1.py (where I change the number 2 as needed) ### Relevant log output ```shell +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.120 Driver Version: 550.120 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 4090 Off | 00000000:01:00.0 On | Off | | 30% 55C P2 292W / 450W | 7140MiB / 24564MiB | 93% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 1365 G /usr/lib/xorg/Xorg 287MiB | | 0 N/A N/A 5273 G xfwm4 8MiB | | 0 N/A N/A 6512 G ...erProcess --variations-seed-version 94MiB | | 0 N/A N/A 6979 G /usr/lib/firefox/firefox 205MiB | | 0 N/A N/A 37498 C /usr/local/bin/ollama 6516MiB | +-----------------------------------------------------------------------------------------+ ``` ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version 0.5.13
GiteaMirror added the bug label 2026-04-12 17:39:41 -05:00
GiteaMirror commented 2026-04-12 17:39:43 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 7, 2025):

Set OLLAMA_NUM_PARALLEL in the server environment, not the client environment.

<!-- gh-comment-id:2705659225 --> @rick-github commented on GitHub (Mar 7, 2025): Set `OLLAMA_NUM_PARALLEL` in the [server environment](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-configure-ollama-server), not the client environment.
GiteaMirror commented 2026-04-12 17:39:44 -05:00
Author
Owner
Copy Link

@EdwinMeriaux commented on GitHub (Mar 7, 2025):

Hi! @rick-github,

Thank you for your comment. I tried it now:

Editing /etc/systemd/system/ollama.service.d/override.conf

Anything between here and the comment below will become the new contents of the file

Environment="OLLAMA_NUM_PARALLEL=10"
Environment="OLLAMA_GPU_MEMORY=20GiB"

Lines below this comment will be discarded

/etc/systemd/system/ollama.service

[Unit]

Description=Ollama Service

After=network-online.target

[Service]

ExecStart=/usr/local/bin/ollama serve

User=ollama

Group=ollama

Restart=always

RestartSec=3

Environment="PATH=/home/thalassa/emeriaux/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/thalassa/emeriaux/.vscode/extensions/ms-python.debugpy-2025.0.0-linux-x64/bun>

[Install]

WantedBy=default.target

And there really isn't any change in the compute time. LLAVA is still only using 6gb of video memory and it is not running more concurrent instances. What should I do?

<!-- gh-comment-id:2706928142 --> @EdwinMeriaux commented on GitHub (Mar 7, 2025): Hi! @rick-github, Thank you for your comment. I tried it now: ### Editing /etc/systemd/system/ollama.service.d/override.conf ### Anything between here and the comment below will become the new contents of the file Environment="OLLAMA_NUM_PARALLEL=10" Environment="OLLAMA_GPU_MEMORY=20GiB" ### Lines below this comment will be discarded ### /etc/systemd/system/ollama.service # [Unit] # Description=Ollama Service # After=network-online.target # # [Service] # ExecStart=/usr/local/bin/ollama serve # User=ollama # Group=ollama # Restart=always # RestartSec=3 # Environment="PATH=/home/thalassa/emeriaux/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/thalassa/emeriaux/.vscode/extensions/ms-python.debugpy-2025.0.0-linux-x64/bun> # # [Install] # WantedBy=default.target And there really isn't any change in the compute time. LLAVA is still only using 6gb of video memory and it is not running more concurrent instances. What should I do?
GiteaMirror commented 2026-04-12 17:39:46 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 7, 2025):

Did you restart the service? What does the server log show?

<!-- gh-comment-id:2706933408 --> @rick-github commented on GitHub (Mar 7, 2025): Did you restart the service? What does the [server log](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) show?
GiteaMirror commented 2026-04-12 17:39:46 -05:00
Author
Owner
Copy Link

@EdwinMeriaux commented on GitHub (Mar 7, 2025):

Hi!

So I did:

sudo systemctl restart ollama

and there was no change. Is that what you meant by restart the service?

and there is nothing in the server log:

ran -> journalctl -u ollama -f
Hint: You are currently not seeing messages from other users and the system.
Users in groups 'adm', 'systemd-journal' can see all messages.
Pass -q to turn off this notice.

<!-- gh-comment-id:2706976327 --> @EdwinMeriaux commented on GitHub (Mar 7, 2025): Hi! So I did: sudo systemctl restart ollama and there was no change. Is that what you meant by restart the service? and there is nothing in the server log: ran -> journalctl -u ollama -f Hint: You are currently not seeing messages from other users and the system. Users in groups 'adm', 'systemd-journal' can see all messages. Pass -q to turn off this notice.
GiteaMirror commented 2026-04-12 17:39:47 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 7, 2025):

sudo journalctl -u ollama --no-pager
<!-- gh-comment-id:2706979867 --> @rick-github commented on GitHub (Mar 7, 2025): ``` sudo journalctl -u ollama --no-pager ```
GiteaMirror commented 2026-04-12 17:39:48 -05:00
Author
Owner
Copy Link

@EdwinMeriaux commented on GitHub (Mar 7, 2025):

Thanks! Ok! Here is the log for the last attempt!

Mar 07 12:06:51 Hermes ollama[312857]: time=2025-03-07T12:06:51.221-05:00 level=INFO source=images.go:432 msg="total blobs: 12"
Mar 07 12:06:51 Hermes ollama[312857]: time=2025-03-07T12:06:51.221-05:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
Mar 07 12:06:51 Hermes ollama[312857]: time=2025-03-07T12:06:51.221-05:00 level=INFO source=routes.go:1277 msg="Listening on 127.0.0.1:11434 (version 0.5.13)"
Mar 07 12:06:51 Hermes ollama[312857]: time=2025-03-07T12:06:51.221-05:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
Mar 07 12:06:51 Hermes ollama[312857]: time=2025-03-07T12:06:51.305-05:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-4e0dac96-1fe0-81b7-2d85-138e9604e2bd library=cuda variant=v12 compute=8.9 driver=12.4 name="NVIDIA GeForce RTX 4090" total="23.6 GiB" available="22.5 GiB"
Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.842-05:00 level=WARN source=ggml.go:136 msg="key not found" key=llama.attention.key_length default=128
Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.842-05:00 level=WARN source=ggml.go:136 msg="key not found" key=llama.attention.value_length default=128
Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.842-05:00 level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 gpu=GPU-4e0dac96-1fe0-81b7-2d85-138e9604e2bd parallel=4 available=24170463232 required="6.5 GiB"
Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.891-05:00 level=INFO source=server.go:97 msg="system memory" total="60.5 GiB" free="54.9 GiB" free_swap="2.0 GiB"
Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.892-05:00 level=WARN source=ggml.go:136 msg="key not found" key=llama.attention.key_length default=128
Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.892-05:00 level=WARN source=ggml.go:136 msg="key not found" key=llama.attention.value_length default=128
Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.892-05:00 level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[22.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.5 GiB" memory.required.partial="6.5 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.5 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.6 GiB" memory.weights.nonrepeating="102.6 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="585.0 MiB" projector.weights="595.5 MiB" projector.graph="0 B"
Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.892-05:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /usr/share/ollama/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --mmproj /usr/share/ollama/.ollama/models/blobs/sha256-72d6f08a42f656d36b356dbe0920675899a99ce21192fd66266fb7d82ed07539 --threads 16 --parallel 4 --port 35805"
Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.893-05:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.893-05:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding"
Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.893-05:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error"
Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.920-05:00 level=INFO source=runner.go:931 msg="starting go runner"
Mar 07 12:07:17 Hermes ollama[312857]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
Mar 07 12:07:17 Hermes ollama[312857]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Mar 07 12:07:17 Hermes ollama[312857]: ggml_cuda_init: found 1 CUDA devices:
Mar 07 12:07:17 Hermes ollama[312857]: Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
Mar 07 12:07:17 Hermes ollama[312857]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so
Mar 07 12:07:17 Hermes ollama[312857]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-icelake.so
Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.960-05:00 level=INFO source=runner.go:934 msg=system info="CPU : LLAMAFILE = 1 | CUDA : ARCHS = 500,600,610,700,750,800,860,870,890,900,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | cgo(gcc)" threads=16
Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.960-05:00 level=INFO source=runner.go:992 msg="Server listening on 127.0.0.1:35805"
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4090) - 23050 MiB free
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: loaded meta data with 24 key-value pairs and 291 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 (version GGUF V3 (latest))
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 0: general.architecture str = llama
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 1: general.name str = liuhaotian
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 2: llama.context_length u32 = 32768
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 3: llama.embedding_length u32 = 4096
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 4: llama.block_count u32 = 32
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 7: llama.attention.head_count u32 = 32
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 10: llama.rope.freq_base f32 = 1000000.000000
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 11: general.file_type u32 = 2
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 12: tokenizer.ggml.model str = llama
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32000] = ["", "", "", "<0x00>", "<...
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000...
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 1
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 17: tokenizer.ggml.eos_token_id u32 = 2
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 18: tokenizer.ggml.unknown_token_id u32 = 0
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 19: tokenizer.ggml.padding_token_id u32 = 0
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 20: tokenizer.ggml.add_bos_token bool = true
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 21: tokenizer.ggml.add_eos_token bool = false
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 22: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 23: general.quantization_version u32 = 2
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - type f32: 65 tensors
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - type q4_0: 225 tensors
Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - type q6_K: 1 tensors
Mar 07 12:07:17 Hermes ollama[312857]: print_info: file format = GGUF V3 (latest)
Mar 07 12:07:17 Hermes ollama[312857]: print_info: file type = Q4_0
Mar 07 12:07:17 Hermes ollama[312857]: print_info: file size = 3.83 GiB (4.54 BPW)
Mar 07 12:07:18 Hermes ollama[312857]: load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
Mar 07 12:07:18 Hermes ollama[312857]: load: special tokens cache size = 3
Mar 07 12:07:18 Hermes ollama[312857]: load: token to piece cache size = 0.1637 MB
Mar 07 12:07:18 Hermes ollama[312857]: print_info: arch = llama
Mar 07 12:07:18 Hermes ollama[312857]: print_info: vocab_only = 0
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_ctx_train = 32768
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_embd = 4096
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_layer = 32
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_head = 32
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_head_kv = 8
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_rot = 128
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_swa = 0
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_embd_head_k = 128
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_embd_head_v = 128
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_gqa = 4
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_embd_k_gqa = 1024
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_embd_v_gqa = 1024
Mar 07 12:07:18 Hermes ollama[312857]: print_info: f_norm_eps = 0.0e+00
Mar 07 12:07:18 Hermes ollama[312857]: print_info: f_norm_rms_eps = 1.0e-05
Mar 07 12:07:18 Hermes ollama[312857]: print_info: f_clamp_kqv = 0.0e+00
Mar 07 12:07:18 Hermes ollama[312857]: print_info: f_max_alibi_bias = 0.0e+00
Mar 07 12:07:18 Hermes ollama[312857]: print_info: f_logit_scale = 0.0e+00
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_ff = 14336
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_expert = 0
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_expert_used = 0
Mar 07 12:07:18 Hermes ollama[312857]: print_info: causal attn = 1
Mar 07 12:07:18 Hermes ollama[312857]: print_info: pooling type = 0
Mar 07 12:07:18 Hermes ollama[312857]: print_info: rope type = 0
Mar 07 12:07:18 Hermes ollama[312857]: print_info: rope scaling = linear
Mar 07 12:07:18 Hermes ollama[312857]: print_info: freq_base_train = 1000000.0
Mar 07 12:07:18 Hermes ollama[312857]: print_info: freq_scale_train = 1
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_ctx_orig_yarn = 32768
Mar 07 12:07:18 Hermes ollama[312857]: print_info: rope_finetuned = unknown
Mar 07 12:07:18 Hermes ollama[312857]: print_info: ssm_d_conv = 0
Mar 07 12:07:18 Hermes ollama[312857]: print_info: ssm_d_inner = 0
Mar 07 12:07:18 Hermes ollama[312857]: print_info: ssm_d_state = 0
Mar 07 12:07:18 Hermes ollama[312857]: print_info: ssm_dt_rank = 0
Mar 07 12:07:18 Hermes ollama[312857]: print_info: ssm_dt_b_c_rms = 0
Mar 07 12:07:18 Hermes ollama[312857]: print_info: model type = 7B
Mar 07 12:07:18 Hermes ollama[312857]: print_info: model params = 7.24 B
Mar 07 12:07:18 Hermes ollama[312857]: print_info: general.name = liuhaotian
Mar 07 12:07:18 Hermes ollama[312857]: print_info: vocab type = SPM
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_vocab = 32000
Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_merges = 0
Mar 07 12:07:18 Hermes ollama[312857]: print_info: BOS token = 1 ''
Mar 07 12:07:18 Hermes ollama[312857]: print_info: EOS token = 2 ''
Mar 07 12:07:18 Hermes ollama[312857]: print_info: UNK token = 0 ''
Mar 07 12:07:18 Hermes ollama[312857]: print_info: PAD token = 0 ''
Mar 07 12:07:18 Hermes ollama[312857]: print_info: LF token = 13 '<0x0A>'
Mar 07 12:07:18 Hermes ollama[312857]: print_info: EOG token = 2 ''
Mar 07 12:07:18 Hermes ollama[312857]: print_info: max token length = 48
Mar 07 12:07:18 Hermes ollama[312857]: load_tensors: loading model tensors, this can take a while... (mmap = true)
Mar 07 12:07:18 Hermes ollama[312857]: load_tensors: offloading 32 repeating layers to GPU
Mar 07 12:07:18 Hermes ollama[312857]: load_tensors: offloading output layer to GPU
Mar 07 12:07:18 Hermes ollama[312857]: load_tensors: offloaded 33/33 layers to GPU
Mar 07 12:07:18 Hermes ollama[312857]: load_tensors: CUDA0 model buffer size = 3847.55 MiB
Mar 07 12:07:18 Hermes ollama[312857]: load_tensors: CPU_Mapped model buffer size = 70.31 MiB
Mar 07 12:07:18 Hermes ollama[312857]: time=2025-03-07T12:07:18.144-05:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model"
Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: n_seq_max = 4
Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: n_ctx = 8192
Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: n_ctx_per_seq = 2048
Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: n_batch = 2048
Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: n_ubatch = 512
Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: flash_attn = 0
Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: freq_base = 1000000.0
Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: freq_scale = 1
Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: n_ctx_per_seq (2048) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
Mar 07 12:07:18 Hermes ollama[312857]: llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 32, can_shift = 1
Mar 07 12:07:18 Hermes ollama[312857]: llama_kv_cache_init: CUDA0 KV buffer size = 1024.00 MiB
Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: KV self size = 1024.00 MiB, K (f16): 512.00 MiB, V (f16): 512.00 MiB
Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: CUDA_Host output buffer size = 0.55 MiB
Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: CUDA0 compute buffer size = 560.00 MiB
Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: CUDA_Host compute buffer size = 24.01 MiB
Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: graph nodes = 1030
Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: graph splits = 2
Mar 07 12:07:18 Hermes ollama[312857]: key clip.use_silu not found in file
Mar 07 12:07:18 Hermes ollama[312857]: key clip.vision.image_grid_pinpoints not found in file
Mar 07 12:07:18 Hermes ollama[312857]: key clip.vision.feature_layer not found in file
Mar 07 12:07:18 Hermes ollama[312857]: key clip.vision.mm_patch_merge_type not found in file
Mar 07 12:07:18 Hermes ollama[312857]: key clip.vision.image_crop_resolution not found in file
Mar 07 12:07:18 Hermes ollama[312857]: time=2025-03-07T12:07:18.395-05:00 level=INFO source=server.go:596 msg="llama runner started in 0.50 seconds"
Mar 07 12:07:19 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:19 | 200 | 2.050519922s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:19 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:19 | 200 | 2.137203874s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:20 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:20 | 200 | 2.825573417s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:21 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:21 | 200 | 3.853311436s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:21 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:21 | 200 | 4.180678599s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:22 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:22 | 200 | 4.469883597s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:23 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:23 | 200 | 5.515971935s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:23 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:23 | 200 | 5.524816156s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:23 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:23 | 200 | 5.623232734s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:24 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:24 | 200 | 6.660335345s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:24 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:24 | 200 | 4.795856041s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:25 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:25 | 200 | 5.490165566s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:25 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:25 | 200 | 4.278931481s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:26 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:26 | 200 | 5.386905738s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:26 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:26 | 200 | 4.3196965s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:27 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:27 | 200 | 4.826283861s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:27 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:27 | 200 | 4.389918954s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:28 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:28 | 200 | 5.264628449s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:29 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:29 | 200 | 4.613452959s | 127.0.0.1 | POST "/api/chat"
Mar 07 12:07:29 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:29 | 200 | 5.804187676s | 127.0.0.1 | POST "/api/chat"

<!-- gh-comment-id:2707097535 --> @EdwinMeriaux commented on GitHub (Mar 7, 2025): Thanks! Ok! Here is the log for the last attempt! Mar 07 12:06:51 Hermes ollama[312857]: time=2025-03-07T12:06:51.221-05:00 level=INFO source=images.go:432 msg="total blobs: 12" Mar 07 12:06:51 Hermes ollama[312857]: time=2025-03-07T12:06:51.221-05:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0" Mar 07 12:06:51 Hermes ollama[312857]: time=2025-03-07T12:06:51.221-05:00 level=INFO source=routes.go:1277 msg="Listening on 127.0.0.1:11434 (version 0.5.13)" Mar 07 12:06:51 Hermes ollama[312857]: time=2025-03-07T12:06:51.221-05:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" Mar 07 12:06:51 Hermes ollama[312857]: time=2025-03-07T12:06:51.305-05:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-4e0dac96-1fe0-81b7-2d85-138e9604e2bd library=cuda variant=v12 compute=8.9 driver=12.4 name="NVIDIA GeForce RTX 4090" total="23.6 GiB" available="22.5 GiB" Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.842-05:00 level=WARN source=ggml.go:136 msg="key not found" key=llama.attention.key_length default=128 Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.842-05:00 level=WARN source=ggml.go:136 msg="key not found" key=llama.attention.value_length default=128 Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.842-05:00 level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 gpu=GPU-4e0dac96-1fe0-81b7-2d85-138e9604e2bd parallel=4 available=24170463232 required="6.5 GiB" Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.891-05:00 level=INFO source=server.go:97 msg="system memory" total="60.5 GiB" free="54.9 GiB" free_swap="2.0 GiB" Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.892-05:00 level=WARN source=ggml.go:136 msg="key not found" key=llama.attention.key_length default=128 Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.892-05:00 level=WARN source=ggml.go:136 msg="key not found" key=llama.attention.value_length default=128 Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.892-05:00 level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[22.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.5 GiB" memory.required.partial="6.5 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.5 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.6 GiB" memory.weights.nonrepeating="102.6 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="585.0 MiB" projector.weights="595.5 MiB" projector.graph="0 B" Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.892-05:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /usr/share/ollama/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --mmproj /usr/share/ollama/.ollama/models/blobs/sha256-72d6f08a42f656d36b356dbe0920675899a99ce21192fd66266fb7d82ed07539 --threads 16 --parallel 4 --port 35805" Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.893-05:00 level=INFO source=sched.go:450 msg="loaded runners" count=1 Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.893-05:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding" Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.893-05:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error" Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.920-05:00 level=INFO source=runner.go:931 msg="starting go runner" Mar 07 12:07:17 Hermes ollama[312857]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no Mar 07 12:07:17 Hermes ollama[312857]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no Mar 07 12:07:17 Hermes ollama[312857]: ggml_cuda_init: found 1 CUDA devices: Mar 07 12:07:17 Hermes ollama[312857]: Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes Mar 07 12:07:17 Hermes ollama[312857]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so Mar 07 12:07:17 Hermes ollama[312857]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-icelake.so Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.960-05:00 level=INFO source=runner.go:934 msg=system info="CPU : LLAMAFILE = 1 | CUDA : ARCHS = 500,600,610,700,750,800,860,870,890,900,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | cgo(gcc)" threads=16 Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.960-05:00 level=INFO source=runner.go:992 msg="Server listening on 127.0.0.1:35805" Mar 07 12:07:17 Hermes ollama[312857]: llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4090) - 23050 MiB free Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: loaded meta data with 24 key-value pairs and 291 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 (version GGUF V3 (latest)) Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 0: general.architecture str = llama Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 1: general.name str = liuhaotian Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 2: llama.context_length u32 = 32768 Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 3: llama.embedding_length u32 = 4096 Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 4: llama.block_count u32 = 32 Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336 Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128 Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 7: llama.attention.head_count u32 = 32 Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8 Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 10: llama.rope.freq_base f32 = 1000000.000000 Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 11: general.file_type u32 = 2 Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 12: tokenizer.ggml.model str = llama Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32000] = ["<unk>", "<s>", "</s>", "<0x00>", "<... Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000... Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 1 Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 17: tokenizer.ggml.eos_token_id u32 = 2 Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 18: tokenizer.ggml.unknown_token_id u32 = 0 Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 19: tokenizer.ggml.padding_token_id u32 = 0 Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 20: tokenizer.ggml.add_bos_token bool = true Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 21: tokenizer.ggml.add_eos_token bool = false Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 22: tokenizer.chat_template str = {{ bos_token }}{% for message in mess... Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - kv 23: general.quantization_version u32 = 2 Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - type f32: 65 tensors Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - type q4_0: 225 tensors Mar 07 12:07:17 Hermes ollama[312857]: llama_model_loader: - type q6_K: 1 tensors Mar 07 12:07:17 Hermes ollama[312857]: print_info: file format = GGUF V3 (latest) Mar 07 12:07:17 Hermes ollama[312857]: print_info: file type = Q4_0 Mar 07 12:07:17 Hermes ollama[312857]: print_info: file size = 3.83 GiB (4.54 BPW) Mar 07 12:07:18 Hermes ollama[312857]: load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect Mar 07 12:07:18 Hermes ollama[312857]: load: special tokens cache size = 3 Mar 07 12:07:18 Hermes ollama[312857]: load: token to piece cache size = 0.1637 MB Mar 07 12:07:18 Hermes ollama[312857]: print_info: arch = llama Mar 07 12:07:18 Hermes ollama[312857]: print_info: vocab_only = 0 Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_ctx_train = 32768 Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_embd = 4096 Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_layer = 32 Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_head = 32 Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_head_kv = 8 Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_rot = 128 Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_swa = 0 Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_embd_head_k = 128 Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_embd_head_v = 128 Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_gqa = 4 Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_embd_k_gqa = 1024 Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_embd_v_gqa = 1024 Mar 07 12:07:18 Hermes ollama[312857]: print_info: f_norm_eps = 0.0e+00 Mar 07 12:07:18 Hermes ollama[312857]: print_info: f_norm_rms_eps = 1.0e-05 Mar 07 12:07:18 Hermes ollama[312857]: print_info: f_clamp_kqv = 0.0e+00 Mar 07 12:07:18 Hermes ollama[312857]: print_info: f_max_alibi_bias = 0.0e+00 Mar 07 12:07:18 Hermes ollama[312857]: print_info: f_logit_scale = 0.0e+00 Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_ff = 14336 Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_expert = 0 Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_expert_used = 0 Mar 07 12:07:18 Hermes ollama[312857]: print_info: causal attn = 1 Mar 07 12:07:18 Hermes ollama[312857]: print_info: pooling type = 0 Mar 07 12:07:18 Hermes ollama[312857]: print_info: rope type = 0 Mar 07 12:07:18 Hermes ollama[312857]: print_info: rope scaling = linear Mar 07 12:07:18 Hermes ollama[312857]: print_info: freq_base_train = 1000000.0 Mar 07 12:07:18 Hermes ollama[312857]: print_info: freq_scale_train = 1 Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_ctx_orig_yarn = 32768 Mar 07 12:07:18 Hermes ollama[312857]: print_info: rope_finetuned = unknown Mar 07 12:07:18 Hermes ollama[312857]: print_info: ssm_d_conv = 0 Mar 07 12:07:18 Hermes ollama[312857]: print_info: ssm_d_inner = 0 Mar 07 12:07:18 Hermes ollama[312857]: print_info: ssm_d_state = 0 Mar 07 12:07:18 Hermes ollama[312857]: print_info: ssm_dt_rank = 0 Mar 07 12:07:18 Hermes ollama[312857]: print_info: ssm_dt_b_c_rms = 0 Mar 07 12:07:18 Hermes ollama[312857]: print_info: model type = 7B Mar 07 12:07:18 Hermes ollama[312857]: print_info: model params = 7.24 B Mar 07 12:07:18 Hermes ollama[312857]: print_info: general.name = liuhaotian Mar 07 12:07:18 Hermes ollama[312857]: print_info: vocab type = SPM Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_vocab = 32000 Mar 07 12:07:18 Hermes ollama[312857]: print_info: n_merges = 0 Mar 07 12:07:18 Hermes ollama[312857]: print_info: BOS token = 1 '<s>' Mar 07 12:07:18 Hermes ollama[312857]: print_info: EOS token = 2 '</s>' Mar 07 12:07:18 Hermes ollama[312857]: print_info: UNK token = 0 '<unk>' Mar 07 12:07:18 Hermes ollama[312857]: print_info: PAD token = 0 '<unk>' Mar 07 12:07:18 Hermes ollama[312857]: print_info: LF token = 13 '<0x0A>' Mar 07 12:07:18 Hermes ollama[312857]: print_info: EOG token = 2 '</s>' Mar 07 12:07:18 Hermes ollama[312857]: print_info: max token length = 48 Mar 07 12:07:18 Hermes ollama[312857]: load_tensors: loading model tensors, this can take a while... (mmap = true) Mar 07 12:07:18 Hermes ollama[312857]: load_tensors: offloading 32 repeating layers to GPU Mar 07 12:07:18 Hermes ollama[312857]: load_tensors: offloading output layer to GPU Mar 07 12:07:18 Hermes ollama[312857]: load_tensors: offloaded 33/33 layers to GPU Mar 07 12:07:18 Hermes ollama[312857]: load_tensors: CUDA0 model buffer size = 3847.55 MiB Mar 07 12:07:18 Hermes ollama[312857]: load_tensors: CPU_Mapped model buffer size = 70.31 MiB Mar 07 12:07:18 Hermes ollama[312857]: time=2025-03-07T12:07:18.144-05:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model" Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: n_seq_max = 4 Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: n_ctx = 8192 Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: n_ctx_per_seq = 2048 Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: n_batch = 2048 Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: n_ubatch = 512 Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: flash_attn = 0 Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: freq_base = 1000000.0 Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: freq_scale = 1 Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: n_ctx_per_seq (2048) < n_ctx_train (32768) -- the full capacity of the model will not be utilized Mar 07 12:07:18 Hermes ollama[312857]: llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 32, can_shift = 1 Mar 07 12:07:18 Hermes ollama[312857]: llama_kv_cache_init: CUDA0 KV buffer size = 1024.00 MiB Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: KV self size = 1024.00 MiB, K (f16): 512.00 MiB, V (f16): 512.00 MiB Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: CUDA_Host output buffer size = 0.55 MiB Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: CUDA0 compute buffer size = 560.00 MiB Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: CUDA_Host compute buffer size = 24.01 MiB Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: graph nodes = 1030 Mar 07 12:07:18 Hermes ollama[312857]: llama_init_from_model: graph splits = 2 Mar 07 12:07:18 Hermes ollama[312857]: key clip.use_silu not found in file Mar 07 12:07:18 Hermes ollama[312857]: key clip.vision.image_grid_pinpoints not found in file Mar 07 12:07:18 Hermes ollama[312857]: key clip.vision.feature_layer not found in file Mar 07 12:07:18 Hermes ollama[312857]: key clip.vision.mm_patch_merge_type not found in file Mar 07 12:07:18 Hermes ollama[312857]: key clip.vision.image_crop_resolution not found in file Mar 07 12:07:18 Hermes ollama[312857]: time=2025-03-07T12:07:18.395-05:00 level=INFO source=server.go:596 msg="llama runner started in 0.50 seconds" Mar 07 12:07:19 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:19 | 200 | 2.050519922s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:19 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:19 | 200 | 2.137203874s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:20 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:20 | 200 | 2.825573417s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:21 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:21 | 200 | 3.853311436s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:21 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:21 | 200 | 4.180678599s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:22 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:22 | 200 | 4.469883597s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:23 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:23 | 200 | 5.515971935s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:23 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:23 | 200 | 5.524816156s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:23 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:23 | 200 | 5.623232734s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:24 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:24 | 200 | 6.660335345s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:24 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:24 | 200 | 4.795856041s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:25 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:25 | 200 | 5.490165566s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:25 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:25 | 200 | 4.278931481s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:26 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:26 | 200 | 5.386905738s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:26 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:26 | 200 | 4.3196965s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:27 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:27 | 200 | 4.826283861s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:27 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:27 | 200 | 4.389918954s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:28 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:28 | 200 | 5.264628449s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:29 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:29 | 200 | 4.613452959s | 127.0.0.1 | POST "/api/chat" Mar 07 12:07:29 Hermes ollama[312857]: [GIN] 2025/03/07 - 12:07:29 | 200 | 5.804187676s | 127.0.0.1 | POST "/api/chat"
GiteaMirror commented 2026-04-12 17:39:49 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 7, 2025):

Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.892-05:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /usr/share/ollama/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --mmproj /usr/share/ollama/.ollama/models/blobs/sha256-72d6f08a42f656d36b356dbe0920675899a99ce21192fd66266fb7d82ed07539 --threads 16 --parallel 4 --port 35805"

OLLAMA_NUM_PARALLEL is not 10, it's unset and ollama is using a default of 4 (--parallel 4 in the above line).

What's the output of

systemctl cat ollama

Perhaps turning it off and on again will help.

sudo systemctl stop ollama
sudo systemctl start ollama
<!-- gh-comment-id:2707211313 --> @rick-github commented on GitHub (Mar 7, 2025): ``` Mar 07 12:07:17 Hermes ollama[312857]: time=2025-03-07T12:07:17.892-05:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /usr/share/ollama/.ollama/models/blobs/sha256-170370233dd5c5415250a2ecd5c71586352850729062ccef1496385647293868 --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --mmproj /usr/share/ollama/.ollama/models/blobs/sha256-72d6f08a42f656d36b356dbe0920675899a99ce21192fd66266fb7d82ed07539 --threads 16 --parallel 4 --port 35805" ``` `OLLAMA_NUM_PARALLEL` is not 10, it's unset and ollama is using a default of 4 (`--parallel 4` in the above line). What's the output of ``` systemctl cat ollama ``` Perhaps turning it off and on again will help. ``` sudo systemctl stop ollama sudo systemctl start ollama ```
GiteaMirror commented 2026-04-12 17:39:50 -05:00
Author
Owner
Copy Link

@EdwinMeriaux commented on GitHub (Mar 7, 2025):

I tried turning it off and on and no change the output of

systemtl cat ollama is:

emeriaux@Hermes:~ sudo systemctl start ollama emeriaux@Hermes:~ systemctl cat ollama

/etc/systemd/system/ollama.service

[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/local/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
Environment="PATH=/home/thalassa/emeriaux/.cargo/bin:/usr/local/sbin:/usr/local>

[Install]
WantedBy=default.target

/etc/systemd/system/ollama.service.d/override.conf

Environment="OLLAMA_NUM_PARALLEL=10"
Environment="OLLAMA_GPU_MEMORY=20GiB"
lines 1-19/19 (END)...skipping...

/etc/systemd/system/ollama.service

[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/local/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
Environment="PATH=/home/thalassa/emeriaux/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/thalassa/emeriaux/.vscode/extensions/ms-python.debugpy-2025.0.0-linux-x64/bundl>

[Install]
WantedBy=default.target

/etc/systemd/system/ollama.service.d/override.conf

Environment="OLLAMA_NUM_PARALLEL=10"
Environment="OLLAMA_GPU_MEMORY=20GiB"

<!-- gh-comment-id:2707237240 --> @EdwinMeriaux commented on GitHub (Mar 7, 2025): I tried turning it off and on and no change the output of systemtl cat ollama is: emeriaux@Hermes:~$ sudo systemctl start ollama emeriaux@Hermes:~$ systemctl cat ollama # /etc/systemd/system/ollama.service [Unit] Description=Ollama Service After=network-online.target [Service] ExecStart=/usr/local/bin/ollama serve User=ollama Group=ollama Restart=always RestartSec=3 Environment="PATH=/home/thalassa/emeriaux/.cargo/bin:/usr/local/sbin:/usr/local> [Install] WantedBy=default.target # /etc/systemd/system/ollama.service.d/override.conf Environment="OLLAMA_NUM_PARALLEL=10" Environment="OLLAMA_GPU_MEMORY=20GiB" lines 1-19/19 (END)...skipping... # /etc/systemd/system/ollama.service [Unit] Description=Ollama Service After=network-online.target [Service] ExecStart=/usr/local/bin/ollama serve User=ollama Group=ollama Restart=always RestartSec=3 Environment="PATH=/home/thalassa/emeriaux/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/thalassa/emeriaux/.vscode/extensions/ms-python.debugpy-2025.0.0-linux-x64/bundl> [Install] WantedBy=default.target # /etc/systemd/system/ollama.service.d/override.conf Environment="OLLAMA_NUM_PARALLEL=10" Environment="OLLAMA_GPU_MEMORY=20GiB"
GiteaMirror commented 2026-04-12 17:39:51 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 7, 2025):

Put

[Service]

above the Environment lines. Also, OLLAMA_GPU_MEMORY is not an ollama configuration variable.

<!-- gh-comment-id:2707251796 --> @rick-github commented on GitHub (Mar 7, 2025): Put ``` [Service] ``` above the `Environment` lines. Also, `OLLAMA_GPU_MEMORY` is not an ollama configuration variable.
GiteaMirror commented 2026-04-12 17:39:52 -05:00
Author
Owner
Copy Link

@EdwinMeriaux commented on GitHub (Mar 7, 2025):

Hello! Amazing that resolved the issue when I use llava!

unfortunately it doesn't change anything when I use llama3.2-vision. it only ever runs sequentially. my code is the same and I know given the 4090 I cannot run 10 parallel llama3.2 but if I run 2 the memory is not doubled even if there is still more than half of the gpu vram free:

import asyncio
import time
import ollama

parallel_num = 10 # Max concurrent requests
total_requests = 20 # Total number of requests

async def describe_image(number):
"""Asynchronous function to send a request to Ollama."""
try:
client = ollama.AsyncClient() # No async with
res = await client.chat(
model="llama3.2-vision",
#model="llava",
messages=[{'role': 'user', 'content': 'Describe Fermat's Last Theorem'}]
)
print(f"Completed request {number}")
except Exception as e:
print(f"Error on request {number}: {e}")

async def main():
start_time = time.time() # Start timing

semaphore = asyncio.Semaphore(parallel_num)  # Limit concurrent tasks

async def limited_request(number):
    async with semaphore:
        await describe_image(number)

tasks = [limited_request(i) for i in range(total_requests)]
await asyncio.gather(*tasks)  # Run tasks in parallel

end_time = time.time()  # End timing
print(f"All requests completed in {end_time - start_time:.2f} seconds.")

if name == "main":
asyncio.run(main()) # Run the async function

Would you know why?

<!-- gh-comment-id:2707303428 --> @EdwinMeriaux commented on GitHub (Mar 7, 2025): Hello! Amazing that resolved the issue when I use llava! unfortunately it doesn't change anything when I use llama3.2-vision. it only ever runs sequentially. my code is the same and I know given the 4090 I cannot run 10 parallel llama3.2 but if I run 2 the memory is not doubled even if there is still more than half of the gpu vram free: import asyncio import time import ollama parallel_num = 10 # Max concurrent requests total_requests = 20 # Total number of requests async def describe_image(number): """Asynchronous function to send a request to Ollama.""" try: client = ollama.AsyncClient() # No async with res = await client.chat( model="llama3.2-vision", #model="llava", messages=[{'role': 'user', 'content': 'Describe Fermat\'s Last Theorem'}] ) print(f"Completed request {number}") except Exception as e: print(f"Error on request {number}: {e}") async def main(): start_time = time.time() # Start timing semaphore = asyncio.Semaphore(parallel_num) # Limit concurrent tasks async def limited_request(number): async with semaphore: await describe_image(number) tasks = [limited_request(i) for i in range(total_requests)] await asyncio.gather(*tasks) # Run tasks in parallel end_time = time.time() # End timing print(f"All requests completed in {end_time - start_time:.2f} seconds.") if __name__ == "__main__": asyncio.run(main()) # Run the async function Would you know why?
GiteaMirror commented 2026-04-12 17:39:54 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 7, 2025):

Some vision models don't support parallelism.

model parallel
gemma3 yes
llama3.2-vision no
granite3.2-vision yes
moondream yes
minicpm-v yes
llava yes
llava-phi3 yes
llava-llama3 yes
bakllava yes
<!-- gh-comment-id:2707382182 --> @rick-github commented on GitHub (Mar 7, 2025): Some vision models don't support parallelism. | model | parallel | | --- | --- | | gemma3 | yes | | llama3.2-vision | no | | granite3.2-vision | yes | | moondream | yes | | minicpm-v | yes | | llava | yes | | llava-phi3 | yes | | llava-llama3 | yes | | bakllava | yes |
GiteaMirror commented 2026-04-12 17:39:56 -05:00
Author
Owner
Copy Link

@EdwinMeriaux commented on GitHub (Mar 7, 2025):

okkkk! Thanks! where is that table which lists this for all models?

<!-- gh-comment-id:2707395143 --> @EdwinMeriaux commented on GitHub (Mar 7, 2025): okkkk! Thanks! where is that table which lists this for all models?
GiteaMirror commented 2026-04-12 17:39:57 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 7, 2025):

I created the table from a script I ran over local vision models I have. Note that llama3.2-vision will probably get parallelism eventually, but other new vision models that are added to the ollama library may not support it out of the gate.

<!-- gh-comment-id:2707404175 --> @rick-github commented on GitHub (Mar 7, 2025): I created the table from a script I ran over local vision models I have. Note that llama3.2-vision will probably get parallelism eventually, but other new vision models that are added to the ollama library may not support it out of the gate.
GiteaMirror commented 2026-04-12 17:39:58 -05:00
Author
Owner
Copy Link

@EdwinMeriaux commented on GitHub (Mar 7, 2025):

ok! Thanks!

On more question on this. So far I was running everything locally. Now I have ollama on a server (compute canada) I don't have access to edit the ollama.service file we edited locally with the OLLAMA_NUM_PARALLEL. Is there another way to make this change?

<!-- gh-comment-id:2707431531 --> @EdwinMeriaux commented on GitHub (Mar 7, 2025): ok! Thanks! On more question on this. So far I was running everything locally. Now I have ollama on a server (compute canada) I don't have access to edit the ollama.service file we edited locally with the OLLAMA_NUM_PARALLEL. Is there another way to make this change?
GiteaMirror commented 2026-04-12 17:39:59 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 7, 2025):

Not currently. Coincidentally, I just created a PR that allows setting num_parallel via the API or Modelfile. It's uncertain if it will be merged as the ollama team is making changes to the API. If it doesn't and the new API doesn't provide a mechanism, you would need to run another service to allow external configuration requests. It might be as simple as an ssh command, depending on the features of your cloud server.

<!-- gh-comment-id:2707490618 --> @rick-github commented on GitHub (Mar 7, 2025): Not currently. Coincidentally, I just created a [PR](https://github.com/ollama/ollama/pull/9546) that allows setting `num_parallel` via the API or Modelfile. It's uncertain if it will be merged as the ollama team is making changes to the API. If it doesn't and the new API doesn't provide a mechanism, you would need to run another service to allow external configuration requests. It might be as simple as an ssh command, depending on the features of your cloud server.
GiteaMirror commented 2026-04-12 17:40:02 -05:00
Author
Owner
Copy Link

@EdwinMeriaux commented on GitHub (Mar 8, 2025):

Ok thanks!

sorry could you explain in anymore detail how I run this other service to do this?

<!-- gh-comment-id:2707816766 --> @EdwinMeriaux commented on GitHub (Mar 8, 2025): Ok thanks! sorry could you explain in anymore detail how I run this other service to do this?
GiteaMirror commented 2026-04-12 17:40:04 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 10, 2025):

I wrote a quick proof-of-concept below. I used docker containers because I find management easier with docker but it's not required, you can split this in to individual standalone serrvices. It also allows me to use dockerfile_inline statements so that the entire thing is one file.

There are three services: an ollama server, an nginx server that does API key verification and request routing, and a server that manipulates the ollama environment. The nginx server offers an extra endpoint, /admin, that a client can call with a list of environment variables to change. For example, if the parallelism needed to be changed:

curl -X PUT localhost:11434/admin -H 'x-api-key: adminkey' -d '{"OLLAMA_NUM_PARALLEL":3}'

You can get the current set of environment varaibles via GET:

curl -X GET localhost:11434/admin -H 'x-api-key: adminkey'

Making a change to the ollama environment requires restarting the server so only PUT /admin if the server is not doing a completion.

There are two keys allowed, one for access to the normal ollama endpoints (/api, /v1) and one that allows access to the /admin endpoint.

services:
  ollama-backend:
    container_name: ollama-backend
    image: ollama/ollama:${OLLAMA_DOCKER_TAG-latest}
    volumes:
      - ${OLLAMA_ROOT-./ollama}:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=${OLLAMA_KEEP_ALIVE--1}
      - OLLAMA_DEBUG=${OLLAMA_DEBUG-1}

  ollama-admin:
    container_name: ollama-admin
    image: ollama-admin
    build:
      dockerfile_inline: |
        FROM python:latest
        WORKDIR /app
        RUN pip install docker
        RUN cat > /app/server.py <<"EOF"
        from http.server import HTTPServer, BaseHTTPRequestHandler
        import json
        import docker

        class Handler(BaseHTTPRequestHandler):
            client = docker.DockerClient(base_url='unix://var/run/docker.sock')
            name = "ollama-backend"

            def respond(self, status, response):
                self.send_response(status)
                self.send_header("Content-type", "application/json")
                self.end_headers()
                self.wfile.write(json.dumps(response, default=str).encode('utf-8'))
                return status

            def do_GET(self):
                backend = self.client.containers.get(self.name)
                self.respond(200, backend.attrs.get("Config", {}).get("Env", []))

            def do_PUT(self):
                content_length_str = self.headers.get('Content-Length')
                if content_length_str is None:
                    content_length_str = '0'
                content_length = int(content_length_str)
                if content_length < 1:
                    return self.respond(400, {"error":"no data"})

                data = self.rfile.read(content_length).decode('utf-8')
                try:
                    data = json.loads(data)
                except:
                    return self.respond(400, {"error":"bad data","data":data})
                backend = self.client.containers.get(self.name)
                config = backend.attrs
                oldEnv = config.get("Config", {}).get("Env", [])
                newEnv = [v for v in oldEnv if v.split("=")[0] not in data.keys()] + [f"{k}={data[k]}" for k in data.keys()]
                try:
                    mounts=[docker.types.Mount(target=m["Destination"],source=m["Source"],type=m["Type"],read_only=not m["RW"],propagation=m["Propagation"]) for m in config["Mounts"]]
                    new_backend = self.client.containers.create(
                        image=config["Config"]["Image"],
                        name=f"new_{self.name}",
                        environment=newEnv,
                        mounts=mounts,
                        network_mode=config["HostConfig"]["NetworkMode"],
                        labels=config["Config"]["Labels"],
                        volumes=config["Config"]["Volumes"],
                        detach=True)
                    backend.stop()
                    new_backend.start()
                    backend.remove()
                    new_backend.rename(self.name)
                    response = {"status": True, "Env": newEnv}
                except Exception as e:
                  response = {"status": False, "error": str(e)}
                return self.respond(200, response)

        httpd = HTTPServer(("0.0.0.0", 80), Handler)
        httpd.serve_forever()
        EOF
        CMD [ "python", "/app/server.py" ]
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - ollama-backend

  ollama-nginx:
    container_name: ollama-nginx
    image: nginx-api-key
    build:
      dockerfile_inline: |
        FROM nginx:latest
        RUN mkdir -p /etc/nginx/templates && cat > /etc/nginx/templates/variables.conf.template <<"EOF"
        map $$http_x_api_key $$valid_api_key {
          default              0;
          "$$OLLAMA_API_KEY"   1;
        }
        map $$http_x_api_key $$valid_admin_key {
          default              0;
          "$$OLLAMA_ADMIN_KEY" 1;
        }
        EOF
        RUN cat > /etc/nginx/conf.d/default.conf <<"EOF"
        server {
          listen 11434;
          location ~ ^/admin {
            if ($$valid_admin_key = 0) {
              return 401; # unauthorized
            }
            proxy_pass http://ollama-admin:80;
          }
          location  ~ ^/(|api|v1) {
            if ($$valid_api_key = 0) {
              return 401; # unauthorized
            }
            proxy_pass http://ollama-backend:11434;
          }
        }
        EOF
    environment:
      OLLAMA_API_KEY: ${OLLAMA_API_KEY-apikey}
      OLLAMA_ADMIN_KEY: ${OLLAMA_ADMIN_KEY-adminkey}
    depends_on:
      - ollama-backend
      - ollama-admin
    ports:
      - 11434:11434

If docker is not to your taste, then you could create a script in the cloud server that takes as parameters a list of variables to change. The script would modify /etc/systemd/system/ollama.service.d/override.conf and then reload the ollama server to make the changes take effect. You would run this script from a remote server using ssh. This is a bit more fincky and error prone as you have to parse the conf file and add/remove elements, and the script has to run as root so perhaps some security issues.

<!-- gh-comment-id:2711492284 --> @rick-github commented on GitHub (Mar 10, 2025): I wrote a quick proof-of-concept below. I used docker containers because I find management easier with docker but it's not required, you can split this in to individual standalone serrvices. It also allows me to use `dockerfile_inline` statements so that the entire thing is one file. There are three services: an ollama server, an nginx server that does API key verification and request routing, and a server that manipulates the ollama environment. The nginx server offers an extra endpoint, `/admin`, that a client can call with a list of environment variables to change. For example, if the parallelism needed to be changed: ```sh curl -X PUT localhost:11434/admin -H 'x-api-key: adminkey' -d '{"OLLAMA_NUM_PARALLEL":3}' ``` You can get the current set of environment varaibles via GET: ```sh curl -X GET localhost:11434/admin -H 'x-api-key: adminkey' ``` Making a change to the ollama environment requires restarting the server so only PUT /admin if the server is not doing a completion. There are two keys allowed, one for access to the normal ollama endpoints (`/api`, `/v1`) and one that allows access to the `/admin` endpoint. ```yaml services: ollama-backend: container_name: ollama-backend image: ollama/ollama:${OLLAMA_DOCKER_TAG-latest} volumes: - ${OLLAMA_ROOT-./ollama}:/root/.ollama environment: - OLLAMA_KEEP_ALIVE=${OLLAMA_KEEP_ALIVE--1} - OLLAMA_DEBUG=${OLLAMA_DEBUG-1} ollama-admin: container_name: ollama-admin image: ollama-admin build: dockerfile_inline: | FROM python:latest WORKDIR /app RUN pip install docker RUN cat > /app/server.py <<"EOF" from http.server import HTTPServer, BaseHTTPRequestHandler import json import docker class Handler(BaseHTTPRequestHandler): client = docker.DockerClient(base_url='unix://var/run/docker.sock') name = "ollama-backend" def respond(self, status, response): self.send_response(status) self.send_header("Content-type", "application/json") self.end_headers() self.wfile.write(json.dumps(response, default=str).encode('utf-8')) return status def do_GET(self): backend = self.client.containers.get(self.name) self.respond(200, backend.attrs.get("Config", {}).get("Env", [])) def do_PUT(self): content_length_str = self.headers.get('Content-Length') if content_length_str is None: content_length_str = '0' content_length = int(content_length_str) if content_length < 1: return self.respond(400, {"error":"no data"}) data = self.rfile.read(content_length).decode('utf-8') try: data = json.loads(data) except: return self.respond(400, {"error":"bad data","data":data}) backend = self.client.containers.get(self.name) config = backend.attrs oldEnv = config.get("Config", {}).get("Env", []) newEnv = [v for v in oldEnv if v.split("=")[0] not in data.keys()] + [f"{k}={data[k]}" for k in data.keys()] try: mounts=[docker.types.Mount(target=m["Destination"],source=m["Source"],type=m["Type"],read_only=not m["RW"],propagation=m["Propagation"]) for m in config["Mounts"]] new_backend = self.client.containers.create( image=config["Config"]["Image"], name=f"new_{self.name}", environment=newEnv, mounts=mounts, network_mode=config["HostConfig"]["NetworkMode"], labels=config["Config"]["Labels"], volumes=config["Config"]["Volumes"], detach=True) backend.stop() new_backend.start() backend.remove() new_backend.rename(self.name) response = {"status": True, "Env": newEnv} except Exception as e: response = {"status": False, "error": str(e)} return self.respond(200, response) httpd = HTTPServer(("0.0.0.0", 80), Handler) httpd.serve_forever() EOF CMD [ "python", "/app/server.py" ] volumes: - /var/run/docker.sock:/var/run/docker.sock depends_on: - ollama-backend ollama-nginx: container_name: ollama-nginx image: nginx-api-key build: dockerfile_inline: | FROM nginx:latest RUN mkdir -p /etc/nginx/templates && cat > /etc/nginx/templates/variables.conf.template <<"EOF" map $$http_x_api_key $$valid_api_key { default 0; "$$OLLAMA_API_KEY" 1; } map $$http_x_api_key $$valid_admin_key { default 0; "$$OLLAMA_ADMIN_KEY" 1; } EOF RUN cat > /etc/nginx/conf.d/default.conf <<"EOF" server { listen 11434; location ~ ^/admin { if ($$valid_admin_key = 0) { return 401; # unauthorized } proxy_pass http://ollama-admin:80; } location ~ ^/(|api|v1) { if ($$valid_api_key = 0) { return 401; # unauthorized } proxy_pass http://ollama-backend:11434; } } EOF environment: OLLAMA_API_KEY: ${OLLAMA_API_KEY-apikey} OLLAMA_ADMIN_KEY: ${OLLAMA_ADMIN_KEY-adminkey} depends_on: - ollama-backend - ollama-admin ports: - 11434:11434 ``` If docker is not to your taste, then you could create a script in the cloud server that takes as parameters a list of variables to change. The script would modify /etc/systemd/system/ollama.service.d/override.conf and then reload the ollama server to make the changes take effect. You would run this script from a remote server using ssh. This is a bit more fincky and error prone as you have to parse the conf file and add/remove elements, and the script has to run as root so perhaps some security issues.
GiteaMirror commented 2026-04-12 17:40:05 -05:00
Author
Owner
Copy Link

@EdwinMeriaux commented on GitHub (Mar 11, 2025):

Hi!

Thank you so much!
I need to check if the docker would be doable.

The script you are referring to would be a bash type script right?

<!-- gh-comment-id:2714998171 --> @EdwinMeriaux commented on GitHub (Mar 11, 2025): Hi! Thank you so much! I need to check if the docker would be doable. The script you are referring to would be a bash type script right?
GiteaMirror commented 2026-04-12 17:40:07 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 11, 2025):

The script you are referring to would be a bash type script right?

Not necessarily, any programming language would do - perl, python, go, etc - it just needs follow the steps I outlined above.

<!-- gh-comment-id:2715014144 --> @rick-github commented on GitHub (Mar 11, 2025): > The script you are referring to would be a bash type script right? Not necessarily, any programming language would do - perl, python, go, etc - it just needs follow the steps I outlined above.
GiteaMirror commented 2026-04-12 17:40:08 -05:00
Author
Owner
Copy Link

@EdwinMeriaux commented on GitHub (Mar 12, 2025):

Hello!

Perfect I will try this! I need to contact the serve admin to get access to do this. I will get access then try ASAP!

Thanks!

<!-- gh-comment-id:2718202392 --> @EdwinMeriaux commented on GitHub (Mar 12, 2025): Hello! Perfect I will try this! I need to contact the serve admin to get access to do this. I will get access then try ASAP! Thanks!
GiteaMirror commented 2026-04-12 17:40:09 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 12, 2025):

Note that there is an intermediate form as well. You can run ollama as a service as you do now, but also run a HTTP server that supports CGI scripts. Then you can hit the webserver with a request, authenticated as required, and have it use the CGI module to modify the ollama service configuration as requested by the caller. There's lots of ways to do it, it just depends on what your comfort zone is.

<!-- gh-comment-id:2718228574 --> @rick-github commented on GitHub (Mar 12, 2025): Note that there is an intermediate form as well. You can run ollama as a service as you do now, but also run a HTTP server that supports CGI scripts. Then you can hit the webserver with a request, authenticated as required, and have it use the CGI module to modify the ollama service configuration as requested by the caller. There's lots of ways to do it, it just depends on what your comfort zone is.
GiteaMirror commented 2026-04-12 17:40:10 -05:00
Author
Owner
Copy Link

@suckseed5 commented on GitHub (Mar 13, 2025):

Hello, I deployed different models on several GPUs using Docker,
modified /etc/systemd/system/ollama.service on the host,
and saw the changes in the log, but there was no concurrency. Why is this?

<!-- gh-comment-id:2721094900 --> @suckseed5 commented on GitHub (Mar 13, 2025): Hello, I deployed different models on several GPUs using Docker, modified /etc/systemd/system/ollama.service on the host, and saw the changes in the log, but there was no concurrency. Why is this?
GiteaMirror commented 2026-04-12 17:40:10 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 13, 2025):

https://github.com/ollama/ollama/blob/main/docs/faq.md#how-does-ollama-handle-concurrent-requests

<!-- gh-comment-id:2721101141 --> @rick-github commented on GitHub (Mar 13, 2025): https://github.com/ollama/ollama/blob/main/docs/faq.md#how-does-ollama-handle-concurrent-requests
GiteaMirror commented 2026-04-12 17:40:11 -05:00
Author
Owner
Copy Link

@EdwinMeriaux commented on GitHub (Mar 17, 2025):

@rick-github Thank you so much! I have ran everything properly on the server I was using. I am not entirely sure how to settup that http server just for future knowledge. would you have an example?

<!-- gh-comment-id:2729649915 --> @EdwinMeriaux commented on GitHub (Mar 17, 2025): @rick-github Thank you so much! I have ran everything properly on the server I was using. I am not entirely sure how to settup that http server just for future knowledge. would you have an example?
GiteaMirror commented 2026-04-12 17:40:13 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 17, 2025):

https://httpd.apache.org/docs/2.4/howto/cgi.html

<!-- gh-comment-id:2729669573 --> @rick-github commented on GitHub (Mar 17, 2025): https://httpd.apache.org/docs/2.4/howto/cgi.html
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#6238
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?