github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-22 05:31:43 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #7081] Ollama performs *much* slower via API than CLI on M1 Mac #30257

New Issue
Closed
opened 2026-04-22 09:47:05 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-04-22 09:47:05 -05:00
Owner
Copy Link

Originally created by @bigxalx on GitHub (Oct 2, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/7081

What is the issue?

CLI

When I run codestral:22b-v0.1-q2_K on my M1 Macbook Air via the CLI with ollama run codestral:22b-v0.1-q2_K it performs a little slowly, but usable. When I look at ollama ps it prints the following:

NAME ID SIZE PROCESSOR UNTIL
codestral:22b-v0.1-q2_K 0e1127d332ef 9.6 GB 100% GPU 4 minutes from now

API

However, when i configure Continue to use the same model via ollama it is much slower (At least 2-5X slower) and uses much more RAM

NAME ID SIZE PROCESSOR UNTIL
codestral:22b-v0.1-q2_K 0e1127d332ef 19 GB 43%/57% CPU/GPU Stopping...

This happens regardless of whether I start ollama with ollama serve or via the Mac app.

EDIT: I just tried Llama3.2 on the CLI and with Enchanted LLM. It seemingly confirms that the problem might be with the API, as it's a different model, different app, but I experience same problem: It runs about 2-3X slower via the API than when I ask "directly" via ollama run...

EDIT2: I tested Llama3.2 and Codestral with LM Studio as backend. When i run the models directly (via GUI in this case), LM Studio is a bit slower than Ollama. When I run via API / Server, LM Studio behaves the same way as with the GUI (meaning the API calls are much faster than ollama)

EDIT3: I did some more research, testing the API with different parameters. I found that setting stream to false makes the API behave like the CLI: E.g. curl http://localhost:11434/api/generate -d '{ "model": "codestral:22b-v0.1-q2_K", "prompt": "Tell me about yourself", "stream": false }'
Same applies to Llama3.2 model
Difference between "stream: false" and "stream:true" (default), is astounding. 4GB RAM usage vs 25GB for Llama 3.2

Here are the server startup logs in case they are relevant:
2024/10/02 16:59:03 routes.go:1153: INFO server config env="map[HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/Users/big/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: http_proxy: https_proxy: no_proxy:]" time=2024-10-02T16:59:03.500+02:00 level=INFO source=images.go:753 msg="total blobs: 16" time=2024-10-02T16:59:03.500+02:00 level=INFO source=images.go:760 msg="total unused blobs removed: 0" time=2024-10-02T16:59:03.501+02:00 level=INFO source=routes.go:1200 msg="Listening on 127.0.0.1:11434 (version 0.3.12)" time=2024-10-02T16:59:03.501+02:00 level=INFO source=common.go:135 msg="extracting embedded files" dir=/var/folders/mb/gh2_kwk50611j_9p8_0pmkn00000gn/T/ollama151879657/runners time=2024-10-02T16:59:03.537+02:00 level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners=[metal] time=2024-10-02T16:59:03.572+02:00 level=INFO source=types.go:107 msg="inference compute" id=0 library=metal variant="" compute="" driver=0.0 name="" total="10.7 GiB" available="10.7 GiB" time=2024-10-02T16:59:16.411+02:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/Users/big/.ollama/models/blobs/sha256-28bfdfaeba9f51611c00ed322ba684ce6db076756dbc46643f98a8a748c5199e gpu=0 parallel=4 available=11453251584 required="3.6 GiB" time=2024-10-02T16:59:16.411+02:00 level=INFO source=server.go:103 msg="system memory" total="16.0 GiB" free="11.7 GiB" free_swap="0 B" time=2024-10-02T16:59:16.411+02:00 level=INFO source=memory.go:326 msg="offload to metal" layers.requested=-1 layers.model=31 layers.offload=31 layers.split="" memory.available="[10.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="3.6 GiB" memory.required.partial="3.6 GiB" memory.required.kv="480.0 MiB" memory.required.allocations="[3.6 GiB]" memory.weights.total="2.0 GiB" memory.weights.repeating="1.9 GiB" memory.weights.nonrepeating="81.0 MiB" memory.graph.full="960.0 MiB" memory.graph.partial="960.0 MiB" time=2024-10-02T16:59:16.412+02:00 level=INFO source=server.go:388 msg="starting llama server" cmd="/var/folders/mb/gh2_kwk50611j_9p8_0pmkn00000gn/T/ollama151879657/runners/metal/ollama_llama_server --model /Users/big/.ollama/models/blobs/sha256-28bfdfaeba9f51611c00ed322ba684ce6db076756dbc46643f98a8a748c5199e --ctx-size 16384 --batch-size 512 --embedding --log-disable --n-gpu-layers 31 --parallel 4 --port 60403" time=2024-10-02T16:59:16.452+02:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 time=2024-10-02T16:59:16.453+02:00 level=INFO source=server.go:587 msg="waiting for llama runner to start responding" time=2024-10-02T16:59:16.454+02:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error" INFO [main] build info | build=3670 commit="194ef086" tid="0x2046d4f40" timestamp=1727881156 INFO [main] system info | n_threads=4 n_threads_batch=4 system_info="AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 1 | SVE = 0 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="0x2046d4f40" timestamp=1727881156 total_threads=8 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="7" port="60403" tid="0x2046d4f40" timestamp=1727881156 llama_model_loader: loaded meta data with 19 key-value pairs and 483 tensors from /Users/big/.ollama/models/blobs/sha256-28bfdfaeba9f51611c00ed322ba684ce6db076756dbc46643f98a8a748c5199e (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = starcoder2 llama_model_loader: - kv 1: general.name str = starcoder2-3b llama_model_loader: - kv 2: starcoder2.block_count u32 = 30 llama_model_loader: - kv 3: starcoder2.context_length u32 = 16384 llama_model_loader: - kv 4: starcoder2.embedding_length u32 = 3072 llama_model_loader: - kv 5: starcoder2.feed_forward_length u32 = 12288 llama_model_loader: - kv 6: starcoder2.attention.head_count u32 = 24 llama_model_loader: - kv 7: starcoder2.attention.head_count_kv u32 = 2 llama_model_loader: - kv 8: starcoder2.rope.freq_base f32 = 999999.437500 llama_model_loader: - kv 9: starcoder2.attention.layer_norm_epsilon f32 = 0.000010 llama_model_loader: - kv 10: general.file_type u32 = 2 llama_model_loader: - kv 11: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 12: tokenizer.ggml.tokens arr[str,49152] = ["<|endoftext|>", "", "' llm_load_print_meta: EOS token = 0 '<|endoftext|>' llm_load_print_meta: UNK token = 0 '<|endoftext|>' llm_load_print_meta: LF token = 164 'Ä' llm_load_print_meta: EOT token = 0 '<|endoftext|>' llm_load_print_meta: max token length = 512 llm_load_tensors: ggml ctx size = 0.40 MiB ggml_backend_metal_log_allocated_size: allocated buffer, size = 1629.03 MiB, ( 1629.09 / 10922.67) llm_load_tensors: offloading 30 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 31/31 layers to GPU llm_load_tensors: CPU buffer size = 81.00 MiB llm_load_tensors: Metal buffer size = 1629.02 MiB llama_new_context_with_model: n_ctx = 16384 llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 999999.4 llama_new_context_with_model: freq_scale = 1 ggml_metal_init: allocating ggml_metal_init: found device: Apple M1 ggml_metal_init: picking default device: Apple M1 ggml_metal_init: using embedded metal library ggml_metal_init: GPU name: Apple M1 ggml_metal_init: GPU family: MTLGPUFamilyApple7 (1007) ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003) ggml_metal_init: GPU family: MTLGPUFamilyMetal3 (5001) ggml_metal_init: simdgroup reduction support = true ggml_metal_init: simdgroup matrix mul. support = true ggml_metal_init: hasUnifiedMemory = true ggml_metal_init: recommendedMaxWorkingSetSize = 11453.25 MB llama_kv_cache_init: Metal KV buffer size = 480.00 MiB llama_new_context_with_model: KV self size = 480.00 MiB, K (f16): 240.00 MiB, V (f16): 240.00 MiB llama_new_context_with_model: CPU output buffer size = 0.80 MiB llama_new_context_with_model: Metal compute buffer size = 824.00 MiB llama_new_context_with_model: CPU compute buffer size = 38.01 MiB llama_new_context_with_model: graph nodes = 1147 llama_new_context_with_model: graph splits = 2 time=2024-10-02T16:59:16.705+02:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server loading model" INFO [main] model loaded | tid="0x2046d4f40" timestamp=1727881158 time=2024-10-02T16:59:18.968+02:00 level=INFO source=server.go:626 msg="llama runner started in 2.51 seconds" [GIN] 2024/10/02 - 16:59:19 | 200 | 3.135537834s | 127.0.0.1 | POST "/api/generate" [GIN] 2024/10/02 - 16:59:27 | 200 | 1.585210792s | 127.0.0.1 | POST "/api/generate" [GIN] 2024/10/02 - 16:59:28 | 200 | 1.736311083s | 127.0.0.1 | POST "/api/generate" [GIN] 2024/10/02 - 16:59:29 | 200 | 2.43617475s | 127.0.0.1 | POST "/api/generate" [GIN] 2024/10/02 - 16:59:30 | 200 | 1.321447333s | 127.0.0.1 | POST "/api/generate" [GIN] 2024/10/02 - 16:59:35 | 200 | 1.341725042s | 127.0.0.1 | POST "/api/generate" [GIN] 2024/10/02 - 16:59:36 | 200 | 1.361396667s | 127.0.0.1 | POST "/api/generate" time=2024-10-02T16:59:46.755+02:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=0 library=metal total="10.7 GiB" available="7.1 GiB" time=2024-10-02T16:59:46.790+02:00 level=INFO source=server.go:103 msg="system memory" total="16.0 GiB" free="11.5 GiB" free_swap="0 B" time=2024-10-02T16:59:46.791+02:00 level=INFO source=memory.go:326 msg="offload to metal" layers.requested=-1 layers.model=57 layers.offload=26 layers.split="" memory.available="[10.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="18.5 GiB" memory.required.partial="10.6 GiB" memory.required.kv="7.0 GiB" memory.required.allocations="[10.6 GiB]" memory.weights.total="14.5 GiB" memory.weights.repeating="14.3 GiB" memory.weights.nonrepeating="157.5 MiB" memory.graph.full="3.1 GiB" memory.graph.partial="3.1 GiB" time=2024-10-02T16:59:46.791+02:00 level=INFO source=server.go:388 msg="starting llama server" cmd="/var/folders/mb/gh2_kwk50611j_9p8_0pmkn00000gn/T/ollama151879657/runners/metal/ollama_llama_server --model /Users/big/.ollama/models/blobs/sha256-a645a2a1d407b876edf4731dd223cf8a09fa168efc96b20496a89bcdf702f7b4 --ctx-size 32768 --batch-size 512 --embedding --log-disable --n-gpu-layers 26 --no-mmap --parallel 1 --port 60468" time=2024-10-02T16:59:46.793+02:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 time=2024-10-02T16:59:46.793+02:00 level=INFO source=server.go:587 msg="waiting for llama runner to start responding" time=2024-10-02T16:59:46.793+02:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error" INFO [main] build info | build=3670 commit="194ef086" tid="0x2046d4f40" timestamp=1727881186 INFO [main] system info | n_threads=4 n_threads_batch=4 system_info="AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 1 | SVE = 0 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="0x2046d4f40" timestamp=1727881186 total_threads=8 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="7" port="60468" tid="0x2046d4f40" timestamp=1727881186 llama_model_loader: loaded meta data with 25 key-value pairs and 507 tensors from /Users/big/.ollama/models/blobs/sha256-a645a2a1d407b876edf4731dd223cf8a09fa168efc96b20496a89bcdf702f7b4 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.name str = Codestral-22B-v0.1 llama_model_loader: - kv 2: llama.block_count u32 = 56 llama_model_loader: - kv 3: llama.context_length u32 = 32768 llama_model_loader: - kv 4: llama.embedding_length u32 = 6144 llama_model_loader: - kv 5: llama.feed_forward_length u32 = 16384 llama_model_loader: - kv 6: llama.attention.head_count u32 = 48 llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 8: llama.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 10: general.file_type u32 = 10 llama_model_loader: - kv 11: llama.vocab_size u32 = 32768 llama_model_loader: - kv 12: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 13: tokenizer.ggml.add_space_prefix bool = true llama_model_loader: - kv 14: tokenizer.ggml.model str = llama llama_model_loader: - kv 15: tokenizer.ggml.pre str = default llama_model_loader: - kv 16: tokenizer.ggml.tokens arr[str,32768] = ["", "", "", "[INST]", "[... llama_model_loader: - kv 17: tokenizer.ggml.scores arr[f32,32768] = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,32768] = [2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ... llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 21: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 24: general.quantization_version u32 = 2 llama_model_loader: - type f32: 113 tensors llama_model_loader: - type q2_K: 225 tensors llama_model_loader: - type q3_K: 112 tensors llama_model_loader: - type q4_K: 56 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special tokens cache size = 771 llm_load_vocab: token to piece cache size = 0.1731 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32768 llm_load_print_meta: n_merges = 0 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 6144 llm_load_print_meta: n_layer = 56 llm_load_print_meta: n_head = 48 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 6 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 16384 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 1000000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = ?B llm_load_print_meta: model ftype = Q2_K - Medium llm_load_print_meta: model params = 22.25 B llm_load_print_meta: model size = 7.70 GiB (2.97 BPW) llm_load_print_meta: general.name = Codestral-22B-v0.1 llm_load_print_meta: BOS token = 1 '' llm_load_print_meta: EOS token = 2 '' llm_load_print_meta: UNK token = 0 '' llm_load_print_meta: LF token = 781 '<0x0A>' llm_load_print_meta: max token length = 48 llm_load_tensors: ggml ctx size = 0.47 MiB llm_load_tensors: offloading 26 repeating layers to GPU llm_load_tensors: offloaded 26/57 layers to GPU llm_load_tensors: CPU buffer size = 4328.18 MiB llm_load_tensors: Metal buffer size = 3559.97 MiB time=2024-10-02T16:59:47.045+02:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server loading model" llama_new_context_with_model: n_ctx = 32768 llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 1000000.0 llama_new_context_with_model: freq_scale = 1 ggml_metal_init: allocating ggml_metal_init: found device: Apple M1 ggml_metal_init: picking default device: Apple M1 ggml_metal_init: using embedded metal library ggml_metal_init: GPU name: Apple M1 ggml_metal_init: GPU family: MTLGPUFamilyApple7 (1007) ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003) ggml_metal_init: GPU family: MTLGPUFamilyMetal3 (5001) ggml_metal_init: simdgroup reduction support = true ggml_metal_init: simdgroup matrix mul. support = true ggml_metal_init: hasUnifiedMemory = true ggml_metal_init: recommendedMaxWorkingSetSize = 11453.25 MB llama_kv_cache_init: CPU KV buffer size = 3840.00 MiB llama_kv_cache_init: Metal KV buffer size = 3328.00 MiB llama_new_context_with_model: KV self size = 7168.00 MiB, K (f16): 3584.00 MiB, V (f16): 3584.00 MiB llama_new_context_with_model: CPU output buffer size = 0.15 MiB llama_new_context_with_model: Metal compute buffer size = 3184.00 MiB llama_new_context_with_model: CPU compute buffer size = 3184.01 MiB llama_new_context_with_model: graph nodes = 1798 llama_new_context_with_model: graph splits = 483 ^Ctime=2024-10-02T17:00:05.986+02:00 level=WARN source=server.go:594 msg="client connection closed before server finished loading, aborting load" time=2024-10-02T17:00:05.987+02:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="timed out waiting for llama runner to start: context canceled" [GIN] 2024/10/02 - 17:00:05 | 499 | 19.265245125s | 127.0.0.1 | POST "/api/generate"

OS

macOS

GPU

Apple

CPU

Apple

Ollama version

0.3.12

Originally created by @bigxalx on GitHub (Oct 2, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/7081 ### What is the issue? ### CLI When I run **codestral:22b-v0.1-q2_K** on my M1 Macbook Air via the CLI with `ollama run codestral:22b-v0.1-q2_K` it performs a little slowly, but usable. When I look at `ollama ps` it prints the following: | NAME | ID | SIZE | PROCESSOR | UNTIL | |--------------------------|---------------|---------|------------|---------------------| | codestral:22b-v0.1-q2_K | 0e1127d332ef | 9.6 GB | 100% GPU | 4 minutes from now | ### API However, when i configure [Continue](https://continue.dev) to use the same model via ollama it is _much_ slower (At least 2-5X slower) and uses much more RAM | NAME | ID | SIZE | PROCESSOR | UNTIL | |--------------------------|---------------|---------|--------------------|--------------| | codestral:22b-v0.1-q2_K | 0e1127d332ef | 19 GB | 43%/57% CPU/GPU | Stopping... | --- This happens regardless of whether I start ollama with `ollama serve` or via the Mac app. EDIT: I just tried Llama3.2 on the CLI and with [Enchanted LLM](https://github.com/AugustDev/enchanted). It seemingly confirms that the problem might be with the API, as it's a different model, different app, but I experience same problem: It runs about 2-3X slower via the API than when I ask "directly" via `ollama run...` EDIT2: I tested Llama3.2 and Codestral with LM Studio as backend. When i run the models directly (via GUI in this case), LM Studio is a bit slower than Ollama. When I run via API / Server, LM Studio behaves the same way as with the GUI (meaning the API calls are much faster than ollama) EDIT3: I did some more research, testing the API with different parameters. I found that setting stream to false makes the API behave like the CLI: E.g. curl ```http://localhost:11434/api/generate -d '{ "model": "codestral:22b-v0.1-q2_K", "prompt": "Tell me about yourself", "stream": false }'``` Same applies to Llama3.2 model Difference between "stream: false" and "stream:true" (default), is astounding. 4GB RAM usage vs 25GB for Llama 3.2 <summary> Here are the server startup logs in case they are relevant: </summary> <details> 2024/10/02 16:59:03 routes.go:1153: INFO server config env="map[HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/Users/big/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: http_proxy: https_proxy: no_proxy:]" time=2024-10-02T16:59:03.500+02:00 level=INFO source=images.go:753 msg="total blobs: 16" time=2024-10-02T16:59:03.500+02:00 level=INFO source=images.go:760 msg="total unused blobs removed: 0" time=2024-10-02T16:59:03.501+02:00 level=INFO source=routes.go:1200 msg="Listening on 127.0.0.1:11434 (version 0.3.12)" time=2024-10-02T16:59:03.501+02:00 level=INFO source=common.go:135 msg="extracting embedded files" dir=/var/folders/mb/gh2_kwk50611j_9p8_0pmkn00000gn/T/ollama151879657/runners time=2024-10-02T16:59:03.537+02:00 level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners=[metal] time=2024-10-02T16:59:03.572+02:00 level=INFO source=types.go:107 msg="inference compute" id=0 library=metal variant="" compute="" driver=0.0 name="" total="10.7 GiB" available="10.7 GiB" time=2024-10-02T16:59:16.411+02:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/Users/big/.ollama/models/blobs/sha256-28bfdfaeba9f51611c00ed322ba684ce6db076756dbc46643f98a8a748c5199e gpu=0 parallel=4 available=11453251584 required="3.6 GiB" time=2024-10-02T16:59:16.411+02:00 level=INFO source=server.go:103 msg="system memory" total="16.0 GiB" free="11.7 GiB" free_swap="0 B" time=2024-10-02T16:59:16.411+02:00 level=INFO source=memory.go:326 msg="offload to metal" layers.requested=-1 layers.model=31 layers.offload=31 layers.split="" memory.available="[10.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="3.6 GiB" memory.required.partial="3.6 GiB" memory.required.kv="480.0 MiB" memory.required.allocations="[3.6 GiB]" memory.weights.total="2.0 GiB" memory.weights.repeating="1.9 GiB" memory.weights.nonrepeating="81.0 MiB" memory.graph.full="960.0 MiB" memory.graph.partial="960.0 MiB" time=2024-10-02T16:59:16.412+02:00 level=INFO source=server.go:388 msg="starting llama server" cmd="/var/folders/mb/gh2_kwk50611j_9p8_0pmkn00000gn/T/ollama151879657/runners/metal/ollama_llama_server --model /Users/big/.ollama/models/blobs/sha256-28bfdfaeba9f51611c00ed322ba684ce6db076756dbc46643f98a8a748c5199e --ctx-size 16384 --batch-size 512 --embedding --log-disable --n-gpu-layers 31 --parallel 4 --port 60403" time=2024-10-02T16:59:16.452+02:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 time=2024-10-02T16:59:16.453+02:00 level=INFO source=server.go:587 msg="waiting for llama runner to start responding" time=2024-10-02T16:59:16.454+02:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error" INFO [main] build info | build=3670 commit="194ef086" tid="0x2046d4f40" timestamp=1727881156 INFO [main] system info | n_threads=4 n_threads_batch=4 system_info="AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 1 | SVE = 0 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="0x2046d4f40" timestamp=1727881156 total_threads=8 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="7" port="60403" tid="0x2046d4f40" timestamp=1727881156 llama_model_loader: loaded meta data with 19 key-value pairs and 483 tensors from /Users/big/.ollama/models/blobs/sha256-28bfdfaeba9f51611c00ed322ba684ce6db076756dbc46643f98a8a748c5199e (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = starcoder2 llama_model_loader: - kv 1: general.name str = starcoder2-3b llama_model_loader: - kv 2: starcoder2.block_count u32 = 30 llama_model_loader: - kv 3: starcoder2.context_length u32 = 16384 llama_model_loader: - kv 4: starcoder2.embedding_length u32 = 3072 llama_model_loader: - kv 5: starcoder2.feed_forward_length u32 = 12288 llama_model_loader: - kv 6: starcoder2.attention.head_count u32 = 24 llama_model_loader: - kv 7: starcoder2.attention.head_count_kv u32 = 2 llama_model_loader: - kv 8: starcoder2.rope.freq_base f32 = 999999.437500 llama_model_loader: - kv 9: starcoder2.attention.layer_norm_epsilon f32 = 0.000010 llama_model_loader: - kv 10: general.file_type u32 = 2 llama_model_loader: - kv 11: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 12: tokenizer.ggml.tokens arr[str,49152] = ["<|endoftext|>", "<fim_prefix>", "<f... llama_model_loader: - kv 13: tokenizer.ggml.token_type arr[i32,49152] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ... llama_model_loader: - kv 14: tokenizer.ggml.merges arr[str,48872] = ["Ġ Ġ", "ĠĠ ĠĠ", "ĠĠĠĠ ĠĠ... llama_model_loader: - kv 15: tokenizer.ggml.bos_token_id u32 = 0 llama_model_loader: - kv 16: tokenizer.ggml.eos_token_id u32 = 0 llama_model_loader: - kv 17: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 18: general.quantization_version u32 = 2 llama_model_loader: - type f32: 302 tensors llama_model_loader: - type q4_0: 181 tensors llm_load_vocab: missing or unrecognized pre-tokenizer type, using: 'default' llm_load_vocab: special tokens cache size = 38 llm_load_vocab: token to piece cache size = 0.2828 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = starcoder2 llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 49152 llm_load_print_meta: n_merges = 48872 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 16384 llm_load_print_meta: n_embd = 3072 llm_load_print_meta: n_layer = 30 llm_load_print_meta: n_head = 24 llm_load_print_meta: n_head_kv = 2 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 12 llm_load_print_meta: n_embd_k_gqa = 256 llm_load_print_meta: n_embd_v_gqa = 256 llm_load_print_meta: f_norm_eps = 1.0e-05 llm_load_print_meta: f_norm_rms_eps = 0.0e+00 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 12288 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 2 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 999999.4 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 16384 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = 3B llm_load_print_meta: model ftype = Q4_0 llm_load_print_meta: model params = 3.03 B llm_load_print_meta: model size = 1.59 GiB (4.51 BPW) llm_load_print_meta: general.name = starcoder2-3b llm_load_print_meta: BOS token = 0 '<|endoftext|>' llm_load_print_meta: EOS token = 0 '<|endoftext|>' llm_load_print_meta: UNK token = 0 '<|endoftext|>' llm_load_print_meta: LF token = 164 'Ä' llm_load_print_meta: EOT token = 0 '<|endoftext|>' llm_load_print_meta: max token length = 512 llm_load_tensors: ggml ctx size = 0.40 MiB ggml_backend_metal_log_allocated_size: allocated buffer, size = 1629.03 MiB, ( 1629.09 / 10922.67) llm_load_tensors: offloading 30 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 31/31 layers to GPU llm_load_tensors: CPU buffer size = 81.00 MiB llm_load_tensors: Metal buffer size = 1629.02 MiB llama_new_context_with_model: n_ctx = 16384 llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 999999.4 llama_new_context_with_model: freq_scale = 1 ggml_metal_init: allocating ggml_metal_init: found device: Apple M1 ggml_metal_init: picking default device: Apple M1 ggml_metal_init: using embedded metal library ggml_metal_init: GPU name: Apple M1 ggml_metal_init: GPU family: MTLGPUFamilyApple7 (1007) ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003) ggml_metal_init: GPU family: MTLGPUFamilyMetal3 (5001) ggml_metal_init: simdgroup reduction support = true ggml_metal_init: simdgroup matrix mul. support = true ggml_metal_init: hasUnifiedMemory = true ggml_metal_init: recommendedMaxWorkingSetSize = 11453.25 MB llama_kv_cache_init: Metal KV buffer size = 480.00 MiB llama_new_context_with_model: KV self size = 480.00 MiB, K (f16): 240.00 MiB, V (f16): 240.00 MiB llama_new_context_with_model: CPU output buffer size = 0.80 MiB llama_new_context_with_model: Metal compute buffer size = 824.00 MiB llama_new_context_with_model: CPU compute buffer size = 38.01 MiB llama_new_context_with_model: graph nodes = 1147 llama_new_context_with_model: graph splits = 2 time=2024-10-02T16:59:16.705+02:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server loading model" INFO [main] model loaded | tid="0x2046d4f40" timestamp=1727881158 time=2024-10-02T16:59:18.968+02:00 level=INFO source=server.go:626 msg="llama runner started in 2.51 seconds" [GIN] 2024/10/02 - 16:59:19 | 200 | 3.135537834s | 127.0.0.1 | POST "/api/generate" [GIN] 2024/10/02 - 16:59:27 | 200 | 1.585210792s | 127.0.0.1 | POST "/api/generate" [GIN] 2024/10/02 - 16:59:28 | 200 | 1.736311083s | 127.0.0.1 | POST "/api/generate" [GIN] 2024/10/02 - 16:59:29 | 200 | 2.43617475s | 127.0.0.1 | POST "/api/generate" [GIN] 2024/10/02 - 16:59:30 | 200 | 1.321447333s | 127.0.0.1 | POST "/api/generate" [GIN] 2024/10/02 - 16:59:35 | 200 | 1.341725042s | 127.0.0.1 | POST "/api/generate" [GIN] 2024/10/02 - 16:59:36 | 200 | 1.361396667s | 127.0.0.1 | POST "/api/generate" time=2024-10-02T16:59:46.755+02:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=0 library=metal total="10.7 GiB" available="7.1 GiB" time=2024-10-02T16:59:46.790+02:00 level=INFO source=server.go:103 msg="system memory" total="16.0 GiB" free="11.5 GiB" free_swap="0 B" time=2024-10-02T16:59:46.791+02:00 level=INFO source=memory.go:326 msg="offload to metal" layers.requested=-1 layers.model=57 layers.offload=26 layers.split="" memory.available="[10.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="18.5 GiB" memory.required.partial="10.6 GiB" memory.required.kv="7.0 GiB" memory.required.allocations="[10.6 GiB]" memory.weights.total="14.5 GiB" memory.weights.repeating="14.3 GiB" memory.weights.nonrepeating="157.5 MiB" memory.graph.full="3.1 GiB" memory.graph.partial="3.1 GiB" time=2024-10-02T16:59:46.791+02:00 level=INFO source=server.go:388 msg="starting llama server" cmd="/var/folders/mb/gh2_kwk50611j_9p8_0pmkn00000gn/T/ollama151879657/runners/metal/ollama_llama_server --model /Users/big/.ollama/models/blobs/sha256-a645a2a1d407b876edf4731dd223cf8a09fa168efc96b20496a89bcdf702f7b4 --ctx-size 32768 --batch-size 512 --embedding --log-disable --n-gpu-layers 26 --no-mmap --parallel 1 --port 60468" time=2024-10-02T16:59:46.793+02:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 time=2024-10-02T16:59:46.793+02:00 level=INFO source=server.go:587 msg="waiting for llama runner to start responding" time=2024-10-02T16:59:46.793+02:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error" INFO [main] build info | build=3670 commit="194ef086" tid="0x2046d4f40" timestamp=1727881186 INFO [main] system info | n_threads=4 n_threads_batch=4 system_info="AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 1 | SVE = 0 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="0x2046d4f40" timestamp=1727881186 total_threads=8 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="7" port="60468" tid="0x2046d4f40" timestamp=1727881186 llama_model_loader: loaded meta data with 25 key-value pairs and 507 tensors from /Users/big/.ollama/models/blobs/sha256-a645a2a1d407b876edf4731dd223cf8a09fa168efc96b20496a89bcdf702f7b4 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.name str = Codestral-22B-v0.1 llama_model_loader: - kv 2: llama.block_count u32 = 56 llama_model_loader: - kv 3: llama.context_length u32 = 32768 llama_model_loader: - kv 4: llama.embedding_length u32 = 6144 llama_model_loader: - kv 5: llama.feed_forward_length u32 = 16384 llama_model_loader: - kv 6: llama.attention.head_count u32 = 48 llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 8: llama.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 10: general.file_type u32 = 10 llama_model_loader: - kv 11: llama.vocab_size u32 = 32768 llama_model_loader: - kv 12: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 13: tokenizer.ggml.add_space_prefix bool = true llama_model_loader: - kv 14: tokenizer.ggml.model str = llama llama_model_loader: - kv 15: tokenizer.ggml.pre str = default llama_model_loader: - kv 16: tokenizer.ggml.tokens arr[str,32768] = ["<unk>", "<s>", "</s>", "[INST]", "[... llama_model_loader: - kv 17: tokenizer.ggml.scores arr[f32,32768] = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,32768] = [2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ... llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 21: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 24: general.quantization_version u32 = 2 llama_model_loader: - type f32: 113 tensors llama_model_loader: - type q2_K: 225 tensors llama_model_loader: - type q3_K: 112 tensors llama_model_loader: - type q4_K: 56 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special tokens cache size = 771 llm_load_vocab: token to piece cache size = 0.1731 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32768 llm_load_print_meta: n_merges = 0 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 6144 llm_load_print_meta: n_layer = 56 llm_load_print_meta: n_head = 48 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 6 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 16384 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 1000000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = ?B llm_load_print_meta: model ftype = Q2_K - Medium llm_load_print_meta: model params = 22.25 B llm_load_print_meta: model size = 7.70 GiB (2.97 BPW) llm_load_print_meta: general.name = Codestral-22B-v0.1 llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 781 '<0x0A>' llm_load_print_meta: max token length = 48 llm_load_tensors: ggml ctx size = 0.47 MiB llm_load_tensors: offloading 26 repeating layers to GPU llm_load_tensors: offloaded 26/57 layers to GPU llm_load_tensors: CPU buffer size = 4328.18 MiB llm_load_tensors: Metal buffer size = 3559.97 MiB time=2024-10-02T16:59:47.045+02:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server loading model" llama_new_context_with_model: n_ctx = 32768 llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 1000000.0 llama_new_context_with_model: freq_scale = 1 ggml_metal_init: allocating ggml_metal_init: found device: Apple M1 ggml_metal_init: picking default device: Apple M1 ggml_metal_init: using embedded metal library ggml_metal_init: GPU name: Apple M1 ggml_metal_init: GPU family: MTLGPUFamilyApple7 (1007) ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003) ggml_metal_init: GPU family: MTLGPUFamilyMetal3 (5001) ggml_metal_init: simdgroup reduction support = true ggml_metal_init: simdgroup matrix mul. support = true ggml_metal_init: hasUnifiedMemory = true ggml_metal_init: recommendedMaxWorkingSetSize = 11453.25 MB llama_kv_cache_init: CPU KV buffer size = 3840.00 MiB llama_kv_cache_init: Metal KV buffer size = 3328.00 MiB llama_new_context_with_model: KV self size = 7168.00 MiB, K (f16): 3584.00 MiB, V (f16): 3584.00 MiB llama_new_context_with_model: CPU output buffer size = 0.15 MiB llama_new_context_with_model: Metal compute buffer size = 3184.00 MiB llama_new_context_with_model: CPU compute buffer size = 3184.01 MiB llama_new_context_with_model: graph nodes = 1798 llama_new_context_with_model: graph splits = 483 ^Ctime=2024-10-02T17:00:05.986+02:00 level=WARN source=server.go:594 msg="client connection closed before server finished loading, aborting load" time=2024-10-02T17:00:05.987+02:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="timed out waiting for llama runner to start: context canceled" [GIN] 2024/10/02 - 17:00:05 | 499 | 19.265245125s | 127.0.0.1 | POST "/api/generate" </details> ### OS macOS ### GPU Apple ### CPU Apple ### Ollama version 0.3.12
GiteaMirror added the bug label 2026-04-22 09:47:05 -05:00
GiteaMirror commented 2026-04-22 09:47:06 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Oct 2, 2024):

Size difference is likely due to differences in context window. When you run ollama run you will be using the default of 2048.

time=2024-10-02T16:59:46.755+02:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=0 library=metal total="10.7 GiB" available="7.1 GiB" time=2024-10-02T16:59:46.790+02:00 level=INFO source=server.go:103 msg="system memory" total="16.0 GiB" free="11.5 GiB" free_swap="0 B"
time=2024-10-02T16:59:46.791+02:00 level=INFO source=memory.go:326 msg="offload to metal" layers.requested=-1 layers.model=57 layers.offload=26 layers.split="" memory.available="[10.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="18.5 GiB" memory.required.partial="10.6 GiB" memory.required.kv="7.0 GiB" memory.required.allocations="[10.6 GiB]" memory.weights.total="14.5 GiB" memory.weights.repeating="14.3 GiB" memory.weights.nonrepeating="157.5 MiB" memory.graph.full="3.1 GiB" memory.graph.partial="3.1 GiB"
time=2024-10-02T16:59:46.791+02:00 level=INFO source=server.go:388 msg="starting llama server" cmd="/var/folders/mb/gh2_kwk50611j_9p8_0pmkn00000gn/T/ollama151879657/runners/metal/ollama_llama_server --model /Users/big/.ollama/models/blobs/sha256-a645a2a1d407b876edf4731dd223cf8a09fa168efc96b20496a89bcdf702f7b4 --ctx-size 32768 --batch-size 512 --embedding --log-disable --n-gpu-layers 26 --no-mmap --parallel 1 --port 60468"

Here the model has been loaded with a context window of 32768, and KV space is 7G. Continue is presumably sending "options":{"num_ctx":32768} which is causing your RAM blowout.

<!-- gh-comment-id:2389914198 --> @rick-github commented on GitHub (Oct 2, 2024): Size difference is likely due to differences in context window. When you run `ollama run` you will be using the default of 2048. ``` time=2024-10-02T16:59:46.755+02:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=0 library=metal total="10.7 GiB" available="7.1 GiB" time=2024-10-02T16:59:46.790+02:00 level=INFO source=server.go:103 msg="system memory" total="16.0 GiB" free="11.5 GiB" free_swap="0 B" time=2024-10-02T16:59:46.791+02:00 level=INFO source=memory.go:326 msg="offload to metal" layers.requested=-1 layers.model=57 layers.offload=26 layers.split="" memory.available="[10.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="18.5 GiB" memory.required.partial="10.6 GiB" memory.required.kv="7.0 GiB" memory.required.allocations="[10.6 GiB]" memory.weights.total="14.5 GiB" memory.weights.repeating="14.3 GiB" memory.weights.nonrepeating="157.5 MiB" memory.graph.full="3.1 GiB" memory.graph.partial="3.1 GiB" time=2024-10-02T16:59:46.791+02:00 level=INFO source=server.go:388 msg="starting llama server" cmd="/var/folders/mb/gh2_kwk50611j_9p8_0pmkn00000gn/T/ollama151879657/runners/metal/ollama_llama_server --model /Users/big/.ollama/models/blobs/sha256-a645a2a1d407b876edf4731dd223cf8a09fa168efc96b20496a89bcdf702f7b4 --ctx-size 32768 --batch-size 512 --embedding --log-disable --n-gpu-layers 26 --no-mmap --parallel 1 --port 60468" ``` Here the model has been loaded with a context window of 32768, and KV space is 7G. Continue is presumably sending `"options":{"num_ctx":32768}` which is causing your RAM blowout.
GiteaMirror commented 2026-04-22 09:47:07 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Oct 3, 2024):

@bigxalx rick is correct here. You're just blowing out your RAM w/ the context size. I haven't used continue, but I think you can change the context length in the config. I found these docs which maybe can help?

I'll go ahead and close the issue.

<!-- gh-comment-id:2392199993 --> @pdevine commented on GitHub (Oct 3, 2024): @bigxalx rick is correct here. You're just blowing out your RAM w/ the context size. I haven't used continue, but I think you can change the context length in the config. I found these [docs](https://docs.continue.dev/customize/config) which maybe can help? I'll go ahead and close the issue.
GiteaMirror commented 2026-04-22 09:47:07 -05:00
Author
Owner
Copy Link

@bigxalx commented on GitHub (Oct 4, 2024):

You were both absolutely right!
For anyone having the same problem, this is the fix for continue.dev
in config.json you can add a contextLength parameter:
For example:

{
  "models": [
    {
      "title": "Codestral",
      "provider": "ollama",
      "model": "codestral:22b-v0.1-q2_K",
      "contextLength": 2048
    }
  ],
  ...
}
<!-- gh-comment-id:2393304526 --> @bigxalx commented on GitHub (Oct 4, 2024): You were both absolutely right! For anyone having the same problem, this is the fix for continue.dev in `config.json` you can add a `contextLength` parameter: For example: <pre> { "models": [ { "title": "Codestral", "provider": "ollama", "model": "codestral:22b-v0.1-q2_K", <strong><mark>"contextLength": 2048</mark></strong> } ], ... } </pre>
GiteaMirror commented 2026-04-22 09:47:07 -05:00
Author
Owner
Copy Link

@chakka-guna-sekhar-venkata-chennaiah commented on GitHub (Dec 1, 2025):

is there any way for dynamic controllisation available on the application level?

<!-- gh-comment-id:3594483275 --> @chakka-guna-sekhar-venkata-chennaiah commented on GitHub (Dec 1, 2025): is there any way for dynamic controllisation available on the application level?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#30257
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?