github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #5661] num_ctx parameter does not work on Linux #3531

New Issue
Closed
opened 2026-04-12 14:14:05 -05:00 by GiteaMirror · 6 comments
GiteaMirror commented 2026-04-12 14:14:05 -05:00
Owner
Copy Link

Originally created by @ronchengang on GitHub (Jul 13, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/5661

What is the issue?

Issue: Although the log shows the n_ctx = 102400, the input prompt still truncates to 2048.
Ollama version: 0.2.1
OS: AWS Linux, instance type: g5.xlarge,
GPU: Nvidia A10 24G GPU, version 12.x
Model: Qwen2-7B-Instruct, GGUF V3

Ollama server log:

  Device 0: NVIDIA A10G, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size =    0.30 MiB
llm_load_tensors: offloading 28 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 29/29 layers to GPU
llm_load_tensors:  CUDA_Host buffer size =   292.36 MiB
llm_load_tensors:      CUDA0 buffer size =  3928.07 MiB
**llama_new_context_with_model: n_ctx      = 102400**
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 1
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1

above log says the n_ctx=102400, but I still get below input truncate warning and the n_ctx's value is 2048.

LLM request log:

[GIN] 2024/07/13 - 02:27:54 | 200 | 17.512894039s |       127.0.0.1 | POST     "/api/chat"
INFO [update_slots] input truncated | **n_ctx=2048** n_erase=1440 n_keep=4 n_left=2044 n_shift=1022 tid="140646121488384" timestamp=1720837794
INFO [update_slots] input truncated | **n_ctx=2048** n_erase=1432 n_keep=4 n_left=2044 n_shift=1022 tid="140646121488384" timestamp=1720837794
INFO [update_slots] input truncated | **n_ctx=2048** n_erase=1441 n_keep=4 n_left=2044 n_shift=1022 tid="140646121488384" timestamp=1720837795

this same model+same ollama version run well on my Mac, but when I move it to AWS Linux, the error occurs.

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.2.1

Originally created by @ronchengang on GitHub (Jul 13, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/5661 ### What is the issue? Issue: Although the log shows the n_ctx = 102400, the input prompt still truncates to 2048. Ollama version: 0.2.1 OS: AWS Linux, instance type: g5.xlarge, GPU: Nvidia A10 24G GPU, version 12.x Model: Qwen2-7B-Instruct, GGUF V3 Ollama server log: ``` Device 0: NVIDIA A10G, compute capability 8.6, VMM: yes llm_load_tensors: ggml ctx size = 0.30 MiB llm_load_tensors: offloading 28 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 29/29 layers to GPU llm_load_tensors: CUDA_Host buffer size = 292.36 MiB llm_load_tensors: CUDA0 buffer size = 3928.07 MiB **llama_new_context_with_model: n_ctx = 102400** llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 1 llama_new_context_with_model: freq_base = 1000000.0 llama_new_context_with_model: freq_scale = 1 ``` above log says the n_ctx=102400, but I still get below input truncate warning and the n_ctx's value is 2048. LLM request log: ``` [GIN] 2024/07/13 - 02:27:54 | 200 | 17.512894039s | 127.0.0.1 | POST "/api/chat" INFO [update_slots] input truncated | **n_ctx=2048** n_erase=1440 n_keep=4 n_left=2044 n_shift=1022 tid="140646121488384" timestamp=1720837794 INFO [update_slots] input truncated | **n_ctx=2048** n_erase=1432 n_keep=4 n_left=2044 n_shift=1022 tid="140646121488384" timestamp=1720837794 INFO [update_slots] input truncated | **n_ctx=2048** n_erase=1441 n_keep=4 n_left=2044 n_shift=1022 tid="140646121488384" timestamp=1720837795 ``` this same model+same ollama version run well on my Mac, but when I move it to AWS Linux, the error occurs. ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.2.1
GiteaMirror added the bug label 2026-04-12 14:14:05 -05:00
GiteaMirror commented 2026-04-12 14:14:07 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Jul 30, 2024):

Hi there, do you have OLLAMA_NUM_PARALLEL set?

<!-- gh-comment-id:2257239943 --> @jmorganca commented on GitHub (Jul 30, 2024): Hi there, do you have `OLLAMA_NUM_PARALLEL` set?
GiteaMirror commented 2026-04-12 14:14:08 -05:00
Author
Owner
Copy Link

@ronchengang commented on GitHub (Jul 30, 2024):

Yes, here is my script

export OLLAMA_HOST=0.0.0.0 
export OLLAMA_KEEP_ALIVE=600m
export OLLAMA_MAX_LOADED_MODELS=5
export OLLAMA_NUM_PARALLEL=50
export GIN_MODE=release
export OLLAMA_MAX_QUEUE=1024
export OLLAMA_ORIGINS="*"
ollama serve
<!-- gh-comment-id:2257296786 --> @ronchengang commented on GitHub (Jul 30, 2024): Yes, here is my script ``` export OLLAMA_HOST=0.0.0.0 export OLLAMA_KEEP_ALIVE=600m export OLLAMA_MAX_LOADED_MODELS=5 export OLLAMA_NUM_PARALLEL=50 export GIN_MODE=release export OLLAMA_MAX_QUEUE=1024 export OLLAMA_ORIGINS="*" ollama serve ```
GiteaMirror commented 2026-04-12 14:14:09 -05:00
Author
Owner
Copy Link

@zifengdexiatian commented on GitHub (Jul 30, 2024):

I have the same problem

<!-- gh-comment-id:2258197671 --> @zifengdexiatian commented on GitHub (Jul 30, 2024): I have the same problem
GiteaMirror commented 2026-04-12 14:14:11 -05:00
Author
Owner
Copy Link

@haseeburrehmanfaheem commented on GitHub (Oct 16, 2024):

Did anyone find a solution to this?

<!-- gh-comment-id:2415795149 --> @haseeburrehmanfaheem commented on GitHub (Oct 16, 2024): Did anyone find a solution to this?
GiteaMirror commented 2026-04-12 14:14:13 -05:00
Author
Owner
Copy Link

@zifengdexiatian commented on GitHub (Oct 16, 2024):

Did anyone find a solution to this?

https://github.com/ollama/ollama/issues/5356#issuecomment-2351795006

<!-- gh-comment-id:2415815943 --> @zifengdexiatian commented on GitHub (Oct 16, 2024): > Did anyone find a solution to this? https://github.com/ollama/ollama/issues/5356#issuecomment-2351795006
GiteaMirror commented 2026-04-12 14:14:13 -05:00
Author
Owner
Copy Link

@rodolfo-costa-vial commented on GitHub (Mar 18, 2025):

OLLAMA_CONTEXT_LENGTH:32768

<!-- gh-comment-id:2731270443 --> @rodolfo-costa-vial commented on GitHub (Mar 18, 2025): OLLAMA_CONTEXT_LENGTH:32768
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#3531
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?