github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 21:25:46 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #6817] llama 3.1 8b params downloaded from huggingface, strange num_ctx behavior #30066

New Issue
Closed
opened 2026-04-22 09:30:20 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-04-22 09:30:20 -05:00
Owner
Copy Link

Originally created by @akseg73 on GitHub (Sep 15, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6817

What is the issue?

I downloaded llama3.1 8b quantized to 8 bits from huggingface. It appears to have a default context size of 132k. Looking at numerous sources on the internet it seemed reasonable that in order to utilize the model i should reduce the context size with Parameter num_ctx 32k.

However when i utilize num_ctx to reduce the context size to 32k from 132k, ollama generates a much larger model. 15GB instead of 10GB for the default parameters. What could be wrong here. Reducing the context size should have perhaps reduced the size of the model generated. Is there something wrong that i have done?

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

No response

Originally created by @akseg73 on GitHub (Sep 15, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6817 ### What is the issue? I downloaded llama3.1 8b quantized to 8 bits from huggingface. It appears to have a default context size of 132k. Looking at numerous sources on the internet it seemed reasonable that in order to utilize the model i should reduce the context size with Parameter num_ctx 32k. However when i utilize num_ctx to reduce the context size to 32k from 132k, ollama generates a much larger model. 15GB instead of 10GB for the default parameters. What could be wrong here. Reducing the context size should have perhaps reduced the size of the model generated. Is there something wrong that i have done? ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version _No response_
GiteaMirror added the bug label 2026-04-22 09:30:20 -05:00
GiteaMirror commented 2026-04-22 09:30:25 -05:00
Author
Owner
Copy Link

@akseg73 commented on GitHub (Sep 15, 2024):

As an alternative is there some way to:
-- periodically refresh the context to start over
-- or specify num_ctx at run time to ollama, will that also change the size of the model?

<!-- gh-comment-id:2351832944 --> @akseg73 commented on GitHub (Sep 15, 2024): As an alternative is there some way to: -- periodically refresh the context to start over -- or specify num_ctx at run time to ollama, will that also change the size of the model?
GiteaMirror commented 2026-04-22 09:30:26 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 16, 2024):

The context size displayed in a model card or found in the KV parameters of a model file is the maximum size that the model is trained to handle. The num_ctx field in a Modelfile or API call is the size of the context window that ollama allocates for handling requests to the model. The default num_ctx that ollama uses (ie, when not overridden by Modelfile or API) is 2048. When you set the context window to 32k, you are increasing it from the default of 2k, which is why the model requires more (V)RAM.

<!-- gh-comment-id:2351854962 --> @rick-github commented on GitHub (Sep 16, 2024): The context size displayed in a model card or found in the KV parameters of a model file is the maximum size that the model is trained to handle. The `num_ctx` field in a Modelfile or API call is the size of the context window that ollama allocates for handling requests to the model. The default `num_ctx` that ollama uses (ie, when not overridden by Modelfile or API) is 2048. When you set the context window to 32k, you are increasing it from the default of 2k, which is why the model requires more (V)RAM.
GiteaMirror commented 2026-04-22 09:30:27 -05:00
Author
Owner
Copy Link

@akseg73 commented on GitHub (Sep 16, 2024):

Thanks for the response. It seems that utilizing a context size of 2k is probably too limiting to the capabilities of the model. Perhaps we to experiment a little bit to see if 4k, 8k, 16k, 32k will generate a size that will fit on the GPU VRAM and we probably have to reduce the quantization to 4 bits instead of 8 bits.

<!-- gh-comment-id:2353216187 --> @akseg73 commented on GitHub (Sep 16, 2024): Thanks for the response. It seems that utilizing a context size of 2k is probably too limiting to the capabilities of the model. Perhaps we to experiment a little bit to see if 4k, 8k, 16k, 32k will generate a size that will fit on the GPU VRAM and we probably have to reduce the quantization to 4 bits instead of 8 bits.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#30066
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?