The issue of context length limitations in Ollama #1650

New Issue

GiteaMirror · 2025-11-11T14:49:22-06:00

GiteaMirror commented

2025-11-11 14:49:22 -06:00

Originally created by @anrgct on GitHub (Jul 31, 2024).

Bug Report

Description

I raised this issue in the Ollama repository. Please take a look at this issue, https://github.com/ollama/ollama/issues/6026#issuecomment-2254547480, where the author provided two solutions:

Specify num_ctx in the modelfile
Pass options during the request

I think method 1 has a drawback: when I set a large num_ctx in modelfile, it will occupy a lot of GPU memory even if the model is not in use. Perhaps method 2 is better? We could increase num_ctx during the request, controlling it based on the length of the messages. Alternatively, we could give users a prompt, saying something like "Your request exceeds the context length and has been truncated for processing."

Originally created by @anrgct on GitHub (Jul 31, 2024). # Bug Report ## Description I raised this issue in the Ollama repository. Please take a look at this issue, https://github.com/ollama/ollama/issues/6026#issuecomment-2254547480, where the author provided two solutions: 1. Specify num_ctx in the modelfile 2. Pass options during the request I think method 1 has a drawback: when I set a large num_ctx in modelfile, it will occupy a lot of GPU memory even if the model is not in use. Perhaps method 2 is better? We could increase num_ctx during the request, controlling it based on the length of the messages. Alternatively, we could give users a prompt, saying something like "Your request exceeds the context length and has been truncated for processing."

GiteaMirror closed this issue

2025-11-11 14:49:23 -06:00

GiteaMirror commented

2025-11-11 14:49:24 -06:00

@justinh-rahb commented on GitHub (Jul 31, 2024):

The issue with changing num_ctx is that it requires a reload of the model each time it's changed. This can have an even more negative effect on perceived user experience than it would seek to solve.

@justinh-rahb commented on GitHub (Jul 31, 2024): The issue with changing `num_ctx` is that it requires a reload of the model each time it's changed. This can have an even more negative effect on perceived user experience than it would seek to solve.