github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 20:54:54 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #7773] Ollama unloads model after ~30 seconds per message with api/chat despite setting keep_alive=-1 and setting OLLAMA_KEEP_ALIVE=-1 in Windows environment variables. #51477

New Issue
Closed
opened 2026-04-28 20:19:14 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-04-28 20:19:14 -05:00
Owner
Copy Link

Originally created by @SingularityMan on GitHub (Nov 21, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/7773

What is the issue?

Basically the title. I'm trying to stream responses from a single model periodically using this method:

async def generate_text_stream(
    self,
    messages: list,
    agent_messages: list,
    system_prompt: str,
    user_input: str,
    context_length: int = 32000,
    temperature: float = 0.7,
    top_p: float = 0.3,
    top_k: int = 10000
    ) -> Tuple[List, List, AsyncGenerator[str, None]]:
        """
        Generates a text response as a stream using ollama.chat.
        Returns an asynchronous generator yielding sentences.
        """
        messages[0] = {"role": "system", "content": system_prompt}
        messages.append({"role": "user", "content": user_input})

        async def fetch_stream():
            loop = asyncio.get_event_loop()
            buffer = ''
            complete_response = ''  # To hold the entire concatenated output

            def run_chat():
                return ollama.chat(
                    model=self.language_model,
                    messages=messages,
                    stream=True,
                    keep_alive=-1,
                    options={
                        "repeat_penalty": 1.15,
                        "temperature": temperature,
                        "top_p": top_p,
                        "top_k": top_k,
                        "num_ctx": context_length
                    }
                )

            # Run ollama.chat in the executor to prevent blocking the event loop
            stream = await loop.run_in_executor(None, run_chat)

            for chunk in stream:
                content = chunk.get('message', {}).get('content', '')
                if content:
                    buffer += content
                    sentences, buffer = split_buffer_into_sentences(buffer)
                    for sentence in sentences:
                        # Clean up the sentence
                        sentence = clean_text(sentence)
                        complete_response += sentence + ' '  # Concatenate to the complete response
                        yield sentence

            # Handle any remaining buffer
            if buffer.strip():
                buffer = clean_text(buffer)
                complete_response += buffer + ' '  # Add remaining buffer to the complete response

            # Append the complete response as a single message
            complete_response = complete_response.strip()
            messages.append({"role": "assistant", "content": complete_response})
            agent_messages.append(
                f"Agent Name:{self.agent_name}, ({self.agent_gender})\nAgent Response: {complete_response}"
            )

        return messages, agent_messages, fetch_stream()

This function streams the chat response and returns two updated lists and an AsyncGenerator() object in the form of fetch_stream(). This is done sequentially between two agents running the same model in Ollama. The problem I noticed is that if I don't send a message within approximately 30 seconds the server automatically unloads the model and then reloads it no matter which value I set keep_alive to.

I also have 48GB VRAM and this model, combined with other, smaller AI models outside of Ollama, collectively use up only 36GB VRAM, even on considerably higher context amounts the VRAM still leaves room for use. I don't really know what I'm doing wrong here but here is the server log:

time=2024-11-20T21:26:14.775-05:00 level=DEBUG source=sched.go:407 msg="context for request finished"
time=2024-11-20T21:26:14.779-05:00 level=DEBUG source=sched.go:339 msg="runner with non-zero duration has gone idle, adding timer" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc duration=2562047h47m16.854775807s
time=2024-11-20T21:26:14.779-05:00 level=DEBUG source=sched.go:357 msg="after processing request finished event" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc refCount=0
time=2024-11-20T21:26:53.535-05:00 level=DEBUG source=sched.go:575 msg="evaluating already loaded" model=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc
time=2024-11-20T21:26:53.535-05:00 level=DEBUG source=sched.go:283 msg="resetting model to expire immediately to make room" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc refCount=0
time=2024-11-20T21:26:53.540-05:00 level=DEBUG source=sched.go:296 msg="waiting for pending requests to complete and unload to occur" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc
time=2024-11-20T21:26:53.540-05:00 level=DEBUG source=sched.go:360 msg="runner expired event received" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc
time=2024-11-20T21:26:53.542-05:00 level=DEBUG source=sched.go:375 msg="got lock to unload" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc
time=2024-11-20T21:26:53.542-05:00 level=DEBUG source=gpu.go:398 msg="updating system memory data" before.total="127.9 GiB" before.free="114.1 GiB" before.free_swap="110.3 GiB" now.total="127.9 GiB" now.free="112.4 GiB" now.free_swap="87.4 GiB"
time=2024-11-20T21:26:53.563-05:00 level=DEBUG source=gpu.go:448 msg="updating cuda memory data" gpu=GPU-86c1a0d8-a857-7035-6d12-957836f9d5d6 name="Quadro RTX 8000" overhead="104.8 MiB" before.total="48.0 GiB" before.free="30.9 GiB" now.total="48.0 GiB" now.free="9.7 GiB" now.used="38.2 GiB"
releasing nvml library
time=2024-11-20T21:26:53.584-05:00 level=DEBUG source=server.go:1068 msg="stopping llama server"
time=2024-11-20T21:26:53.585-05:00 level=DEBUG source=server.go:1074 msg="waiting for llama server to exit"
time=2024-11-20T21:26:53.826-05:00 level=DEBUG source=gpu.go:398 msg="updating system memory data" before.total="127.9 GiB" before.free="112.4 GiB" before.free_swap="87.4 GiB" now.total="127.9 GiB" now.free="113.8 GiB" now.free_swap="108.7 GiB"
time=2024-11-20T21:26:53.841-05:00 level=DEBUG source=gpu.go:448 msg="updating cuda memory data" gpu=GPU-86c1a0d8-a857-7035-6d12-957836f9d5d6 name="Quadro RTX 8000" overhead="104.8 MiB" before.total="48.0 GiB" before.free="9.7 GiB" now.total="48.0 GiB" now.free="30.9 GiB" now.used="17.0 GiB"
releasing nvml library
time=2024-11-20T21:26:53.842-05:00 level=DEBUG source=sched.go:659 msg="gpu VRAM free memory converged after 0.30 seconds" model=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc
time=2024-11-20T21:26:53.845-05:00 level=DEBUG source=server.go:1078 msg="llama server stopped"
time=2024-11-20T21:26:53.845-05:00 level=DEBUG source=sched.go:380 msg="runner released" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc
time=2024-11-20T21:26:53.845-05:00 level=DEBUG source=sched.go:384 msg="sending an unloaded event" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc
time=2024-11-20T21:26:53.846-05:00 level=DEBUG source=sched.go:302 msg="unload completed" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc
time=2024-11-20T21:26:53.846-05:00 level=DEBUG source=gpu.go:398 msg="updating system memory data" before.total="127.9 GiB" before.free="113.8 GiB" before.free_swap="108.7 GiB" now.total="127.9 GiB" now.free="114.1 GiB" now.free_swap="110.3 GiB"
time=2024-11-20T21:26:53.856-05:00 level=DEBUG source=gpu.go:448 msg="updating cuda memory data" gpu=GPU-86c1a0d8-a857-7035-6d12-957836f9d5d6 name="Quadro RTX 8000" overhead="104.8 MiB" before.total="48.0 GiB" before.free="30.9 GiB" now.total="48.0 GiB" now.free="30.9 GiB" now.used="17.0 GiB"
releasing nvml library
time=2024-11-20T21:26:53.900-05:00 level=DEBUG source=sched.go:224 msg="loading first model" model=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc
time=2024-11-20T21:26:53.900-05:00 level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[30.9 GiB]"
time=2024-11-20T21:26:53.902-05:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc gpu=GPU-86c1a0d8-a857-7035-6d12-957836f9d5d6 parallel=8 available=33200353280 required="29.1 GiB"
time=2024-11-20T21:26:53.902-05:00 level=DEBUG source=gpu.go:398 msg="updating system memory data" before.total="127.9 GiB" before.free="114.1 GiB" before.free_swap="110.3 GiB" now.total="127.9 GiB" now.free="114.1 GiB" now.free_swap="110.3 GiB"
time=2024-11-20T21:26:53.918-05:00 level=DEBUG source=gpu.go:448 msg="updating cuda memory data" gpu=GPU-86c1a0d8-a857-7035-6d12-957836f9d5d6 name="Quadro RTX 8000" overhead="104.8 MiB" before.total="48.0 GiB" before.free="30.9 GiB" now.total="48.0 GiB" now.free="30.9 GiB" now.used="17.0 GiB"
releasing nvml library
time=2024-11-20T21:26:53.919-05:00 level=INFO source=server.go:105 msg="system memory" total="127.9 GiB" free="114.1 GiB" free_swap="110.3 GiB"
time=2024-11-20T21:26:53.920-05:00 level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[30.9 GiB]"
time=2024-11-20T21:26:53.920-05:00 level=INFO source=memory.go:343 msg="offload to cuda" layers.requested=-1 layers.model=47 layers.offload=47 layers.split="" memory.available="[30.9 GiB]" memory.gpu_overhead="476.8 MiB" memory.required.full="29.1 GiB" memory.required.partial="29.1 GiB" memory.required.kv="11.5 GiB" memory.required.allocations="[29.1 GiB]" memory.weights.total="25.1 GiB" memory.weights.repeating="24.2 GiB" memory.weights.nonrepeating="922.9 MiB" memory.graph.full="2.1 GiB" memory.graph.partial="2.2 GiB"

Based on this I don't know if the issue lies in actually timing out or if Ollama is auto-unloading the model due to resource usage and if its the latter than what can I do about it? I have plenty of VRAM leftover but Ollama keeps unloading the model for whatever reason.

I also ran ollama ps and it returned this:

NAME                        ID              SIZE     PROCESSOR    UNTIL
gemma2:27b-instruct-q4_0    53261bc9c192    31 GB    100% GPU     Forever

OS

Windows

GPU

Nvidia

CPU

Intel

Ollama version

0.4.2

Originally created by @SingularityMan on GitHub (Nov 21, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/7773 ### What is the issue? Basically the title. I'm trying to stream responses from a single model periodically using this method: ``` async def generate_text_stream( self, messages: list, agent_messages: list, system_prompt: str, user_input: str, context_length: int = 32000, temperature: float = 0.7, top_p: float = 0.3, top_k: int = 10000 ) -> Tuple[List, List, AsyncGenerator[str, None]]: """ Generates a text response as a stream using ollama.chat. Returns an asynchronous generator yielding sentences. """ messages[0] = {"role": "system", "content": system_prompt} messages.append({"role": "user", "content": user_input}) async def fetch_stream(): loop = asyncio.get_event_loop() buffer = '' complete_response = '' # To hold the entire concatenated output def run_chat(): return ollama.chat( model=self.language_model, messages=messages, stream=True, keep_alive=-1, options={ "repeat_penalty": 1.15, "temperature": temperature, "top_p": top_p, "top_k": top_k, "num_ctx": context_length } ) # Run ollama.chat in the executor to prevent blocking the event loop stream = await loop.run_in_executor(None, run_chat) for chunk in stream: content = chunk.get('message', {}).get('content', '') if content: buffer += content sentences, buffer = split_buffer_into_sentences(buffer) for sentence in sentences: # Clean up the sentence sentence = clean_text(sentence) complete_response += sentence + ' ' # Concatenate to the complete response yield sentence # Handle any remaining buffer if buffer.strip(): buffer = clean_text(buffer) complete_response += buffer + ' ' # Add remaining buffer to the complete response # Append the complete response as a single message complete_response = complete_response.strip() messages.append({"role": "assistant", "content": complete_response}) agent_messages.append( f"Agent Name:{self.agent_name}, ({self.agent_gender})\nAgent Response: {complete_response}" ) return messages, agent_messages, fetch_stream() ``` This function streams the chat response and returns two updated lists and an `AsyncGenerator()` object in the form of `fetch_stream()`. This is done sequentially between two agents running the same model in Ollama. The problem I noticed is that if I don't send a message within approximately 30 seconds the server automatically unloads the model and then reloads it no matter which value I set keep_alive to. I also have 48GB VRAM and this model, combined with other, smaller AI models outside of Ollama, collectively use up only 36GB VRAM, even on considerably higher context amounts the VRAM still leaves room for use. I don't really know what I'm doing wrong here but here is the server log: ``` time=2024-11-20T21:26:14.775-05:00 level=DEBUG source=sched.go:407 msg="context for request finished" time=2024-11-20T21:26:14.779-05:00 level=DEBUG source=sched.go:339 msg="runner with non-zero duration has gone idle, adding timer" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc duration=2562047h47m16.854775807s time=2024-11-20T21:26:14.779-05:00 level=DEBUG source=sched.go:357 msg="after processing request finished event" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc refCount=0 time=2024-11-20T21:26:53.535-05:00 level=DEBUG source=sched.go:575 msg="evaluating already loaded" model=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc time=2024-11-20T21:26:53.535-05:00 level=DEBUG source=sched.go:283 msg="resetting model to expire immediately to make room" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc refCount=0 time=2024-11-20T21:26:53.540-05:00 level=DEBUG source=sched.go:296 msg="waiting for pending requests to complete and unload to occur" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc time=2024-11-20T21:26:53.540-05:00 level=DEBUG source=sched.go:360 msg="runner expired event received" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc time=2024-11-20T21:26:53.542-05:00 level=DEBUG source=sched.go:375 msg="got lock to unload" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc time=2024-11-20T21:26:53.542-05:00 level=DEBUG source=gpu.go:398 msg="updating system memory data" before.total="127.9 GiB" before.free="114.1 GiB" before.free_swap="110.3 GiB" now.total="127.9 GiB" now.free="112.4 GiB" now.free_swap="87.4 GiB" time=2024-11-20T21:26:53.563-05:00 level=DEBUG source=gpu.go:448 msg="updating cuda memory data" gpu=GPU-86c1a0d8-a857-7035-6d12-957836f9d5d6 name="Quadro RTX 8000" overhead="104.8 MiB" before.total="48.0 GiB" before.free="30.9 GiB" now.total="48.0 GiB" now.free="9.7 GiB" now.used="38.2 GiB" releasing nvml library time=2024-11-20T21:26:53.584-05:00 level=DEBUG source=server.go:1068 msg="stopping llama server" time=2024-11-20T21:26:53.585-05:00 level=DEBUG source=server.go:1074 msg="waiting for llama server to exit" time=2024-11-20T21:26:53.826-05:00 level=DEBUG source=gpu.go:398 msg="updating system memory data" before.total="127.9 GiB" before.free="112.4 GiB" before.free_swap="87.4 GiB" now.total="127.9 GiB" now.free="113.8 GiB" now.free_swap="108.7 GiB" time=2024-11-20T21:26:53.841-05:00 level=DEBUG source=gpu.go:448 msg="updating cuda memory data" gpu=GPU-86c1a0d8-a857-7035-6d12-957836f9d5d6 name="Quadro RTX 8000" overhead="104.8 MiB" before.total="48.0 GiB" before.free="9.7 GiB" now.total="48.0 GiB" now.free="30.9 GiB" now.used="17.0 GiB" releasing nvml library time=2024-11-20T21:26:53.842-05:00 level=DEBUG source=sched.go:659 msg="gpu VRAM free memory converged after 0.30 seconds" model=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc time=2024-11-20T21:26:53.845-05:00 level=DEBUG source=server.go:1078 msg="llama server stopped" time=2024-11-20T21:26:53.845-05:00 level=DEBUG source=sched.go:380 msg="runner released" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc time=2024-11-20T21:26:53.845-05:00 level=DEBUG source=sched.go:384 msg="sending an unloaded event" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc time=2024-11-20T21:26:53.846-05:00 level=DEBUG source=sched.go:302 msg="unload completed" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc time=2024-11-20T21:26:53.846-05:00 level=DEBUG source=gpu.go:398 msg="updating system memory data" before.total="127.9 GiB" before.free="113.8 GiB" before.free_swap="108.7 GiB" now.total="127.9 GiB" now.free="114.1 GiB" now.free_swap="110.3 GiB" time=2024-11-20T21:26:53.856-05:00 level=DEBUG source=gpu.go:448 msg="updating cuda memory data" gpu=GPU-86c1a0d8-a857-7035-6d12-957836f9d5d6 name="Quadro RTX 8000" overhead="104.8 MiB" before.total="48.0 GiB" before.free="30.9 GiB" now.total="48.0 GiB" now.free="30.9 GiB" now.used="17.0 GiB" releasing nvml library time=2024-11-20T21:26:53.900-05:00 level=DEBUG source=sched.go:224 msg="loading first model" model=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc time=2024-11-20T21:26:53.900-05:00 level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[30.9 GiB]" time=2024-11-20T21:26:53.902-05:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc gpu=GPU-86c1a0d8-a857-7035-6d12-957836f9d5d6 parallel=8 available=33200353280 required="29.1 GiB" time=2024-11-20T21:26:53.902-05:00 level=DEBUG source=gpu.go:398 msg="updating system memory data" before.total="127.9 GiB" before.free="114.1 GiB" before.free_swap="110.3 GiB" now.total="127.9 GiB" now.free="114.1 GiB" now.free_swap="110.3 GiB" time=2024-11-20T21:26:53.918-05:00 level=DEBUG source=gpu.go:448 msg="updating cuda memory data" gpu=GPU-86c1a0d8-a857-7035-6d12-957836f9d5d6 name="Quadro RTX 8000" overhead="104.8 MiB" before.total="48.0 GiB" before.free="30.9 GiB" now.total="48.0 GiB" now.free="30.9 GiB" now.used="17.0 GiB" releasing nvml library time=2024-11-20T21:26:53.919-05:00 level=INFO source=server.go:105 msg="system memory" total="127.9 GiB" free="114.1 GiB" free_swap="110.3 GiB" time=2024-11-20T21:26:53.920-05:00 level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[30.9 GiB]" time=2024-11-20T21:26:53.920-05:00 level=INFO source=memory.go:343 msg="offload to cuda" layers.requested=-1 layers.model=47 layers.offload=47 layers.split="" memory.available="[30.9 GiB]" memory.gpu_overhead="476.8 MiB" memory.required.full="29.1 GiB" memory.required.partial="29.1 GiB" memory.required.kv="11.5 GiB" memory.required.allocations="[29.1 GiB]" memory.weights.total="25.1 GiB" memory.weights.repeating="24.2 GiB" memory.weights.nonrepeating="922.9 MiB" memory.graph.full="2.1 GiB" memory.graph.partial="2.2 GiB" ``` Based on this I don't know if the issue lies in actually timing out or if Ollama is auto-unloading the model due to resource usage and if its the latter than what can I do about it? I have plenty of VRAM leftover but Ollama keeps unloading the model for whatever reason. I also ran `ollama ps` and it returned this: ``` NAME ID SIZE PROCESSOR UNTIL gemma2:27b-instruct-q4_0 53261bc9c192 31 GB 100% GPU Forever ``` ### OS Windows ### GPU Nvidia ### CPU Intel ### Ollama version 0.4.2
GiteaMirror added the bug label 2026-04-28 20:19:14 -05:00
GiteaMirror commented 2026-04-28 20:19:19 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Nov 21, 2024):

time=2024-11-20T21:26:53.535-05:00 level=DEBUG source=sched.go:575 msg="evaluating already loaded" model=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc
time=2024-11-20T21:26:53.535-05:00 level=DEBUG source=sched.go:283 msg="resetting model to expire immediately to make room" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc refCount=0

ollama received a request that changed the parameters of the model, probably a different num_ctx.

<!-- gh-comment-id:2490387608 --> @rick-github commented on GitHub (Nov 21, 2024): ``` time=2024-11-20T21:26:53.535-05:00 level=DEBUG source=sched.go:575 msg="evaluating already loaded" model=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc time=2024-11-20T21:26:53.535-05:00 level=DEBUG source=sched.go:283 msg="resetting model to expire immediately to make room" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc refCount=0 ``` ollama received a request that changed the parameters of the model, probably a different `num_ctx`.
GiteaMirror commented 2026-04-28 20:19:20 -05:00
Author
Owner
Copy Link

@SingularityMan commented on GitHub (Nov 21, 2024):

time=2024-11-20T21:26:53.535-05:00 level=DEBUG source=sched.go:575 msg="evaluating already loaded" model=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc
time=2024-11-20T21:26:53.535-05:00 level=DEBUG source=sched.go:283 msg="resetting model to expire immediately to make room" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc refCount=0

ollama received a request that changed the parameters of the model, probably a different num_ctx.

Wait a minute, this might actually be it! In the original script, the context_length for the API call would change between two constants based on certain conditions met. So you're telling me I can't just change the context_length to whatever I want on the fly, I have to reload the model entirely?

<!-- gh-comment-id:2490767817 --> @SingularityMan commented on GitHub (Nov 21, 2024): > ``` > time=2024-11-20T21:26:53.535-05:00 level=DEBUG source=sched.go:575 msg="evaluating already loaded" model=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc > time=2024-11-20T21:26:53.535-05:00 level=DEBUG source=sched.go:283 msg="resetting model to expire immediately to make room" modelPath=C:\Users\me\.ollama\models\blobs\sha256-d7e4b00a7d7a8d03d4eed9b0f3f61a427e9f0fc5dea6aeb414e41dee23dc8ecc refCount=0 > ``` > > ollama received a request that changed the parameters of the model, probably a different `num_ctx`. Wait a minute, this might actually be it! In the original script, the context_length for the API call would change between two constants based on certain conditions met. So you're telling me I can't just change the context_length to whatever I want on the fly, I have to reload the model entirely?
GiteaMirror commented 2026-04-28 20:19:21 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Nov 21, 2024):

The context window is a fundamental parameter of a model, if it changes the model needs to be reloaded. You can either adjust context_length to max(constant1, constant2), or remove the num_ctx arguments from your calls altogether and load the model with an appropriate num_ctx in the class constructor.

  def __init__(self, ...):
     ollama.chat(
       model=self.language_model,
       options={
         "num_ctx": context_length
       }
    )
<!-- gh-comment-id:2490852501 --> @rick-github commented on GitHub (Nov 21, 2024): The context window is a fundamental parameter of a model, if it changes the model needs to be reloaded. You can either adjust `context_length` to `max(constant1, constant2)`, or remove the `num_ctx` arguments from your calls altogether and load the model with an appropriate `num_ctx` in the class constructor. ```python def __init__(self, ...): ollama.chat( model=self.language_model, options={ "num_ctx": context_length } ) ```
GiteaMirror commented 2026-04-28 20:19:22 -05:00
Author
Owner
Copy Link

@SingularityMan commented on GitHub (Nov 21, 2024):

The context window is a fundamental parameter of a model, if it changes the model needs to be reloaded. You can either adjust context_length to max(constant1, constant2), or remove the num_ctx arguments from your calls altogether and load the model with an appropriate num_ctx in the class constructor.

  def __init__(self, ...):
     ollama.chat(
       model=self.language_model,
       options={
         "num_ctx": context_length
       }
    )

Got it! Thanks a lot!

<!-- gh-comment-id:2491014038 --> @SingularityMan commented on GitHub (Nov 21, 2024): > The context window is a fundamental parameter of a model, if it changes the model needs to be reloaded. You can either adjust `context_length` to `max(constant1, constant2)`, or remove the `num_ctx` arguments from your calls altogether and load the model with an appropriate `num_ctx` in the class constructor. > > ```python > def __init__(self, ...): > ollama.chat( > model=self.language_model, > options={ > "num_ctx": context_length > } > ) > ``` Got it! Thanks a lot!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#51477
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?