github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #10104] How to force mmap=True in low-memory conditions? (can't load/test a 250Gb quantized model with 200Gb RAM and 80Gb VRAM) #6625

New Issue
Closed
opened 2026-04-12 18:18:03 -05:00 by GiteaMirror · 5 comments
GiteaMirror commented 2026-04-12 18:18:03 -05:00
Owner
Copy Link

Originally created by @vadimkantorov on GitHub (Apr 3, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/10104

What is the issue?

I'm running:

ollama serve &> ollama.log.txt &
ollama run sunny-g/deepseek-v3-0324:ud-q2_k_xl # 250Gb model

The model size is 250Gb, and I have only 200Gb RAM and 1xH100 80 Gb.

When running, this prints:

Error: llama runner process has terminated: error loading model: unable to allocate CPU buffer
llama_model_load_from_file_impl: failed to load model

It would be nicer if this log message explained RAM/VRAM requirements more clear.

However, in the log I have:

time=2025-04-03T07:49:51.293Z level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /home/inferencer/.ollama/models/blobs/sha256-79ebc258e26755a8d69ef2a373bf368541c756a4bcec23bf6ab678fdfa4c538c --ctx-size 2048 --batch-size 512 --n-gpu-layers 21 --threads 8 --no-mmap --parallel 1 --port 42079"

load_tensors: loading model tensors, this can take a while... (mmap = false)
ggml_aligned_malloc: insufficient memory (attempted to allocate 236119.06 MB)
ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 247588783616
llama_model_load: error loading model: unable to allocate CPU buffer
llama_model_load_from_file_impl: failed to load model

It appears that somehow --no-mmap is used implicitly. Is it possible to turn it off and still run the model with offloading/mmap (relying on system's mmap page handling)?

In ollama serve --help / ollama run --help are no pointers to controlling --no-mmap or other runner switches:

$ ollama serve --help

Start ollama

Usage:
  ollama serve [flags]

Aliases:
  serve, start

Flags:
  -h, --help   help for serve

Environment Variables:
      OLLAMA_DEBUG               Show additional debug information (e.g. OLLAMA_DEBUG=1)
      OLLAMA_HOST                IP Address for the ollama server (default 127.0.0.1:11434)
      OLLAMA_KEEP_ALIVE          The duration that models stay loaded in memory (default "5m")
      OLLAMA_MAX_LOADED_MODELS   Maximum number of loaded models per GPU
      OLLAMA_MAX_QUEUE           Maximum number of queued requests
      OLLAMA_MODELS              The path to the models directory
      OLLAMA_NUM_PARALLEL        Maximum number of parallel requests
      OLLAMA_NOPRUNE             Do not prune model blobs on startup
      OLLAMA_ORIGINS             A comma separated list of allowed origins
      OLLAMA_SCHED_SPREAD        Always schedule model across all GPUs

      OLLAMA_FLASH_ATTENTION     Enabled flash attention
      OLLAMA_KV_CACHE_TYPE       Quantization type for the K/V cache (default: f16)
      OLLAMA_LLM_LIBRARY         Set LLM library to bypass autodetection
      OLLAMA_GPU_OVERHEAD        Reserve a portion of VRAM per GPU (bytes)
      OLLAMA_LOAD_TIMEOUT        How long to allow model loads to stall before giving up (default "5m")

$ ollama run --help

Usage:
  ollama run MODEL [PROMPT] [flags]

Flags:
      --format string      Response format (e.g. json)
  -h, --help               help for run
      --insecure           Use an insecure registry
      --keepalive string   Duration to keep a model loaded (e.g. 5m)
      --nowordwrap         Don't wrap words to the next line automatically
      --verbose            Show timings for response

Environment Variables:
      OLLAMA_HOST                IP Address for the ollama server (default 127.0.0.1:11434)
      OLLAMA_NOHISTORY           Do not preserve readline history

Thanks!

Relevant log output

OS

No response

GPU

No response

CPU

Intel

Ollama version

0.6.2

Originally created by @vadimkantorov on GitHub (Apr 3, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/10104 ### What is the issue? I'm running: ```shell ollama serve &> ollama.log.txt & ollama run sunny-g/deepseek-v3-0324:ud-q2_k_xl # 250Gb model ``` The model size is 250Gb, and I have only 200Gb RAM and 1xH100 80 Gb. When running, this prints: ``` Error: llama runner process has terminated: error loading model: unable to allocate CPU buffer llama_model_load_from_file_impl: failed to load model ``` It would be nicer if this log message explained RAM/VRAM requirements more clear. However, in the log I have: ``` time=2025-04-03T07:49:51.293Z level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /home/inferencer/.ollama/models/blobs/sha256-79ebc258e26755a8d69ef2a373bf368541c756a4bcec23bf6ab678fdfa4c538c --ctx-size 2048 --batch-size 512 --n-gpu-layers 21 --threads 8 --no-mmap --parallel 1 --port 42079" load_tensors: loading model tensors, this can take a while... (mmap = false) ggml_aligned_malloc: insufficient memory (attempted to allocate 236119.06 MB) ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 247588783616 llama_model_load: error loading model: unable to allocate CPU buffer llama_model_load_from_file_impl: failed to load model ``` It appears that somehow `--no-mmap` is used implicitly. Is it possible to turn it off and still run the model with offloading/mmap (relying on system's mmap page handling)? In `ollama serve --help` / `ollama run --help` are no pointers to controlling `--no-mmap` or other runner switches: ``` $ ollama serve --help Start ollama Usage: ollama serve [flags] Aliases: serve, start Flags: -h, --help help for serve Environment Variables: OLLAMA_DEBUG Show additional debug information (e.g. OLLAMA_DEBUG=1) OLLAMA_HOST IP Address for the ollama server (default 127.0.0.1:11434) OLLAMA_KEEP_ALIVE The duration that models stay loaded in memory (default "5m") OLLAMA_MAX_LOADED_MODELS Maximum number of loaded models per GPU OLLAMA_MAX_QUEUE Maximum number of queued requests OLLAMA_MODELS The path to the models directory OLLAMA_NUM_PARALLEL Maximum number of parallel requests OLLAMA_NOPRUNE Do not prune model blobs on startup OLLAMA_ORIGINS A comma separated list of allowed origins OLLAMA_SCHED_SPREAD Always schedule model across all GPUs OLLAMA_FLASH_ATTENTION Enabled flash attention OLLAMA_KV_CACHE_TYPE Quantization type for the K/V cache (default: f16) OLLAMA_LLM_LIBRARY Set LLM library to bypass autodetection OLLAMA_GPU_OVERHEAD Reserve a portion of VRAM per GPU (bytes) OLLAMA_LOAD_TIMEOUT How long to allow model loads to stall before giving up (default "5m") $ ollama run --help Usage: ollama run MODEL [PROMPT] [flags] Flags: --format string Response format (e.g. json) -h, --help help for run --insecure Use an insecure registry --keepalive string Duration to keep a model loaded (e.g. 5m) --nowordwrap Don't wrap words to the next line automatically --verbose Show timings for response Environment Variables: OLLAMA_HOST IP Address for the ollama server (default 127.0.0.1:11434) OLLAMA_NOHISTORY Do not preserve readline history ``` Thanks! ### Relevant log output ```shell ``` ### OS _No response_ ### GPU _No response_ ### CPU Intel ### Ollama version 0.6.2
GiteaMirror added the bug label 2026-04-12 18:18:03 -05:00
GiteaMirror commented 2026-04-12 18:18:07 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 3, 2025):

The ollama CLI is pretty simple, if you want better control you can use a different client or roll your own:

#!/usr/bin/env python3

import ollama
import argparse
import readline
import sys

parser = argparse.ArgumentParser()
parser.add_argument("model")
parser.add_argument("prompts", nargs='*')
args = parser.parse_args()

client = ollama.Client()
userprompt = ">>> " if sys.stdin.isatty() else ""

def chat(messages, prompt):
  messages.append({"role":"user", "content": prompt})
  response = client.chat(
      model=args.model,
      options={"use_mmap":True},
      messages=messages,
      stream=True)
  m = ''
  for r in response:
    c = r['message']['content']
    print(c, end='', flush=True)
    m = m + c
  print()
  messages.append({"role": "assistant", "content": m})
  return messages

messages = []
for prompt in args.prompts:
  messages = chat(messages, prompt)
while True:
  try:
    prompt = input(userprompt)
  except:
    print()
    break
  if prompt == "/bye":
    break
  messages = chat(messages, prompt)
<!-- gh-comment-id:2775345331 --> @rick-github commented on GitHub (Apr 3, 2025): The ollama CLI is pretty simple, if you want better control you can use a [different client](https://github.com/ollama/ollama?tab=readme-ov-file#community-integrations) or roll your own: ```python #!/usr/bin/env python3 import ollama import argparse import readline import sys parser = argparse.ArgumentParser() parser.add_argument("model") parser.add_argument("prompts", nargs='*') args = parser.parse_args() client = ollama.Client() userprompt = ">>> " if sys.stdin.isatty() else "" def chat(messages, prompt): messages.append({"role":"user", "content": prompt}) response = client.chat( model=args.model, options={"use_mmap":True}, messages=messages, stream=True) m = '' for r in response: c = r['message']['content'] print(c, end='', flush=True) m = m + c print() messages.append({"role": "assistant", "content": m}) return messages messages = [] for prompt in args.prompts: messages = chat(messages, prompt) while True: try: prompt = input(userprompt) except: print() break if prompt == "/bye": break messages = chat(messages, prompt) ```
GiteaMirror commented 2026-04-12 18:18:08 -05:00
Author
Owner
Copy Link

@vadimkantorov commented on GitHub (Apr 3, 2025):

My question is about controlling the use of mmap / RAM model loading, not on how to query ollama models...

<!-- gh-comment-id:2775401376 --> @vadimkantorov commented on GitHub (Apr 3, 2025): My question is about controlling the use of `mmap` / RAM model loading, not on how to query ollama models...
GiteaMirror commented 2026-04-12 18:18:09 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 3, 2025):

mmap/RAM model loading can't be controlled from the ollama CLI because it is too simple. In your bug report, you show running the model using ollama run, so I gave you a pointer to a list of potential client replacements, and an example of how to quickly resolve your situation using a python script.

If your requirements are for controlling the use of mmap by means other than the ollama CLI, then the API documentation gives examples. You can also modify the Modelfile of a model to assign use_mmap to true.

<!-- gh-comment-id:2775443190 --> @rick-github commented on GitHub (Apr 3, 2025): mmap/RAM model loading can't be controlled from the ollama CLI because it is too simple. In your bug report, you show running the model using `ollama run`, so I gave you a pointer to a list of potential client replacements, and an example of how to quickly resolve your situation using a python script. If your requirements are for controlling the use of mmap by means other than the ollama CLI, then the [API documentation](https://github.com/ollama/ollama/blob/main/docs/api.md#generate-request-with-options:~:text=vocab_only%22%3A%20false%2C%0A%20%20%20%20%22-,use_mmap,-%22%3A%20true%2C%0A%20%20%20%20%22use_mlock) gives examples. You can also modify the [Modelfile](https://github.com/ollama/ollama/blob/main/docs/modelfile.md) of a model to assign `use_mmap` to true.
GiteaMirror commented 2026-04-12 18:18:11 -05:00
Author
Owner
Copy Link

@vadimkantorov commented on GitHub (Apr 3, 2025):

In https://github.com/ollama/ollama/blob/main/docs/modelfile.md are no mentions of mmap (or use_mmap), it only appears to exist as --no-mmap switch to the runner (which is somehow gets used by ollama run)

<!-- gh-comment-id:2775462205 --> @vadimkantorov commented on GitHub (Apr 3, 2025): In https://github.com/ollama/ollama/blob/main/docs/modelfile.md are no mentions of `mmap` (or `use_mmap`), it only appears to exist as `--no-mmap` switch to the runner (which is somehow gets used by `ollama run`)
GiteaMirror commented 2026-04-12 18:18:13 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 3, 2025):

echo FROM sunny-g/deepseek-v3-0324:ud-q2_k_xl > Modelfile
echo PARAMETER use_mmap true >> Modelfile
ollama create deepseek-v3-0324:mmap
<!-- gh-comment-id:2775470304 --> @rick-github commented on GitHub (Apr 3, 2025): ``` echo FROM sunny-g/deepseek-v3-0324:ud-q2_k_xl > Modelfile echo PARAMETER use_mmap true >> Modelfile ollama create deepseek-v3-0324:mmap ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#6625
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?