github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-17 05:36:16 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[PR #15685] Reduce memory allocations by 86-99% through buffer reuse optimization #77556

New Issue
Open
opened 2026-05-05 10:13:53 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-05-05 10:13:53 -05:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/ollama/ollama/pull/15685
Author: @emad-elsaid
Created: 4/19/2026
Status: 🔄 Open

Base: main ← Head: optimizations-clean

📝 Commits (1)

  • ea0f5cb Reduce memory allocations by 86-99% through buffer reuse optimization

📊 Changes

10 files changed (+266 additions, -409 deletions)

View changed files

📝 kvcache/causal.go (+10 -1)
📝 kvcache/causal_test.go (+11 -8)
📝 ml/backend.go (+1 -1)
📝 ml/backend/ggml/ggml.go (+154 -333)
📝 ml/nn/pooling/pooling_test.go (+1 -1)
📝 runner/common/logprob.go (+37 -5)
📝 runner/ollamarunner/multimodal.go (+1 -1)
📝 runner/ollamarunner/runner.go (+27 -4)
📝 sample/samplers.go (+12 -3)
📝 sample/transforms.go (+12 -52)

📄 Description

Changes are done with Opencode/Claude code + Sonnet 4.5

Summary

This PR significantly reduces memory allocations in the inference hot path through systematic
buffer reuse, achieving 86-99% allocation reduction depending on workload configuration.

Optimizations Implemented

Core Inference Path

  • Sampling buffer reuse: Reuse candidate buffers across sampling calls (99.7% reduction)
  • TopK heap elimination: Remove allocations in TopK sampling using in-place selection
  • Tensor wrapper pooling: Pool tensor wrappers via sync.Pool (1.3M objects recycled)
  • Batch buffer reuse: Eliminate per-batch allocations in computeBatch (3 allocs/batch → 0)
  • Logits buffer reuse: Reuse logits buffer across decode steps via FloatsInto pattern

Optional Features

  • Logprob buffer pooling: sync.Pool for logprob calculations (99.45% reduction when enabled)
  • Grammar sampler buffers: Reuse token data buffers in grammar sampling (99.76% reduction)
  • KV cache mask reuse: Reuse causal attention masks (98.9% reduction)

Performance Impact

Standard workload (no logprobs/grammar):

  • Allocations: 803 MB → 116 MB (86% reduction)
  • Objects allocated: 1.4M → 200K (86% reduction)

With logprobs enabled:

  • Allocations: 874 MB → 4 MB (99.5% reduction)
  • Allocation rate: 758 MB/s → 4 MB/s

With grammar sampling:

  • Allocations: 1,126 MB → 4 MB (99.6% reduction)
  • Allocation rate: 1.0 GB/s → 2.4 MB/s

Implementation Details

All optimizations follow established patterns:

  • sync.Pool for expensive-to-allocate buffers (logprob, tensor wrappers)
  • Grow-as-needed buffers for variable-sized data (logits, batch inputs)
  • In-place algorithms to avoid temporary allocations (TopK selection)

Benchmarking

  • Benchmark code was introduced in this fork main branch https://github.com/emad-elsaid/ollama
  • The benchmark code is left out to reduce the size of this PR let me know If I should include it in this PR or maybe another separate PR
  • Also Profiling was done during the runner request to the model but also left it out
  • Benchmarks were using Linux, CPU, Qwen2.5 coder 1.5b to check memory usages and cpu profiling.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/ollama/ollama/pull/15685 **Author:** [@emad-elsaid](https://github.com/emad-elsaid) **Created:** 4/19/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `optimizations-clean` --- ### 📝 Commits (1) - [`ea0f5cb`](https://github.com/ollama/ollama/commit/ea0f5cb2160ded88d451e1de33edde5cd3f93b9a) Reduce memory allocations by 86-99% through buffer reuse optimization ### 📊 Changes **10 files changed** (+266 additions, -409 deletions) <details> <summary>View changed files</summary> 📝 `kvcache/causal.go` (+10 -1) 📝 `kvcache/causal_test.go` (+11 -8) 📝 `ml/backend.go` (+1 -1) 📝 `ml/backend/ggml/ggml.go` (+154 -333) 📝 `ml/nn/pooling/pooling_test.go` (+1 -1) 📝 `runner/common/logprob.go` (+37 -5) 📝 `runner/ollamarunner/multimodal.go` (+1 -1) 📝 `runner/ollamarunner/runner.go` (+27 -4) 📝 `sample/samplers.go` (+12 -3) 📝 `sample/transforms.go` (+12 -52) </details> ### 📄 Description Changes are done with Opencode/Claude code + Sonnet 4.5 ## Summary This PR significantly reduces memory allocations in the inference hot path through systematic buffer reuse, achieving 86-99% allocation reduction depending on workload configuration. ## Optimizations Implemented ### Core Inference Path - **Sampling buffer reuse**: Reuse candidate buffers across sampling calls (99.7% reduction) - **TopK heap elimination**: Remove allocations in TopK sampling using in-place selection - **Tensor wrapper pooling**: Pool tensor wrappers via sync.Pool (1.3M objects recycled) - **Batch buffer reuse**: Eliminate per-batch allocations in computeBatch (3 allocs/batch → 0) - **Logits buffer reuse**: Reuse logits buffer across decode steps via FloatsInto pattern ### Optional Features - **Logprob buffer pooling**: sync.Pool for logprob calculations (99.45% reduction when enabled) - **Grammar sampler buffers**: Reuse token data buffers in grammar sampling (99.76% reduction) - **KV cache mask reuse**: Reuse causal attention masks (98.9% reduction) ## Performance Impact **Standard workload (no logprobs/grammar):** - Allocations: 803 MB → 116 MB (86% reduction) - Objects allocated: 1.4M → 200K (86% reduction) **With logprobs enabled:** - Allocations: 874 MB → 4 MB (99.5% reduction) - Allocation rate: 758 MB/s → 4 MB/s **With grammar sampling:** - Allocations: 1,126 MB → 4 MB (99.6% reduction) - Allocation rate: 1.0 GB/s → 2.4 MB/s ## Implementation Details All optimizations follow established patterns: - `sync.Pool` for expensive-to-allocate buffers (logprob, tensor wrappers) - Grow-as-needed buffers for variable-sized data (logits, batch inputs) - In-place algorithms to avoid temporary allocations (TopK selection) ## Benchmarking - Benchmark code was introduced in this fork main branch https://github.com/emad-elsaid/ollama - The benchmark code is left out to reduce the size of this PR let me know If I should include it in this PR or maybe another separate PR - Also Profiling was done during the runner request to the model but also left it out - Benchmarks were using Linux, CPU, Qwen2.5 coder 1.5b to check memory usages and cpu profiling. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-05-05 10:13:53 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#77556
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?