github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 21:25:46 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[PR #3879] [CLOSED] Use ReadFull over CopyN when decoding GGUFs #11303

New Issue
Closed
opened 2026-04-12 23:27:17 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-04-12 23:27:17 -05:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/ollama/ollama/pull/3879
Author: @brycereitano
Created: 4/24/2024
Status: ❌ Closed

Base: main ← Head: readfull

📝 Commits (2)

  • abe154a Use ReadFull over CopyN when decoding GGUFs
  • 030aee7 Load arrays in chunks

📊 Changes

1 file changed (+71 additions, -45 deletions)

View changed files

📝 llm/gguf.go (+71 -45)

📄 Description

Opting to use io.ReadFull with preallocated []byte slices as bytes.Buffer requires multiple grows/allocs to read in long strings.

In addition, preallocating the array slices to prevent underlying allocs to append to the slice.

I observed real world performance improvements as well as by using a small microbenchmark reading phi3 4Q from disk (file was cached by the OS) I see the following improvements:

goos: linux
goarch: amd64
pkg: github.com/ollama/ollama/llm
cpu: 12th Gen Intel(R) Core(TM) i7-1260P
          │   old.out   │               new.out               │
          │   sec/op    │   sec/op     vs base                │
Decode-16   63.57m ± 2%   35.28m ± 2%  -44.50% (p=0.000 n=10)

          │    old.out    │               new.out                │
          │     B/op      │     B/op      vs base                │
Decode-16   58.691Mi ± 0%   3.195Mi ± 0%  -94.56% (p=0.000 n=10)

          │   old.out   │               new.out               │
          │  allocs/op  │  allocs/op   vs base                │
Decode-16   324.4k ± 0%   227.1k ± 0%  -29.97% (p=0.000 n=10)

Results my vary depending on disk performance and whether the file was cached by the OS.

I may dig around for performance improvements in the future.

  • Using a shared bytes.buffer for loading in strings to cut down on allocs.
  • Using bufio.Reader to buffer reads from disk.
  • Prevent repeated allocs of reading small datatypes by using a reading method that allows you to specify an existing buffer.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/ollama/ollama/pull/3879 **Author:** [@brycereitano](https://github.com/brycereitano) **Created:** 4/24/2024 **Status:** ❌ Closed **Base:** `main` ← **Head:** `readfull` --- ### 📝 Commits (2) - [`abe154a`](https://github.com/ollama/ollama/commit/abe154a4e65eff09d63e596b0e2bcf7c6ca1d58c) Use ReadFull over CopyN when decoding GGUFs - [`030aee7`](https://github.com/ollama/ollama/commit/030aee748ffccce481d37fc6045cea1f91141eed) Load arrays in chunks ### 📊 Changes **1 file changed** (+71 additions, -45 deletions) <details> <summary>View changed files</summary> 📝 `llm/gguf.go` (+71 -45) </details> ### 📄 Description Opting to use `io.ReadFull` with preallocated `[]byte` slices as `bytes.Buffer` requires multiple grows/allocs to read in long strings. In addition, preallocating the array slices to prevent underlying allocs to append to the slice. I observed real world performance improvements as well as by using a small microbenchmark reading phi3 4Q from disk (file was cached by the OS) I see the following improvements: ``` goos: linux goarch: amd64 pkg: github.com/ollama/ollama/llm cpu: 12th Gen Intel(R) Core(TM) i7-1260P │ old.out │ new.out │ │ sec/op │ sec/op vs base │ Decode-16 63.57m ± 2% 35.28m ± 2% -44.50% (p=0.000 n=10) │ old.out │ new.out │ │ B/op │ B/op vs base │ Decode-16 58.691Mi ± 0% 3.195Mi ± 0% -94.56% (p=0.000 n=10) │ old.out │ new.out │ │ allocs/op │ allocs/op vs base │ Decode-16 324.4k ± 0% 227.1k ± 0% -29.97% (p=0.000 n=10) ``` Results my vary depending on disk performance and whether the file was cached by the OS. I may dig around for performance improvements in the future. - Using a shared bytes.buffer for loading in strings to cut down on allocs. - Using bufio.Reader to buffer reads from disk. - Prevent repeated allocs of reading small datatypes by using a reading method that allows you to specify an existing buffer. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-04-12 23:27:17 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#11303
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?