github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 08:30:05 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[PR #13500] fix: preserve partial UTF-8 bytes in logprobs API response #40111

New Issue
Open
opened 2026-04-23 01:05:45 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-04-23 01:05:45 -05:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/ollama/ollama/pull/13500
Author: @nathannewyen
Created: 12/16/2025
Status: 🔄 Open

Base: main ← Head: fix/logprobs-utf8-bytes

📝 Commits (1)

  • 8dac29b fix: preserve partial UTF-8 bytes in logprobs API response

📊 Changes

6 files changed (+297 additions, -2 deletions)

View changed files

📝 llm/server.go (+5 -0)
📝 runner/common/logprob.go (+4 -0)
📝 runner/common/logprob_test.go (+135 -0)
📝 server/logprob.go (+20 -2)
➕ server/logprob_test.go (+127 -0)
📝 server/routes_generate_test.go (+6 -0)

📄 Description

Summary

  • Fix issue where partial UTF-8 tokens returned wrong bytes in logprobs API
  • Store raw bytes before JSON encoding to preserve them through the transfer
  • Add tests verifying the fix

Problem

When logprobs are requested, partial UTF-8 tokens (like individual bytes of an emoji 😊) return [239, 191, 189] (replacement character) instead of actual bytes like [240].

Root Cause

Token strings get corrupted during JSON marshaling between runner and server - invalid UTF-8 is replaced with U+FFFD.

Solution

  1. Added Bytes []byte field to llm.TokenLogprob
  2. Populate bytes in CalculateLogprobs() before JSON encoding
  3. Use stored bytes in toAPILogprobs() instead of converting from corrupted string

Test plan

  • Added TestCalculateLogprobsPartialUTF8Bytes - verifies raw bytes preserved
  • Added TestLogprobBytesJSONRoundTrip - verifies JSON round-trip preservation
  • Added TestToAPILogprobsPreservesBytes - verifies API conversion uses stored bytes

Fixes #13497

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/ollama/ollama/pull/13500 **Author:** [@nathannewyen](https://github.com/nathannewyen) **Created:** 12/16/2025 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/logprobs-utf8-bytes` --- ### 📝 Commits (1) - [`8dac29b`](https://github.com/ollama/ollama/commit/8dac29b9a9569d1ed8a6bb8c23226c55cb07c2e2) fix: preserve partial UTF-8 bytes in logprobs API response ### 📊 Changes **6 files changed** (+297 additions, -2 deletions) <details> <summary>View changed files</summary> 📝 `llm/server.go` (+5 -0) 📝 `runner/common/logprob.go` (+4 -0) 📝 `runner/common/logprob_test.go` (+135 -0) 📝 `server/logprob.go` (+20 -2) ➕ `server/logprob_test.go` (+127 -0) 📝 `server/routes_generate_test.go` (+6 -0) </details> ### 📄 Description ## Summary - Fix issue where partial UTF-8 tokens returned wrong bytes in logprobs API - Store raw bytes before JSON encoding to preserve them through the transfer - Add tests verifying the fix ## Problem When logprobs are requested, partial UTF-8 tokens (like individual bytes of an emoji 😊) return `[239, 191, 189]` (replacement character) instead of actual bytes like `[240]`. ## Root Cause Token strings get corrupted during JSON marshaling between runner and server - invalid UTF-8 is replaced with U+FFFD. ## Solution 1. Added `Bytes []byte` field to `llm.TokenLogprob` 2. Populate bytes in `CalculateLogprobs()` before JSON encoding 3. Use stored bytes in `toAPILogprobs()` instead of converting from corrupted string ## Test plan - Added `TestCalculateLogprobsPartialUTF8Bytes` - verifies raw bytes preserved - Added `TestLogprobBytesJSONRoundTrip` - verifies JSON round-trip preservation - Added `TestToAPILogprobsPreservesBytes` - verifies API conversion uses stored bytes Fixes #13497 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-04-23 01:05:45 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#40111
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?