github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #4628] aya model : error when using the generate endpoint #2906

New Issue
Open
opened 2026-04-12 13:15:37 -05:00 by GiteaMirror · 2 comments
GiteaMirror commented 2026-04-12 13:15:37 -05:00
Owner
Copy Link

Originally created by @saurabhkumar on GitHub (May 25, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/4628

What is the issue?

I am running aya model locally. When i just start the model with ollama run aya and interact in the terminal, it works fine. But when I try using it via POSTMAN on Windows 10 at (http://127.0.0.1:11434/api/generate) with the following data:

{
  "model": "aya",
  "prompt": "Are there multilingual open source large language models available? Respond using JSON",
  "format": "json",
  "stream": false
}

i get the following error:

{ "error": "error reading llm response: read tcp 127.0.0.1:60589->127.0.0.1:60580: wsarecv: An existing connection was forcibly closed by the remote host." }

For other models (e.g. phi3), it works correctly with the same setup.
e.g:

{
  "model": "phi3:mini",
  "prompt": "Are there multilingual open source large language models available? Respond using JSON",
  "format": "json",
  "stream": false
}'

works.

I can see the following logging for aya (the last line shows only the error but not any other details.):

llm_load_tensors: ggml ctx size = 0.27 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors: CPU buffer size = 820.31 MiB
llm_load_tensors: CUDA0 buffer size = 4564.83 MiB
........................................................................
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: n_batch = 512
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CUDA0 KV buffer size = 256.00 MiB
llama_new_context_with_model: KV self size = 256.00 MiB, K (f16): 128.00 MiB, V (f16): 128.00 MiB
llama_new_context_with_model: CUDA_Host output buffer size = 0.99 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 508.00 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 12.01 MiB
llama_new_context_with_model: graph nodes = 968
llama_new_context_with_model: graph splits = 2
INFO [wmain] model loaded | tid="9016" timestamp=1716611056
time=2024-05-25T06:24:16.906+02:00 level=INFO source=server.go:545 msg="llama runner started in 8.06 seconds"
[GIN] 2024/05/25 - 06:24:19 | 500 | 14.8679199s | 127.0.0.1 | POST "/api/generate"

OS

Windows

GPU

Nvidia

CPU

Intel

Ollama version

No response

Originally created by @saurabhkumar on GitHub (May 25, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/4628 ### What is the issue? I am running aya model locally. When i just start the model with `ollama run aya` and interact in the terminal, it works fine. But when I try using it via POSTMAN on Windows 10 at (http://127.0.0.1:11434/api/generate) with the following data: ``` { "model": "aya", "prompt": "Are there multilingual open source large language models available? Respond using JSON", "format": "json", "stream": false } ``` i get the following error: `{ "error": "error reading llm response: read tcp 127.0.0.1:60589->127.0.0.1:60580: wsarecv: An existing connection was forcibly closed by the remote host." }` For other models (e.g. phi3), it works correctly with the same setup. e.g: ``` { "model": "phi3:mini", "prompt": "Are there multilingual open source large language models available? Respond using JSON", "format": "json", "stream": false }' ``` works. I can see the following logging for aya (the last line shows only the error but not any other details.): > llm_load_tensors: ggml ctx size = 0.27 MiB > llm_load_tensors: offloading 32 repeating layers to GPU > llm_load_tensors: offloading non-repeating layers to GPU > llm_load_tensors: offloaded 33/33 layers to GPU > llm_load_tensors: CPU buffer size = 820.31 MiB > llm_load_tensors: CUDA0 buffer size = 4564.83 MiB > ........................................................................ > llama_new_context_with_model: n_ctx = 2048 > llama_new_context_with_model: n_batch = 512 > llama_new_context_with_model: n_ubatch = 512 > llama_new_context_with_model: freq_base = 10000.0 > llama_new_context_with_model: freq_scale = 1 > llama_kv_cache_init: CUDA0 KV buffer size = 256.00 MiB > llama_new_context_with_model: KV self size = 256.00 MiB, K (f16): 128.00 MiB, V (f16): 128.00 MiB > llama_new_context_with_model: CUDA_Host output buffer size = 0.99 MiB > llama_new_context_with_model: CUDA0 compute buffer size = 508.00 MiB > llama_new_context_with_model: CUDA_Host compute buffer size = 12.01 MiB > llama_new_context_with_model: graph nodes = 968 > llama_new_context_with_model: graph splits = 2 > INFO [wmain] model loaded | tid="9016" timestamp=1716611056 > time=2024-05-25T06:24:16.906+02:00 level=INFO source=server.go:545 msg="llama runner started in 8.06 seconds" > [GIN] 2024/05/25 - 06:24:19 | 500 | 14.8679199s | 127.0.0.1 | POST "/api/generate" ### OS Windows ### GPU Nvidia ### CPU Intel ### Ollama version _No response_
GiteaMirror added the bug label 2026-04-12 13:15:37 -05:00
GiteaMirror commented 2026-04-12 13:15:38 -05:00
Author
Owner
Copy Link

@diedetap commented on GitHub (May 26, 2024):

I've got the same problem running Linux/nvidia. The error is triggered when the api call contains "format": "json" but works correctly when not using json mode.

<!-- gh-comment-id:2132368623 --> @diedetap commented on GitHub (May 26, 2024): I've got the same problem running Linux/nvidia. The error is triggered when the api call contains `"format": "json"` but works correctly when not using json mode.
GiteaMirror commented 2026-04-12 13:15:39 -05:00
Author
Owner
Copy Link

@r-worbis commented on GitHub (May 27, 2024):

Same here. Also with "format": "json"

<!-- gh-comment-id:2132725685 --> @r-worbis commented on GitHub (May 27, 2024): Same here. Also with "format": "json"
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#2906
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?