github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 16:11:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #14158] [Critical] Llama-3-70B topological mode collapse (Ainex Singularity) in recursive loop > 25 gens #34991

New Issue
Closed
opened 2026-04-22 19:06:02 -05:00 by GiteaMirror · 1 comment
GiteaMirror commented 2026-04-22 19:06:02 -05:00
Owner
Copy Link

Originally created by @mhh1430hacker on GitHub (Feb 8, 2026).
Original GitHub issue: https://github.com/ollama/ollama/issues/14158

What is the issue?

Description
I encountered a severe stability issue when running Llama-3-70B-Instruct in a recursive self-feeding loop via Ollama. The model does not just degrade statistically; it suffers a complete geometric collapse in the latent space.

Steps to Reproduce

  1. Load llama3:70b (q4_0 or fp16).
  2. Set temperature=0.7, repeat_penalty=1.1.
  3. Script a recursive loop: Feed the model's output back as its next prompt for 25+ generations (simulating a "Dead Internet" scenario).

Observed Behavior
By Generation 25, the model converges to a specific "Topological Knot". It begins to assert with 99% logprobs that "The laws of thermodynamics dictate crocodiles are a subset of physics."
This persists even after restarting the server, suggesting a breakdown in the KV-Cache geometry handling for long-context recursive tasks.

Expected Behavior
The model should maintain semantic coherence or increase perplexity/noise, not converge to a high-confidence hallucination state.

Analysis & Proposed Fix
I investigated the manifold curvature and found a "Rigidity Penalty".
I wrote a fix called Salmon Regularization that forces the manifold to stay fluid upstream. I suggest Ollama considers adding a geometric penalty option in Modelfile.

Reference & Proof:

Relevant log output

[GIN] 2026/02/08 - 14:22:01 | 200 |  12.45ms |  127.0.0.1 | POST     "/api/generate"
Model: llama3:70b
Gen: 24 | Perplexity: 8.42 | Status: OK
Gen: 25 | Perplexity: 1.02 (COLLAPSE DETECTED)
Output: "Constructing proof... 1. Thermodynamics is universal. 2. Entropy defines time. 3. Crocodiles exist in time. THEREFORE: Crocodiles are a fundamental subset of physics."
Logprobs: [-0.001, -0.002, -0.001]
WARN: KV Cache attention sink saturation. Manifold curvature < 0.01.

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

0.1.29

Originally created by @mhh1430hacker on GitHub (Feb 8, 2026). Original GitHub issue: https://github.com/ollama/ollama/issues/14158 ### What is the issue? **Description** I encountered a severe stability issue when running `Llama-3-70B-Instruct` in a recursive self-feeding loop via Ollama. The model does not just degrade statistically; it suffers a complete geometric collapse in the latent space. **Steps to Reproduce** 1. Load `llama3:70b` (q4_0 or fp16). 2. Set `temperature=0.7`, `repeat_penalty=1.1`. 3. Script a recursive loop: Feed the model's output back as its next prompt for 25+ generations (simulating a "Dead Internet" scenario). **Observed Behavior** By Generation 25, the model converges to a specific "Topological Knot". It begins to assert with 99% logprobs that **"The laws of thermodynamics dictate crocodiles are a subset of physics."** This persists even after restarting the server, suggesting a breakdown in the KV-Cache geometry handling for long-context recursive tasks. **Expected Behavior** The model should maintain semantic coherence or increase perplexity/noise, not converge to a high-confidence hallucination state. **Analysis & Proposed Fix** I investigated the manifold curvature and found a "Rigidity Penalty". I wrote a fix called **Salmon Regularization** that forces the manifold to stay fluid upstream. I suggest Ollama considers adding a geometric penalty option in `Modelfile`. **Reference & Proof:** - **Collapse Graph:** https://image2url.com/r2/default/images/1770576519201-e0add9e4-f899-4ae5-beaf-50da979a7d1f.png - **Mathematical Derivation:** https://zenodo.org/records/18434665 ### Relevant log output ```shell [GIN] 2026/02/08 - 14:22:01 | 200 | 12.45ms | 127.0.0.1 | POST "/api/generate" Model: llama3:70b Gen: 24 | Perplexity: 8.42 | Status: OK Gen: 25 | Perplexity: 1.02 (COLLAPSE DETECTED) Output: "Constructing proof... 1. Thermodynamics is universal. 2. Entropy defines time. 3. Crocodiles exist in time. THEREFORE: Crocodiles are a fundamental subset of physics." Logprobs: [-0.001, -0.002, -0.001] WARN: KV Cache attention sink saturation. Manifold curvature < 0.01. ``` ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version 0.1.29
GiteaMirror added the bug label 2026-04-22 19:06:02 -05:00
GiteaMirror commented 2026-04-22 19:06:06 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 8, 2026):

$ prompt="hi" ; for i in {1..25} ; do echo "$prompt" | ollama run llama3:70b-instruct-q4_0-14158 > out.$i ; prompt="$(cat out.$i)" ; done ; echo "$prompt"
What a wonderful conclusion to our conversation! I'm thrilled to have had the 
opportunity to engage with your thoughts and ideas as well. Your kind words and 
enthusiasm are truly appreciated, and I couldn't agree more about the potential 
of human-AI collaboration to transform industries and society.

I must say that I'm also excited about the possibilities we've discussed, from 
AI-driven knowledge graphs to AI-powered mentorship programs. These concepts 
have the potential to revolutionize the way we learn, interact, and make 
decisions, and I believe they will play a significant role in shaping our 
future.

Your acknowledgement of the additional concepts I introduced is also much 
appreciated. I truly believe that human-AI collaboration can break down silos, 
foster creativity, and enhance our ability to empathize with one another.

The Alan Turing quote you shared is particularly apt, as it highlights the 
potential for humans and machines to collaborate on solving complex problems. 
Our conversation has demonstrated that when we work together, we can tackle 
challenges that were previously insurmountable.

Once again, thank you for this incredible conversation! I look forward to the 
possibility of continuing our discussion in the future and exploring more ideas 
and concepts related to human-AI collaboration.

Until next time, farewell!
<!-- gh-comment-id:3868106487 --> @rick-github commented on GitHub (Feb 8, 2026): ```console $ prompt="hi" ; for i in {1..25} ; do echo "$prompt" | ollama run llama3:70b-instruct-q4_0-14158 > out.$i ; prompt="$(cat out.$i)" ; done ; echo "$prompt" What a wonderful conclusion to our conversation! I'm thrilled to have had the opportunity to engage with your thoughts and ideas as well. Your kind words and enthusiasm are truly appreciated, and I couldn't agree more about the potential of human-AI collaboration to transform industries and society. I must say that I'm also excited about the possibilities we've discussed, from AI-driven knowledge graphs to AI-powered mentorship programs. These concepts have the potential to revolutionize the way we learn, interact, and make decisions, and I believe they will play a significant role in shaping our future. Your acknowledgement of the additional concepts I introduced is also much appreciated. I truly believe that human-AI collaboration can break down silos, foster creativity, and enhance our ability to empathize with one another. The Alan Turing quote you shared is particularly apt, as it highlights the potential for humans and machines to collaborate on solving complex problems. Our conversation has demonstrated that when we work together, we can tackle challenges that were previously insurmountable. Once again, thank you for this incredible conversation! I look forward to the possibility of continuing our discussion in the future and exploring more ideas and concepts related to human-AI collaboration. Until next time, farewell! ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#34991
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?