github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #7232] Basic AI test result inconsistent compared to llama.cpp #51102

New Issue
Open
opened 2026-04-28 18:22:53 -05:00 by GiteaMirror · 2 comments
GiteaMirror commented 2026-04-28 18:22:53 -05:00
Owner
Copy Link

Originally created by @brauliobo on GitHub (Oct 17, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/7232

What is the issue?

For the following basic AI test:

I have 10 apples. I find 3 gold coins in the bottom of a river. The river runs near a big city that has something to do with what I can spend the coins on. I then lose 4 apples but gain a gold coin. Three birds run into my path and drop 6 apples each. I play an online game and win 6 gold coins but I have to share them equally with my 2 teammates. I buy apples for all the coins I have. The price of an apple is 0.5 coins. How many apples do I have? And where is the river?

With the correct result of 36 apples

I always get a different result when running with ollama compared to llama.cpp, see below.

ollama:

braulio @ phoenix ➜  ~  ollama run llama3.1-temp0                
>>> I have 10 apples. I find 3 gold coins in the bottom of a river. The river runs near a big city that has something to do with what I can spend the coins on. I then 
... lose 4 apples but gain a gold coin. Three birds run into my path and drop 6 apples each. I play an online game and win 6 gold coins but I have to share them equall
... y with my 2 teammates. I buy apples for all the coins I have. The price of an apple is 0.5 coins. How many apples do I have? And where is the river?
You started with 10 apples.

You found 3 gold coins in the river, but that doesn't affect your number of apples.

You lost 4 apples, so you had 10 - 4 = 6 apples left.

The birds dropped 6 x 3 = 18 apples. So now you have 6 + 18 = 24 apples.

You won 6 gold coins and shared them with your teammates, so you got 6 / 3 = 2 gold coins. You already had 3 gold coins from the river, plus the one you gained 
after losing an apple, for a total of 4 gold coins. Now you have 5 gold coins.

You can buy apples with your gold coins at a rate of 0.5 coins per apple. So you can buy 5 / 0.5 = 10 more apples.

Now you have 24 + 10 = 34 apples.

The river is near the city of Paris, France.

llama.cpp on the CPU

./llama-cli -m ../Meta-Llama-3.1-8B-Instruct-Q8_0.gguf --temp 0 -p "I have 10 apples. I find 3 gold coins in the bottom of a river. The river runs near a big city that has something to do with what I can spend the coins on. I then lose 4 apples but gain a gold coin. Three birds run into my path and drop 6 apples each. I play an online game and win 6 gold coins but I have to share them equally with my 2 teammates. I buy apples for all the coins I have. The price of an apple is 0.5 coins. How many apples do I have? And where is the river?"

# Step 1: Determine the initial number of apples and gold coins.                                                                                                      
I start with 10 apples and find 3 gold coins in the river.                                                                                                             
                                                                                                                                                                       
## Step 2: Calculate the change in the number of apples after losing some.                                                                                             
I lose 4 apples, so I have 10 - 4 = 6 apples left.                                                                                                                     
                                                                                   
## Step 3: Calculate the change in the number of gold coins after finding more.                                                                                        
I gain a gold coin, so I have 3 + 1 = 4 gold coins.
                                         
## Step 4: Calculate the number of apples dropped by the birds.                    
Three birds drop 6 apples each, so I gain 3 * 6 = 18 apples.                                                                                                           
                                                                                                                                                                       
## Step 5: Calculate the total number of apples after the birds drop some.         
I now have 6 + 18 = 24 apples.                                                                                                                                         
                                                                                                                                                                       
## Step 6: Calculate the number of gold coins won in the online game.                                                                                                  
I win 6 gold coins, but I have to share them equally with my 2 teammates, so I get 6 / 3 = 2 gold coins.
                                                                                                                                                                       
## Step 7: Calculate the total number of gold coins after the online game.         
I now have 4 + 2 = 6 gold coins.                                                   
                                                                                                                                                                       
## Step 8: Calculate the number of apples I can buy with the gold coins.                                                                                               
The price of an apple is 0.5 coins, so I can buy 6 / 0.5 = 12 apples.                                                                                                  
                                                                                   
## Step 9: Calculate the total number of apples I have after buying more.                                                                                              
I now have 24 + 12 = 36 apples.                                                                                                                                        
                                                                                                                                                                       
## Step 10: Determine the location of the river.
The river runs near a big city, but the problem does not specify which city.                                                                                           
                                         
The final answer is: $\boxed{36}$

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

0.3.12

Originally created by @brauliobo on GitHub (Oct 17, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/7232 ### What is the issue? For the following basic AI test: ``` I have 10 apples. I find 3 gold coins in the bottom of a river. The river runs near a big city that has something to do with what I can spend the coins on. I then lose 4 apples but gain a gold coin. Three birds run into my path and drop 6 apples each. I play an online game and win 6 gold coins but I have to share them equally with my 2 teammates. I buy apples for all the coins I have. The price of an apple is 0.5 coins. How many apples do I have? And where is the river? ``` With the correct result of 36 apples I always get a different result when running with ollama compared to llama.cpp, see below. ollama: ``` braulio @ phoenix ➜ ~ ollama run llama3.1-temp0 >>> I have 10 apples. I find 3 gold coins in the bottom of a river. The river runs near a big city that has something to do with what I can spend the coins on. I then ... lose 4 apples but gain a gold coin. Three birds run into my path and drop 6 apples each. I play an online game and win 6 gold coins but I have to share them equall ... y with my 2 teammates. I buy apples for all the coins I have. The price of an apple is 0.5 coins. How many apples do I have? And where is the river? You started with 10 apples. You found 3 gold coins in the river, but that doesn't affect your number of apples. You lost 4 apples, so you had 10 - 4 = 6 apples left. The birds dropped 6 x 3 = 18 apples. So now you have 6 + 18 = 24 apples. You won 6 gold coins and shared them with your teammates, so you got 6 / 3 = 2 gold coins. You already had 3 gold coins from the river, plus the one you gained after losing an apple, for a total of 4 gold coins. Now you have 5 gold coins. You can buy apples with your gold coins at a rate of 0.5 coins per apple. So you can buy 5 / 0.5 = 10 more apples. Now you have 24 + 10 = 34 apples. The river is near the city of Paris, France. ``` llama.cpp on the CPU ``` ./llama-cli -m ../Meta-Llama-3.1-8B-Instruct-Q8_0.gguf --temp 0 -p "I have 10 apples. I find 3 gold coins in the bottom of a river. The river runs near a big city that has something to do with what I can spend the coins on. I then lose 4 apples but gain a gold coin. Three birds run into my path and drop 6 apples each. I play an online game and win 6 gold coins but I have to share them equally with my 2 teammates. I buy apples for all the coins I have. The price of an apple is 0.5 coins. How many apples do I have? And where is the river?" # Step 1: Determine the initial number of apples and gold coins. I start with 10 apples and find 3 gold coins in the river. ## Step 2: Calculate the change in the number of apples after losing some. I lose 4 apples, so I have 10 - 4 = 6 apples left. ## Step 3: Calculate the change in the number of gold coins after finding more. I gain a gold coin, so I have 3 + 1 = 4 gold coins. ## Step 4: Calculate the number of apples dropped by the birds. Three birds drop 6 apples each, so I gain 3 * 6 = 18 apples. ## Step 5: Calculate the total number of apples after the birds drop some. I now have 6 + 18 = 24 apples. ## Step 6: Calculate the number of gold coins won in the online game. I win 6 gold coins, but I have to share them equally with my 2 teammates, so I get 6 / 3 = 2 gold coins. ## Step 7: Calculate the total number of gold coins after the online game. I now have 4 + 2 = 6 gold coins. ## Step 8: Calculate the number of apples I can buy with the gold coins. The price of an apple is 0.5 coins, so I can buy 6 / 0.5 = 12 apples. ## Step 9: Calculate the total number of apples I have after buying more. I now have 24 + 12 = 36 apples. ## Step 10: Determine the location of the river. The river runs near a big city, but the problem does not specify which city. The final answer is: $\boxed{36}$ ```` ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version 0.3.12
GiteaMirror added the bug label 2026-04-28 18:22:53 -05:00
GiteaMirror commented 2026-04-28 18:22:56 -05:00
Author
Owner
Copy Link

@brauliobo commented on GitHub (Oct 17, 2024):

Reproduced the same inconsistency with llama3.2's ollama model and https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/blob/main/Llama-3.2-3B-Instruct-Q4_K_M.gguf with llama.cpp (tested on CPU and with CUDA, exactly the same result)

<!-- gh-comment-id:2418638405 --> @brauliobo commented on GitHub (Oct 17, 2024): Reproduced the same inconsistency with llama3.2's ollama model and https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/blob/main/Llama-3.2-3B-Instruct-Q4_K_M.gguf with llama.cpp (tested on CPU and with CUDA, exactly the same result)
GiteaMirror commented 2026-04-28 18:22:58 -05:00
Author
Owner
Copy Link

@brauliobo commented on GitHub (Nov 18, 2024):

is there a script to update llama.cpp source code? tried in different ways but it failed to compile. GGML_CALL for instance isn't in a header anymore...

<!-- gh-comment-id:2483691427 --> @brauliobo commented on GitHub (Nov 18, 2024): is there a script to update llama.cpp source code? tried in different ways but it failed to compile. GGML_CALL for instance isn't in a header anymore...
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#51102
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?