github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #1459] LiteLLM does not forward temperature to Ollama models #783

New Issue
Closed
opened 2026-04-12 10:28:00 -05:00 by GiteaMirror · 1 comment
GiteaMirror commented 2026-04-12 10:28:00 -05:00
Owner
Copy Link

Originally created by @scpedicini on GitHub (Dec 11, 2023).
Original GitHub issue: https://github.com/ollama/ollama/issues/1459

There seems to be an issue with temperature setting not being properly passed through LiteLLM to Ollama.

When running against Ollama API directly

curl http://localhost:11434/api/chat -d '{
  "model": "mistral",
  "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "Write a single paragraph about DNA."
        }
    ],
    "options": {
        "temperature": 0.0
    },
    "stream": false
}'

Response

{"model":"mistral","created_at":"2023-12-11T03:45:37.121290844Z","message":{"role":"assistant","content":"DNA, short for deoxyribonucleic acid, is a complex, long-chain molecule that carries the genetic code and instructions used in the growth, development, functioning, and reproduction of all living organisms. It is composed of four chemical building blocks or nucleotides, which are adenine (A), guanine (G), cytosine (C), and thymine (T). The sequence of these nucleotides within DNA determines the genetic code, which is unique to each individual and passed down from parents during reproduction. DNA is organized into 23 pairs of chromosomes, which contain all the genetic information necessary for the development and survival of an organism."},"done":true,"total_duration":16615312883,"prompt_eval_count":22,"prompt_eval_duration":2769643000,"eval_count":148,"eval_duration":13834034000}%

Ran this several times and produced exact same output which is what I would expect given temperature of 0.0.

However, when the same command is sent to LiteLLM which is connected to Ollama via the config YAML:

model_list: 
    - model_name: gpt-3.5-turbo # user-facing model alias
      litellm_params: # all params accepted by litellm.completion() - https://docs.litellm.ai/docs/completion/input
          model: ollama/mistral
          api_base: http://ollama:11434

litellm_settings: 
    drop_params: True
    set_verbose: True

Hitting chat/completions LiteLLM:

curl --location 'http://0.0.0.0:4000/chat/completions' \
    --header 'Content-Type: application/json' \
    --data ' {
    "model": "gpt-3.5-turbo",
    "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "Write a single paragraph about DNA."
        }
    ],
    "stream": false,
    "temperature": 0.0
}'

Response 1

{"id":"chatcmpl-a877894a-cbf1-42c0-8948-5b4fcc988614","choices":[{"finish_reason":"stop","index":0,"message":{"content":" DNA, short for deoxyribonucleic acid, is a complex molecule that contains the genetic information necessary for the growth, development, and reproduction of all living organisms. It is composed of long strands of nucleotides, which are the building blocks of DNA. These nucleotides are made up of a sugar molecule, a phosphate group, and one of four nitrogenous bases: adenine (A), guanine (G), cytosine (C), and thymine (T). The sequence of these bases within DNA determines the genetic code, which is used to create and control the characteristics and functions of cells, tissues and organs. DNA replication, or the process of copying DNA, is essential for cell division and the transmission of genetic information from one generation to the next.","role":"assistant"}}],"created":1702266508,"model":"ollama/mistral","object":"chat.completion","system_fingerprint":null,"usage":{"prompt_tokens":12,"completion_tokens":159,"total_tokens":171}}%

Response 2

{"id":"chatcmpl-715a5f08-d1b8-42fe-94dd-493da90763c8","choices":[{"finish_reason":"stop","index":0,"message":{"content":"DNA, or deoxyribonucleic acid, is a complex, double-stranded molecule that carries the genetic code and instructions for the development, functioning and reproduction of all living organisms. It consists of four chemical building blocks, called nucleotides, which are adenine (A), cytosine (C), guanine (G) and thymine (T). The molecule is structured in a twisted, ladder-like formation known as the double helix, with the sugar-phosphate backbone on the outside and the nucleotides paired up on the inside. DNA replication, or the process of copying the genetic code, is essential for cell division and the transmission of traits from one generation to the next.","role":"assistant"}}],"created":1702266527,"model":"ollama/mistral","object":"chat.completion","system_fingerprint":null,"usage":{"prompt_tokens":12,"completion_tokens":143,"total_tokens":155}}%

Also something to note - I don't think that LiteLLM is doing the proper templatized conversion correctly. LiteLLM verbose log shows this is what it is sending to ollama:

2023-12-10 21:48:47 POST Request Sent from LiteLLM:
2023-12-10 21:48:47 curl -X POST \
2023-12-10 21:48:47 http://ollama:11434/api/generate \
2023-12-10 21:48:47 -d '{'model': 'mistral', 'prompt': 'You are a helpful assistant.Write a single paragraph about DNA.', 'temperature': 0.0}'

It looks like LiteLLM is just concatenating all the role contents into a single prompt (assistant, system, user).

Also, Ollama takes an options dictionary for its parameters (temperature, frequency_penalty, etc.)
https://github.com/jmorganca/ollama/blob/main/docs/api.md

I think this is the issue. Additionally, Ollama pushed a new update to their repo (and docker image) which adds a new api endpoint for chat messages:

curl http://localhost:11434/api/chat -d '{
  "model": "llama2",
  "messages": [
    {
      "role": "user",
      "content": "why is the sky blue?"
    }
  ]
}'

I think that LiteLLM might need to switch between /api/generate and /api/chat depending on the model and data passed, or depending on if somebody uses openai.completions.create vs openai.chat.completions.create.

Originally created by @scpedicini on GitHub (Dec 11, 2023). Original GitHub issue: https://github.com/ollama/ollama/issues/1459 There seems to be an issue with temperature setting not being properly passed through LiteLLM to Ollama. When running against Ollama API directly ```bash curl http://localhost:11434/api/chat -d '{ "model": "mistral", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "Write a single paragraph about DNA." } ], "options": { "temperature": 0.0 }, "stream": false }' ``` Response ```json {"model":"mistral","created_at":"2023-12-11T03:45:37.121290844Z","message":{"role":"assistant","content":"DNA, short for deoxyribonucleic acid, is a complex, long-chain molecule that carries the genetic code and instructions used in the growth, development, functioning, and reproduction of all living organisms. It is composed of four chemical building blocks or nucleotides, which are adenine (A), guanine (G), cytosine (C), and thymine (T). The sequence of these nucleotides within DNA determines the genetic code, which is unique to each individual and passed down from parents during reproduction. DNA is organized into 23 pairs of chromosomes, which contain all the genetic information necessary for the development and survival of an organism."},"done":true,"total_duration":16615312883,"prompt_eval_count":22,"prompt_eval_duration":2769643000,"eval_count":148,"eval_duration":13834034000}% ``` Ran this several times and produced exact same output which is what I would expect given temperature of 0.0. However, when the same command is sent to LiteLLM which is connected to Ollama via the config YAML: ```yaml model_list: - model_name: gpt-3.5-turbo # user-facing model alias litellm_params: # all params accepted by litellm.completion() - https://docs.litellm.ai/docs/completion/input model: ollama/mistral api_base: http://ollama:11434 litellm_settings: drop_params: True set_verbose: True ``` Hitting `chat/completions` LiteLLM: ```bash curl --location 'http://0.0.0.0:4000/chat/completions' \ --header 'Content-Type: application/json' \ --data ' { "model": "gpt-3.5-turbo", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "Write a single paragraph about DNA." } ], "stream": false, "temperature": 0.0 }' ``` **Response 1** ```json {"id":"chatcmpl-a877894a-cbf1-42c0-8948-5b4fcc988614","choices":[{"finish_reason":"stop","index":0,"message":{"content":" DNA, short for deoxyribonucleic acid, is a complex molecule that contains the genetic information necessary for the growth, development, and reproduction of all living organisms. It is composed of long strands of nucleotides, which are the building blocks of DNA. These nucleotides are made up of a sugar molecule, a phosphate group, and one of four nitrogenous bases: adenine (A), guanine (G), cytosine (C), and thymine (T). The sequence of these bases within DNA determines the genetic code, which is used to create and control the characteristics and functions of cells, tissues and organs. DNA replication, or the process of copying DNA, is essential for cell division and the transmission of genetic information from one generation to the next.","role":"assistant"}}],"created":1702266508,"model":"ollama/mistral","object":"chat.completion","system_fingerprint":null,"usage":{"prompt_tokens":12,"completion_tokens":159,"total_tokens":171}}% ``` **Response 2** ```json {"id":"chatcmpl-715a5f08-d1b8-42fe-94dd-493da90763c8","choices":[{"finish_reason":"stop","index":0,"message":{"content":"DNA, or deoxyribonucleic acid, is a complex, double-stranded molecule that carries the genetic code and instructions for the development, functioning and reproduction of all living organisms. It consists of four chemical building blocks, called nucleotides, which are adenine (A), cytosine (C), guanine (G) and thymine (T). The molecule is structured in a twisted, ladder-like formation known as the double helix, with the sugar-phosphate backbone on the outside and the nucleotides paired up on the inside. DNA replication, or the process of copying the genetic code, is essential for cell division and the transmission of traits from one generation to the next.","role":"assistant"}}],"created":1702266527,"model":"ollama/mistral","object":"chat.completion","system_fingerprint":null,"usage":{"prompt_tokens":12,"completion_tokens":143,"total_tokens":155}}% ``` Also something to note - I don't think that LiteLLM is doing the proper templatized conversion correctly. LiteLLM verbose log shows this is what it is sending to ollama: ``` 2023-12-10 21:48:47 POST Request Sent from LiteLLM: 2023-12-10 21:48:47 curl -X POST \ 2023-12-10 21:48:47 http://ollama:11434/api/generate \ 2023-12-10 21:48:47 -d '{'model': 'mistral', 'prompt': 'You are a helpful assistant.Write a single paragraph about DNA.', 'temperature': 0.0}' ``` It looks like LiteLLM is just concatenating all the role contents into a single prompt (`assistant`, `system`, `user`). Also, Ollama takes an `options` dictionary for its parameters (`temperature`, `frequency_penalty`, etc.) https://github.com/jmorganca/ollama/blob/main/docs/api.md I think this is the issue. Additionally, Ollama pushed a new update to their repo (and docker image) which adds a new api endpoint for chat messages: ```bash curl http://localhost:11434/api/chat -d '{ "model": "llama2", "messages": [ { "role": "user", "content": "why is the sky blue?" } ] }' ``` I think that LiteLLM might need to switch between `/api/generate` and `/api/chat` depending on the model and data passed, or depending on if somebody uses `openai.completions.create` vs `openai.chat.completions.create`.
GiteaMirror commented 2026-04-12 10:28:01 -05:00
Author
Owner
Copy Link

@scpedicini commented on GitHub (Dec 11, 2023):

Sorry meant to post this over on the LiteLLM repo. 🤪

<!-- gh-comment-id:1849291161 --> @scpedicini commented on GitHub (Dec 11, 2023): Sorry meant to post this over on the LiteLLM repo. 🤪
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#783
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?