enable max tolkens for anthropic/claude-3-sonnet-20240229 #439

New Issue

Closed

opened 2025-11-11 14:21:21 -06:00 by GiteaMirror · 15 comments

GiteaMirror commented 2025-11-11 14:21:21 -06:00

Owner

Copy Link

Copy Source

Originally created by @bjornjorgensen on GitHub (Mar 7, 2024).

Bug Report

Cant change max tolken when running anthropic/claude-3-sonnet-20240229
so the text is cut.

Description

Bug Summary:
[Provide a brief but clear summary of the bug]

Steps to Reproduce:
[Outline the steps to reproduce the bug. Be as detailed as possible.]
add anthropic/claude-3-sonnet-20240229

as for something bigger then 2 + 2 ?

Expected Behavior:
[Describe what you expected to happen.]

Actual Behavior:
[Describe what actually happened.]

Environment

• Operating System: [e.g., Windows 10, macOS Big Sur, Ubuntu 20.04]
• k8s I am using dev image. ghcr.io/open-webui/open-webui:dev
• Browser (if applicable): [e.g., Chrome 100.0, Firefox 98.0]
  breave

Reproduction Details

Confirmation:

• I have read and followed all the instructions provided in the README.md.
• I am on the latest version of both Open WebUI and Ollama.
• I have included the browser console logs.
• I have included the Docker container logs.

Logs and Screenshots

Browser Console Logs:
[Include relevant browser console logs, if applicable]

Docker Container Logs:
[Include relevant Docker container logs, if applicable]

Screenshots (if applicable):
[Attach any relevant screenshots to help illustrate the issue]

Installation Method

[Describe the method you used to install the project, e.g., manual installation, Docker, package manager, etc.]

Additional Information

[Include any additional details that may help in understanding and reproducing the issue. This could include specific configurations, error messages, or anything else relevant to the bug.]

Note

If the bug report is incomplete or does not follow the provided instructions, it may not be addressed. Please ensure that you have followed the steps outlined in the README.md and troubleshooting.md documents, and provide all necessary information for us to reproduce and address the issue. Thank you!

Originally created by @bjornjorgensen on GitHub (Mar 7, 2024). # Bug Report Cant change max tolken when running anthropic/claude-3-sonnet-20240229 so the text is cut. ## Description  **Bug Summary:** [Provide a brief but clear summary of the bug] **Steps to Reproduce:** [Outline the steps to reproduce the bug. Be as detailed as possible.] add anthropic/claude-3-sonnet-20240229  as for something bigger then 2 + 2 ? **Expected Behavior:** [Describe what you expected to happen.] **Actual Behavior:** [Describe what actually happened.] ## Environment - **Operating System:** [e.g., Windows 10, macOS Big Sur, Ubuntu 20.04] - k8s I am using dev image. ghcr.io/open-webui/open-webui:dev - **Browser (if applicable):** [e.g., Chrome 100.0, Firefox 98.0] breave ## Reproduction Details **Confirmation:** - [ ] I have read and followed all the instructions provided in the README.md. - [ ] I am on the latest version of both Open WebUI and Ollama. - [ ] I have included the browser console logs. - [ ] I have included the Docker container logs. ## Logs and Screenshots **Browser Console Logs:** [Include relevant browser console logs, if applicable] **Docker Container Logs:** [Include relevant Docker container logs, if applicable] **Screenshots (if applicable):** [Attach any relevant screenshots to help illustrate the issue] ## Installation Method [Describe the method you used to install the project, e.g., manual installation, Docker, package manager, etc.] ## Additional Information [Include any additional details that may help in understanding and reproducing the issue. This could include specific configurations, error messages, or anything else relevant to the bug.] ## Note If the bug report is incomplete or does not follow the provided instructions, it may not be addressed. Please ensure that you have followed the steps outlined in the README.md and troubleshooting.md documents, and provide all necessary information for us to reproduce and address the issue. Thank you!

GiteaMirror closed this issue 2025-11-11 14:21:21 -06:00

GiteaMirror commented 2025-11-11 14:21:22 -06:00

Author

Owner

Copy Link

Copy Source

@tjbck commented on GitHub (Mar 8, 2024):

I have a suspicion that it's an issue on the LiteLLM-side, could you try isolating the issue by trying with LiteLLM only? Keep us updated!

@tjbck commented on GitHub (Mar 8, 2024): I have a suspicion that it's an issue on the LiteLLM-side, could you try isolating the issue by trying with LiteLLM only? Keep us updated!

GiteaMirror commented 2025-11-11 14:21:22 -06:00

Author

Owner

Copy Link

Copy Source

@justinh-rahb commented on GitHub (Mar 8, 2024):

I have a suspicion that it's an issue on the LiteLLM-side, could you try isolating the issue by trying with LiteLLM only? Keep us updated!

I'm not so sure that it's LiteLLM to blame here, I've tried now with an older version of it and the same behaviour is happening on Claude 2 as well, from other clients than WebUI. I believe this behavior started the day that Claude 3 was released. By chance @bjornjorgensen are you using an Anthropic developer account like mine? I am beginning to wonder if they simply limit max_tokens now for free dev keys.

@justinh-rahb commented on GitHub (Mar 8, 2024): > I have a suspicion that it's an issue on the LiteLLM-side, could you try isolating the issue by trying with LiteLLM only? Keep us updated! I'm not so sure that it's LiteLLM to blame here, I've tried now with an older version of it and the same behaviour is happening on Claude 2 as well, from other clients than WebUI. I believe this behavior started the day that Claude 3 was released. By chance @bjornjorgensen are you using an Anthropic developer account like mine? I am beginning to wonder if they simply limit `max_tokens` now for free dev keys.

GiteaMirror commented 2025-11-11 14:21:23 -06:00

Author

Owner

Copy Link

Copy Source

@justinh-rahb commented on GitHub (Mar 8, 2024):

Update: now I'm not so sure where the blame lies. Testing in another chat app works fine with Claude 3 endpoints. So it could indeed be a LiteLLM issue then, but it also wasn't working with older versions of it that previously did work? Really need someone with actual paid API keys to test this I think.

@justinh-rahb commented on GitHub (Mar 8, 2024): Update: now I'm not so sure where the blame lies. Testing in another chat app works fine with Claude 3 endpoints. So it could indeed be a LiteLLM issue then, but it also wasn't working with older versions of it that previously did work? Really need someone with actual paid API keys to test this I think.

GiteaMirror commented 2025-11-11 14:21:23 -06:00

Author

Owner

Copy Link

Copy Source

@bjornjorgensen commented on GitHub (Mar 8, 2024):

it is a free key that i'm are using

it does however work on

So its not the key that are the problem.

@bjornjorgensen commented on GitHub (Mar 8, 2024): it is a free key that i'm are using  it does however work on  So its not the key that are the problem.

GiteaMirror commented 2025-11-11 14:21:23 -06:00

Author

Owner

Copy Link

Copy Source

@justinh-rahb commented on GitHub (Mar 8, 2024):

Yes I just tried it with Lobechat as well and got a full response. So ball seems to be back in LiteLLM's court but I need to do further testing with components in isolation to be really certain of this.

@justinh-rahb commented on GitHub (Mar 8, 2024): Yes I just tried it with Lobechat as well and got a full response. So ball seems to be back in LiteLLM's court but I need to do further testing with components in isolation to be really certain of this.

GiteaMirror commented 2025-11-11 14:21:23 -06:00

Author

Owner

Copy Link

Copy Source

@justinh-rahb commented on GitHub (Mar 8, 2024):

So after futher testing... I can only observe this happening when WebUI is involved. So it seems it may be something to do with our code, I just cannot at the moment nail down what it could possibly be. It seemingly only affects Claude API via LiteLLM in Open WebUI

@justinh-rahb commented on GitHub (Mar 8, 2024): So after futher testing... I can only observe this happening when WebUI is involved. So it seems it may be something to do with our code, I just cannot at the moment nail down what it could possibly be. It seemingly only affects Claude API via LiteLLM in Open WebUI

GiteaMirror commented 2025-11-11 14:21:23 -06:00

Author

Owner

Copy Link

Copy Source

@bjornjorgensen commented on GitHub (Mar 8, 2024):

are there any configs that over rights tolkens that it can print on outputs?

@bjornjorgensen commented on GitHub (Mar 8, 2024): are there any configs that over rights tolkens that it can print on outputs?

GiteaMirror commented 2025-11-11 14:21:24 -06:00

Author

Owner

Copy Link

Copy Source

@justinh-rahb commented on GitHub (Mar 8, 2024):

@bjornjorgensen nah, in our testing we've checked we're not sending anything like that would limit the max tokens, but nonetheless the API response says the stop reason is length which would indicate that it's been given one and reached it... very strange. Still being investigated and I hope we'll have an answer soon!

@justinh-rahb commented on GitHub (Mar 8, 2024): @bjornjorgensen nah, in our testing we've checked we're not sending anything like that would limit the max tokens, but nonetheless the API response says the stop reason is `length` which would indicate that it's been given one and reached it... very strange. Still being investigated and I hope we'll have an answer soon!

GiteaMirror commented 2025-11-11 14:21:24 -06:00

Author

Owner

Copy Link

Copy Source

@justinh-rahb commented on GitHub (Mar 8, 2024):

Ladies and gentleman, we got em. Claude's API now requires that the max_tokens param be sent in the payload, and LiteLLM will set a default of 256 tokens if you don't specify this. Currently the WebUI does not send a max_tokens param when using external APIs, so the proposed fix would be to add that feature, or allow this parameter override to be set in the LiteLLM configuration UI. For now, it can be worked around by mounting and modifying the config.yaml file as such:

- litellm_params:
    api_key: your_api_key
    model: anthropic/claude-3-sonnet-20240229
    max_tokens: 4096
  model_info:
    id: 810226a0-61e2-4d97-9de0-822bd4300fcd
  model_name: claude-3-sonnet

Note: the maximum value is 4096, you'll get an error from Anthropic's API if you request more.

@justinh-rahb commented on GitHub (Mar 8, 2024): _Ladies and gentleman, we got em._ Claude's API now **requires** that the `max_tokens` param be sent in the payload, and LiteLLM will set a default of 256 tokens if you don't specify this. Currently the WebUI does not send a `max_tokens` param when using external APIs, so the proposed fix would be to add that feature, or allow this parameter override to be set in the LiteLLM configuration UI. For now, it can be worked around by mounting and modifying the `config.yaml` file as such: ```yaml - litellm_params: api_key: your_api_key model: anthropic/claude-3-sonnet-20240229 max_tokens: 4096 model_info: id: 810226a0-61e2-4d97-9de0-822bd4300fcd model_name: claude-3-sonnet ``` Note: the maximum value is 4096, you'll get an error from Anthropic's API if you request more.

GiteaMirror commented 2025-11-11 14:21:24 -06:00

Author

Owner

Copy Link

Copy Source

@tjbck commented on GitHub (Mar 8, 2024):

https://docs.anthropic.com/claude/docs/models-overview

@tjbck commented on GitHub (Mar 8, 2024): https://docs.anthropic.com/claude/docs/models-overview

GiteaMirror commented 2025-11-11 14:21:25 -06:00

Author

Owner

Copy Link

Copy Source

@justinh-rahb commented on GitHub (Mar 8, 2024):

v0.1.111 (not merged to :main yet) has a new field in the LiteLLM UI to configure the max_tokens parameter override, which will make modifying your config.yaml by hand unneccesary. This can be tested now in the :dev branch.

@justinh-rahb commented on GitHub (Mar 8, 2024): **v0.1.111** (not merged to `:main` yet) has a new field in the LiteLLM UI to configure the `max_tokens` parameter override, which will make modifying your `config.yaml` by hand unneccesary. This can be tested now in the `:dev` branch.

GiteaMirror commented 2025-11-11 14:21:25 -06:00

Author

Owner

Copy Link

Copy Source

@tjbck commented on GitHub (Mar 10, 2024):

max_tokens: 4096 should be explicitly set from the settings!

@tjbck commented on GitHub (Mar 10, 2024): max_tokens: 4096 should be explicitly set from the settings!

GiteaMirror commented 2025-11-11 14:21:25 -06:00

Author

Owner

Copy Link

Copy Source

@bjornjorgensen commented on GitHub (Mar 11, 2024):

yes, I have to delete the old one and add it back.. but now it works :)
Thanks

@bjornjorgensen commented on GitHub (Mar 11, 2024): yes, I have to delete the old one and add it back.. but now it works :) Thanks

GiteaMirror commented 2025-11-11 14:21:25 -06:00

Author

Owner

Copy Link

Copy Source

@bjornjorgensen commented on GitHub (Mar 22, 2024):

hmm.. have some issues to day with dev images.. i cant see whats wrong there but when I try main it works
but i have deleted my storage for openchat and now i have to set everything up again. I add claude3 opus without seting enything othere then the model name and api key

so must i set max_tokens: 4096 when i use claude-3 models? if so then it must be in a readme somewhere.

@bjornjorgensen commented on GitHub (Mar 22, 2024): hmm.. have some issues to day with dev images.. i cant see whats wrong there but when I try main it works but i have deleted my storage for openchat and now i have to set everything up again. I add claude3 opus without seting enything othere then the model name and api key  so must i set max_tokens: 4096 when i use claude-3 models? if so then it must be in a readme somewhere.

GiteaMirror commented 2025-11-11 14:21:26 -06:00

Author

Owner

Copy Link

Copy Source

@justinh-rahb commented on GitHub (Mar 22, 2024):

@bjornjorgensen I haven't migrated this to the docs site yet, there's a thread:

• Tried the manual install on Ubuntu 20.04 wsl - Chroma requires sqlite3 (#1038)

max_tokens must be 4096 to get the most out of Claude API, as noted there.

@justinh-rahb commented on GitHub (Mar 22, 2024): @bjornjorgensen I haven't migrated this to the docs site yet, there's a thread: - #1038 `max_tokens` must be `4096` to get the most out of Claude API, as noted there.

GiteaMirror referenced this issue 2025-11-11 17:17:52 -06:00

[PR #439] [MERGED] feat: better rag context management #7133

GiteaMirror referenced this issue 2026-04-20 02:54:24 -05:00

[PR #439] [MERGED] feat: better rag context management #20337

GiteaMirror referenced this issue 2026-04-25 10:04:20 -05:00

[PR #439] [MERGED] feat: better rag context management #35967

GiteaMirror referenced this issue 2026-04-29 17:29:44 -05:00

[PR #439] [MERGED] feat: better rag context management #43385

GiteaMirror referenced this issue 2026-05-06 01:04:56 -05:00

[PR #439] [MERGED] feat: better rag context management #59193

GiteaMirror referenced this issue 2026-05-13 08:10:39 -05:00

[PR #439] [MERGED] feat: better rag context management #75210

GiteaMirror referenced this issue 2026-05-15 16:42:15 -05:00

[PR #439] [MERGED] feat: better rag context management #91375

GiteaMirror referenced this issue 2026-05-18 06:32:52 -05:00

[PR #439] [MERGED] feat: better rag context management #107648

GiteaMirror referenced this issue 2026-05-21 03:39:38 -05:00

[PR #439] [MERGED] feat: better rag context management #123994

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

dev

v0.11.0

v0.10.2

v0.10.1

v0.10.0

v0.9.6

v0.9.5

v0.9.4

v0.9.3

v0.9.2

v0.9.1

v0.9.0

v0.8.12

v0.8.11

v0.8.10

v0.8.9

v0.8.8

v0.8.7

v0.8.6

v0.8.5

v0.8.4

v0.8.3

v0.8.2

v0.8.1

v0.8.0

v0.7.2

v0.7.1

v0.7.0

v0.6.43

v0.6.42

v0.6.41

v0.6.40

v0.6.39

v0.6.38

v0.6.37

v0.6.36

v0.6.35

v0.6.34

v0.6.33

v0.6.32

v0.6.31

v0.6.30

v0.6.29

v0.6.28

v0.6.27

v0.6.26

v0.6.25

v0.6.24

v0.6.23

v0.6.22

v0.6.21

v0.6.20

v0.6.19

v0.6.18

v0.6.17

v0.6.16

v0.6.15

v0.6.14

v0.6.13

v0.6.12

v0.6.11

v0.6.10

v0.6.9

v0.6.8

v0.6.7

v0.6.6

v0.6.5

v0.6.4

v0.6.3

v0.6.2

v0.6.1

v0.6.0

v0.5.20

v0.5.19

v0.5.18

v0.5.17

v0.5.16

v0.5.15

v0.5.14

v0.5.13

v0.5.12

v0.5.11

v0.5.10

v0.5.9

v0.5.8

v0.5.7

v0.5.6

v0.5.5

v0.5.4

v0.5.3

v0.5.2

v0.5.1

v0.5.0

v0.4.8

v0.4.7

v0.4.6

v0.4.5

v0.4.4

v0.4.3

v0.4.2

v0.4.1

v0.4.0

v0.3.35

v0.3.34

v0.3.33

v0.3.32

v0.3.31

v0.3.30

v0.3.29

v0.3.28

v0.3.27

v0.3.26

v0.3.25

v0.3.24

v0.3.23

v0.3.22

v0.3.21

v0.3.20

v0.3.19

v0.3.18

v0.3.17

v0.3.16

v0.3.15

v0.3.14

v0.3.13

v0.3.12

v0.3.11

v0.3.10

v0.3.9

v0.3.8

v0.3.7

v0.3.6

v0.3.5

v0.3.4

v0.3.3

v0.3.2

v0.3.1

v0.3.0

v0.2.5

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.125

v0.1.124

v0.1.123

v0.1.122

v0.1.121

v0.1.120

v0.1.119

v0.1.118

v0.1.117

v0.1.116

v0.1.115

v0.1.114

v0.1.113

v0.1.112

v0.1.111

v0.1.110

v0.1.109

v0.1.108

v0.1.107

v0.1.106

v0.1.105

v0.1.104

v0.1.103

v0.1.102

Labels

Clear labels

bug

confirmed

confirmed issue

core

documentation

enhancement

good first issue

help wanted

non-core

pull-request

Mirrored from GitHub Pull Request

python

question

testing wanted

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

GiteaMirror ninjasurge (Kaehlen Rog)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#439