mirror of
https://github.com/open-webui/open-webui.git
synced 2026-08-24 14:23:59 -05:00
Classic298andGitHub
091c44c621
Streamed responses are scanned for reasoning and code interpreter tags. To work out where the last complete tag ended, the scanner searched backwards from the start of the accumulated text on every chunk, once per tag set. Ordinary prose contains no angle bracket, so that search never stopped early and read the entire response back every time. The cost grows with the square of the response length, and this scanning is on unless a model turns it off. The two positions are now carried forward as the text grows, so each chunk only scans the characters it added. Measured on CPython 3.12, a 270 KB response streamed in 27000 chunks: | response text | before | after | |---|---|---| | no newlines | 7690 ms | 40.6 ms | | with newlines | 5695 ms | 41.7 ms | The carried positions match a full rescan at every step of 36282 randomized replays, covering text with no markers, newlines only, dense markers, real tags and truncation part way through.