mirror of
https://github.com/open-webui/open-webui.git
synced 2026-07-11 10:34:13 -05:00
[GH-ISSUE #13547] issue: S3 Storage: Compatibility Issues with Cloudflare R2 Leading to Upload Failures (Tagging & Accelerate) #103940
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @MeiTetsuH on GitHub (May 6, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/13547
Check Existing Issues
Installation Method
Git Clone
Open WebUI Version
v0.6.6
Ollama Version (if applicable)
No response
Operating System
windows11
Browser (if applicable)
No response
Confirmation
README.md.Expected Behavior
When configuring Open WebUI to use Cloudflare R2 as an S3-compatible storage backend, users encounter several issues that prevent successful file (e.g., image) uploads:
PutObjectTagging Not Implemented:
Error: botocore.exceptions.ClientError: An error occurred (NotImplemented) when calling the PutObjectTagging operation: PutObjectTagging not implemented
Cause: Open WebUI attempts to set object tags using a separate PutObjectTagging call after the initial upload_file (which uses PutObject internally). Cloudflare R2 does not support the PutObjectTagging API operation.
x-amz-tagging Header Not Implemented with upload_file ExtraArgs:
Error (after attempting to fix 1 by passing tags via ExtraArgs): botocore.exceptions.ClientError: An error occurred (NotImplemented) when calling the PutObject operation: Header 'x-amz-tagging' with value '...' not implemented
Cause: When trying to set tags during the PutObject operation (via s3_client.upload_file's ExtraArgs={'Tagging': 'url_encoded_string'}), Cloudflare R2 reports that the x-amz-tagging header itself is not implemented in this context, or the format provided by boto3 is not accepted. This occurs even though R2 documentation suggests support for tagging via this header during PutObject.
Incorrect Type for S3_USE_ACCELERATE_ENDPOINT (Initial User Configuration Issue):
Error: botocore.exceptions.EndpointResolutionError: Value (Accelerate) is the wrong type. Must be <class 'bool'>.
Cause: The S3_USE_ACCELERATE_ENDPOINT environment variable, if set to a string like "False" or "True", was being passed directly to boto3 without conversion to a Python boolean, causing a type error. (While this was a configuration/initialization issue, it's relevant to R2 as this feature should be False for R2).
Actual Behavior
roubleshooting & Solution Path Taken:
S3_USE_ACCELERATE_ENDPOINT Type Error:
Solution: Ensured the environment variable string is converted to a Python boolean (e.g., (os.getenv("S3_USE_ACCELERATE_ENDPOINT", "false").lower() == "true")). For R2, this should always resolve to False.
PutObjectTagging Not Implemented:
Attempted Fix 1: Moved tagging logic to be part of the upload_file call by preparing tags as a URL-encoded string and passing them via ExtraArgs={'Tagging': '...'}.
Result of Attempted Fix 1: Led to the x-amz-tagging header error (Problem 2).
x-amz-tagging Header Not Implemented:
Current Workaround: Modified S3StorageProvider.upload_file to completely remove the ExtraArgs={'Tagging': ...} part from the self.s3_client.upload_file() call when targeting R2 (or any provider exhibiting this behavior). This allows files to be uploaded successfully but without any S3 object tags.
Steps to Reproduce
Steps to Reproduce:
Setup Cloudflare R2 Bucket:
GetObject,PutObject,DeleteObject,ListBucket).Configure Open WebUI Environment Variables for R2:
Set the following environment variables for Open WebUI, replacing placeholders with your actual R2 details:
Start Open WebUI.
Attempt Image Generation / File Upload:
Observe Errors in Logs:
S3_USE_ACCELERATE_ENDPOINTis misconfigured as a string by user):botocore.exceptions.EndpointResolutionError: Value (Accelerate) is the wrong type. Must be <class 'bool'>.botocore.exceptions.ClientError: An error occurred (NotImplemented) when calling the PutObjectTagging operation: PutObjectTagging not implementedThis error occurs because
S3StorageProvider.upload_filecallss3_client.put_object_tagging()afters3_client.upload_file().upload_fileExtraArgs(as a common first fix attempt for the above):botocore.exceptions.ClientError: An error occurred (NotImplemented) when calling the PutObject operation: Header 'x-amz-tagging' with value '...' not implementedThis occurs if the code is modified to pass
ExtraArgs={'Tagging': 'url_encoded_string'}tos3_client.upload_file().Expected Behavior:
Files (images, documents) should be successfully uploaded to the Cloudflare R2 bucket, ideally with any relevant metadata tags applied if R2 supports tagging via
PutObjectheaders.Actual Behavior:
File uploads fail due to S3 API compatibility issues between Open WebUI's S3 client usage (specifically for tagging) and Cloudflare R2's implementation of the S3 API. The upload only succeeds if tagging attempts are completely removed from the
S3StorageProvider.upload_filemethod.Logs & Screenshots
`2025-05-06 14:27:22.553 | ERROR | open_webui.routers.files:upload_file:177 - Error uploading file to S3: An error occurred (NotImplemented) when calling the PutObjectTagging operation: PutObjectTagging not implemented - {}
Traceback (most recent call last):
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\open_webui\storage\provider.py", line 147, in upload_file
self.s3_client.put_object_tagging(
│ │ └ <function ClientCreator._create_api_method.._api_call at 0x00000209EFBDC540>
│ └ <botocore.client.S3 object at 0x00000209EFBE8710>
└ <open_webui.storage.provider.S3StorageProvider object at 0x00000209EF2B11D0>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\botocore\client.py", line 569, in _api_call
return self._make_api_call(operation_name, kwargs)
│ │ │ └ {'Bucket': 'openwebui', 'Key': 'd2d105d1-e272-4a6d-9112-ede574b89654_generated-image.jpg', 'Tagging': {'TagSet': [{'Key': 'Op...
│ │ └ 'PutObjectTagging'
│ └ <function BaseClient._make_api_call at 0x00000209EDAC54E0>
└ <botocore.client.S3 object at 0x00000209EFBE8710>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\botocore\client.py", line 1023, in _make_api_call
raise error_class(parsed_response, operation_name)
│ │ └ 'PutObjectTagging'
│ └ {'Error': {'Code': 'NotImplemented', 'Message': 'PutObjectTagging not implemented'}, 'ResponseMetadata': {'HTTPStatusCode': 5...
└ <class 'botocore.exceptions.ClientError'>
botocore.exceptions.ClientError: An error occurred (NotImplemented) when calling the PutObjectTagging operation: PutObjectTagging not implemented
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in run_code
File "C:\ProgramData\anaconda3\envs\open-webui\Scripts\open-webui.exe_main.py", line 7, in
sys.exit(app())
│ │ └ <typer.main.Typer object at 0x00000209D1232090>
│ └
└ <module 'sys' (built-in)>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\typer\main.py", line 323, in call
return get_command(self)(*args, **kwargs)
│ │ │ └ {}
│ │ └ ()
│ └ <typer.main.Typer object at 0x00000209D1232090>
└ <function get_command at 0x00000209D3A2E7A0>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\click\core.py", line 1161, in call
return self.main(*args, **kwargs)
│ │ │ └ {}
│ │ └ ()
│ └ <function TyperGroup.main at 0x00000209D3A1AE80>
└
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\typer\core.py", line 743, in main
return _main(
└ <function _main at 0x00000209D3A1A0C0>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\typer\core.py", line 198, in _main
rv = self.invoke(ctx)
│ │ └ <click.core.Context object at 0x00000209D460C4D0>
│ └ <function MultiCommand.invoke at 0x00000209D170CFE0>
└
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\click\core.py", line 1697, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
│ │ │ │ └ <click.core.Context object at 0x00000209D12D7A50>
│ │ │ └ <function Command.invoke at 0x00000209D170C9A0>
│ │ └
│ └ <click.core.Context object at 0x00000209D12D7A50>
└ <function MultiCommand.invoke.._process_result at 0x00000209D45F34C0>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\click\core.py", line 1443, in invoke
return ctx.invoke(self.callback, **ctx.params)
│ │ │ │ │ └ {'host': '0.0.0.0', 'port': 8080}
│ │ │ │ └ <click.core.Context object at 0x00000209D12D7A50>
│ │ │ └ <function serve at 0x00000209D45F3380>
│ │ └
│ └ <function Context.invoke at 0x00000209D16F72E0>
└ <click.core.Context object at 0x00000209D12D7A50>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\click\core.py", line 788, in invoke
return _callback(*args, **kwargs)
│ └ {'host': '0.0.0.0', 'port': 8080}
└ ()
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\typer\main.py", line 698, in wrapper
return callback(**use_params)
│ └ {'host': '0.0.0.0', 'port': 8080}
└ <function serve at 0x00000209D44C07C0>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\open_webui_init.py", line 78, in serve
uvicorn.run(
│ └ <function run at 0x00000209D4487CE0>
└ <module 'uvicorn' from 'C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\uvicorn\init.py'>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\uvicorn\main.py", line 579, in run
server.run()
│ └ <function Server.run at 0x00000209D44C0040>
└ <uvicorn.server.Server object at 0x0000020986AB1C90>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\uvicorn\server.py", line 66, in run
return asyncio.run(self.serve(sockets=sockets))
│ │ │ │ └ None
│ │ │ └ <function Server.serve at 0x00000209D44C00E0>
│ │ └ <uvicorn.server.Server object at 0x0000020986AB1C90>
│ └ <function run at 0x00000209D3B53CE0>
└ <module 'asyncio' from 'C:\ProgramData\anaconda3\envs\open-webui\Lib\asyncio\init.py'>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\asyncio\runners.py", line 190, in run
return runner.run(main)
│ │ └ <coroutine object Server.serve at 0x00000209D3C12E30>
│ └ <function Runner.run at 0x00000209D3B7CCC0>
└ <asyncio.runners.Runner object at 0x0000020988690F10>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\asyncio\runners.py", line 118, in run
return self._loop.run_until_complete(task)
│ │ │ └ <Task pending name='Task-1' coro=<Server.serve() running at C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\uvicor...
│ │ └ <function BaseEventLoop.run_until_complete at 0x00000209D3B7A8E0>
│ └
└ <asyncio.runners.Runner object at 0x0000020988690F10>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\asyncio\base_events.py", line 641, in run_until_complete
self.run_forever()
│ └ <function ProactorEventLoop.run_forever at 0x00000209D3BEFB00>
└
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\asyncio\windows_events.py", line 321, in run_forever
super().run_forever()
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\asyncio\base_events.py", line 608, in run_forever
self._run_once()
│ └ <function BaseEventLoop._run_once at 0x00000209D3B7C680>
└
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\asyncio\base_events.py", line 1936, in _run_once
handle._run()
│ └ <function Handle._run at 0x00000209D3AFD4E0>
└ <Handle Task.task_wakeup(<Future finis...sponse [200]>>)>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\asyncio\events.py", line 84, in _run
self._context.run(self._callback, *self._args)
│ │ │ │ │ └ <member '_args' of 'Handle' objects>
│ │ │ │ └ <Handle Task.task_wakeup(<Future finis...sponse [200]>>)>
│ │ │ └ <member '_callback' of 'Handle' objects>
│ │ └ <Handle Task.task_wakeup(<Future finis...sponse [200]>>)>
│ └ <member '_context' of 'Handle' objects>
└ <Handle Task.task_wakeup(<Future finis...sponse [200]>>)>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\starlette\middleware\base.py", line 141, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
│ │ │ │ └ <function BaseHTTPMiddleware.call..call_next..send_no_error at 0x000002098895EF20>
│ │ │ └ <function BaseHTTPMiddleware.call..call_next..receive_or_disconnect at 0x000002098895EDE0>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8080), 'cl...
│ └ <starlette.middleware.exceptions.ExceptionMiddleware object at 0x0000020986F120D0>
└ <open_webui.main.RedirectMiddleware object at 0x0000020986F112D0>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\starlette\middleware\exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
│ │ │ │ │ │ └ <function BaseHTTPMiddleware.call..call_next..send_no_error at 0x000002098895EF20>
│ │ │ │ │ └ <function BaseHTTPMiddleware.call..call_next..receive_or_disconnect at 0x000002098895EDE0>
│ │ │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8080), 'cl...
│ │ │ └ <starlette.requests.Request object at 0x0000020988936A50>
│ │ └ <fastapi.routing.APIRouter object at 0x00000209F2FED390>
│ └ <starlette.middleware.exceptions.ExceptionMiddleware object at 0x0000020986F120D0>
└ <function wrap_app_handling_exceptions at 0x00000209D6ED4540>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\starlette_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
│ │ │ └ <function wrap_app_handling_exceptions..wrapped_app..sender at 0x000002098895F2E0>
│ │ └ <function BaseHTTPMiddleware.call..call_next..receive_or_disconnect at 0x000002098895EDE0>
│ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8080), 'cl...
└ <fastapi.routing.APIRouter object at 0x00000209F2FED390>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\starlette\routing.py", line 715, in call
await self.middleware_stack(scope, receive, send)
│ │ │ │ └ <function wrap_app_handling_exceptions..wrapped_app..sender at 0x000002098895F2E0>
│ │ │ └ <function BaseHTTPMiddleware.call..call_next..receive_or_disconnect at 0x000002098895EDE0>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8080), 'cl...
│ └ <bound method Router.app of <fastapi.routing.APIRouter object at 0x00000209F2FED390>>
└ <fastapi.routing.APIRouter object at 0x00000209F2FED390>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\starlette\routing.py", line 735, in app
await route.handle(scope, receive, send)
│ │ │ │ └ <function wrap_app_handling_exceptions..wrapped_app..sender at 0x000002098895F2E0>
│ │ │ └ <function BaseHTTPMiddleware.call..call_next..receive_or_disconnect at 0x000002098895EDE0>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8080), 'cl...
│ └ <function Route.handle at 0x00000209D6ED5BC0>
└ APIRoute(path='/api/chat/completions', name='chat_completion', methods=['POST'])
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\starlette\routing.py", line 288, in handle
await self.app(scope, receive, send)
│ │ │ │ └ <function wrap_app_handling_exceptions..wrapped_app..sender at 0x000002098895F2E0>
│ │ │ └ <function BaseHTTPMiddleware.call..call_next..receive_or_disconnect at 0x000002098895EDE0>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8080), 'cl...
│ └ <function request_response..app at 0x0000020986E1CEA0>
└ APIRoute(path='/api/chat/completions', name='chat_completion', methods=['POST'])
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\starlette\routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
│ │ │ │ │ └ <function wrap_app_handling_exceptions..wrapped_app..sender at 0x000002098895F2E0>
│ │ │ │ └ <function BaseHTTPMiddleware.call..call_next..receive_or_disconnect at 0x000002098895EDE0>
│ │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8080), 'cl...
│ │ └ <starlette.requests.Request object at 0x000002098896F750>
│ └ <function request_response..app..app at 0x000002098895D800>
└ <function wrap_app_handling_exceptions at 0x00000209D6ED4540>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\starlette_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
│ │ │ └ <function wrap_app_handling_exceptions..wrapped_app..sender at 0x000002098895D760>
│ │ └ <function BaseHTTPMiddleware.call..call_next..receive_or_disconnect at 0x000002098895EDE0>
│ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8080), 'cl...
└ <function request_response..app..app at 0x000002098895D800>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\starlette\routing.py", line 73, in app
response = await f(request)
│ └ <starlette.requests.Request object at 0x000002098896F750>
└ <function get_request_handler..app at 0x0000020986E1D1C0>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\fastapi\routing.py", line 301, in app
raw_response = await run_endpoint_function(
└ <function run_endpoint_function at 0x00000209D6ED79C0>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\fastapi\routing.py", line 212, in run_endpoint_function
return await dependant.call(**values)
│ │ └ {'user': UserModel(id='3c3016fd-7f6e-49cc-9bff-d5e36772d982', name='Akira', email='0o3p4a7dm@mozmail.com', role='admin', prof...
│ └ <function chat_completion at 0x000002098870E7A0>
└ Dependant(path_params=[], query_params=[], header_params=[], cookie_params=[], body_params=[ModelField(field_info=Body(Pydant...
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\open_webui\main.py", line 1178, in chat_completion
form_data, metadata, events = await process_chat_payload(
│ │ └ <function process_chat_payload at 0x00000209F2E88EA0>
│ └ {'user_id': '3c3016fd-7f6e-49cc-9bff-d5e36772d982', 'chat_id': '32599c7d-1d5f-4f97-9a8d-a5560e44653b', 'message_id': '0ccb246...
└ {'stream': True, 'model': 'chat-hmz', 'messages': [{'role': 'user', 'content': '生成一张汽车图片'}], 'metadata': {'user_id': '3c3016f...
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\open_webui\utils\middleware.py", line 805, in process_chat_payload
form_data = await chat_image_generation_handler(
└ <function chat_image_generation_handler at 0x00000209F2E88CC0>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\open_webui\utils\middleware.py", line 528, in chat_image_generation_handler
images = await image_generations(
└ <function image_generations at 0x00000209F0CF5940>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\open_webui\routers\images.py", line 525, in image_generations
url = upload_image(request, data, image_data, content_type, user)
│ │ │ │ │ └ UserModel(id='3c3016fd-7f6e-49cc-9bff-d5e36772d982', name='Akira', email='0o3p4a7dm@mozmail.com', role='admin', profile_image...
│ │ │ │ └ 'image/jpeg'
│ │ │ └ b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x02\x00\x00\x01\x00\x01\x00\x00\xff\xc0\x00\x11\x08\x03\x00\x04\x00\x03\x01\x11\x00\x0...
│ │ └ {'model': 'grok-2-image-1212', 'prompt': "Generate an image of a car, specifically a standard sedan. The car has a sleek, elo...
│ └ <starlette.requests.Request object at 0x000002098896F750>
└ <function upload_image at 0x00000209F0CF5800>
File "C:\ProgramData\anaconda3\envs\open-webui\Lib\site-packages\open_webui\routers\images.py", line 462, in upload_image
file_item = upload_file(request, file, user, file_metadata=image_metadata)
│ │ │ │ └ {'model': 'grok-2-image-1212', 'prompt': "Generate an image of a car, specifically a standard sedan. The car has a sleek, elo...
│ │ │ └ UserModel(id='3c3016fd-7f6e-49cc-9bff-d5e36772d982', name='Akira', email='0o3p4a7dm@mozmail.com', role='admin', profile_image...
│ │ └ UploadFile(filename='generated-image.jpg', size=None, headers={'content-type': 'image/jpeg'})
│ └ <starlette.requests.Request object at 0x000002098896F750>
└ <function upload_file at 0x00000209ED38E160>
Additional Information
Graceful Handling for Tagging on R2/Incompatible S3 Providers:
Option A (Preferred for R2): Investigate and implement setting tags directly during the s3_client.put_object() call (not upload_file's ExtraArgs if it remains problematic, but by potentially using put_object for smaller files or if R2 supports tagging headers with CreateMultipartUpload). R2 documentation indicates x-amz-tagging is supported on PutObject. The discrepancy might be in how boto3's upload_file high-level API translates ExtraArgs for R2.
Option B (General Fallback): If Option A is complex or unreliable across all S3-compatibles, provide a configuration option (e.g., S3_DISABLE_TAGGING=true or autodetect based on known provider issues) to disable object tagging attempts. Log a warning if tagging is disabled.
Option C (Try-Catch): Wrap the tagging attempt (whether PutObjectTagging or x-amz-tagging via ExtraArgs) in a try-except block. If a NotImplemented error (or similar indicating lack of support) occurs, log a warning and proceed without tags, rather than failing the entire upload.
@tjbck commented on GitHub (May 6, 2025):
PR Welcome.