I have searched the existing issues and discussions.
I am using the latest version of Open WebUI.
Installation Method
Docker
Open WebUI Version
v0.6.22
Ollama Version (if applicable)
No response
Operating System
Debian 12
Browser (if applicable)
No response
Confirmation
I have read and followed all instructions in README.md.
I am using the latest version of both Open WebUI and Ollama.
I have included the browser console logs.
I have included the Docker container logs.
I have provided every relevant configuration, setting, and environment variable used in my setup.
I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
Start with the initial platform/version/OS and dependencies used,
Specify exact install/launch/configure commands,
List URLs visited, user input (incl. example values/emails/passwords if needed),
Describe all options and toggles enabled or changed,
Include any files or environmental changes,
Identify the expected and actual result at each stage,
Ensure any reasonably skilled user can follow and hit the same issue.
Expected Behavior
Documents should upload sucessfuly
Actual Behavior
I get an error if the chunk size is to big (it works with 300 30 overlap , but not with 400 40 overlap)
Steps to Reproduce
Deploy portainer
Deploy OpenWebUI with this stack configuration on portainer :
2025-08-12 13:35:14.822 | ERROR | open_webui.routers.files:upload_file:192 - 400: An error occurred (ValidationException) when calling the PutVectors operation: Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2048 bytes
Traceback (most recent call last):
File "/app/backend/open_webui/routers/retrieval.py", line 1509, in process_file
raise e
File "/app/backend/open_webui/routers/retrieval.py", line 1481, in process_file
result = save_docs_to_vector_db(
└ <function save_docs_to_vector_db at 0x7f43410ae8e0>
File "/app/backend/open_webui/routers/retrieval.py", line 1323, in save_docs_to_vector_db
raise e
File "/app/backend/open_webui/routers/retrieval.py", line 1315, in save_docs_to_vector_db
VECTOR_DB_CLIENT.insert(
│ └ <function S3VectorClient.insert at 0x7f43446cc680>
└ <open_webui.retrieval.vector.dbs.s3vector.S3VectorClient object at 0x7f4344af9250>
File "/app/backend/open_webui/retrieval/vector/dbs/s3vector.py", line 201, in insert
self.client.put_vectors(
│ │ └ <function ClientCreator._create_api_method.<locals>._api_call at 0x7f43446ce980>
│ └ <botocore.client.S3Vectors object at 0x7f4344020450>
└ <open_webui.retrieval.vector.dbs.s3vector.S3VectorClient object at 0x7f4344af9250>
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 602, in _api_call
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 1078, in _make_api_call
raise error_class(parsed_response, operation_name)
│ │ └ 'PutVectors'
│ └ {'Error': {'Message': "Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2...
└ <class 'botocore.errorfactory.ValidationException'>
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the PutVectors operation: Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2048 bytes
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
│ └ <function Thread._bootstrap_inner at 0x7f437a4749a0>
└ <WorkerThread(AnyIO worker thread, started 139926653867712)>
File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
self.run()
│ └ <function WorkerThread.run at 0x7f433bffb1a0>
└ <WorkerThread(AnyIO worker thread, started 139926653867712)>
File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run
result = context.run(func, *args)
│ │ │ └ ()
│ │ └ functools.partial(<function upload_file at 0x7f4343e98180>, user=UserModel(id='ec30dbab-f0cb-4e08-a1b1-550da2623356', name='C...
│ └ <method 'run' of '_contextvars.Context' objects>
└ <_contextvars.Context object at 0x7f433abb6ec0>
File "/app/backend/open_webui/routers/files.py", line 183, in upload_file
fastapi.exceptions.HTTPException: 400: An error occurred (ValidationException) when calling the PutVectors operation: Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2048 bytes
It might just be because of S3 specifications and limitations but if i can be enhanced then it's worth looking at i guess
Originally created by @Cqban on GitHub (Aug 12, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/16527
### Check Existing Issues
- [x] I have searched the existing issues and discussions.
- [x] I am using the latest version of Open WebUI.
### Installation Method
Docker
### Open WebUI Version
v0.6.22
### Ollama Version (if applicable)
_No response_
### Operating System
Debian 12
### Browser (if applicable)
_No response_
### Confirmation
- [x] I have read and followed all instructions in `README.md`.
- [x] I am using the latest version of **both** Open WebUI and Ollama.
- [x] I have included the browser console logs.
- [x] I have included the Docker container logs.
- [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.**
- [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
- [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps:
- Start with the initial platform/version/OS and dependencies used,
- Specify exact install/launch/configure commands,
- List URLs visited, user input (incl. example values/emails/passwords if needed),
- Describe all options and toggles enabled or changed,
- Include any files or environmental changes,
- Identify the expected and actual result at each stage,
- Ensure any reasonably skilled user can follow and hit the same issue.
### Expected Behavior
Documents should upload sucessfuly
### Actual Behavior
I get an error if the chunk size is to big (it works with 300 30 overlap , but not with 400 40 overlap)
### Steps to Reproduce
1. Deploy portainer
2. Deploy OpenWebUI with this stack configuration on portainer :
name: open-webui
services:
open-webui:
container_name: open-webui
image: ghcr.io/open-webui/open-webui:main
ports:
- 3000:8080
volumes:
- open-webui:/app/backend/data
restart: unless-stopped
environment:
VECTOR_DB: s3vector
S3_VECTOR_BUCKET_NAME: Hidden
AWS_ACCESS_KEY_ID: Hidden
AWS_SECRET_ACCESS_KEY: Hidden
AWS_REGION: Hidden
S3_VECTOR_REGION: Hidden
tika:
container_name: tika
image: apache/tika:latest-full
ports:
- 9998:9998
restart: unless-stopped
volumes:
open-webui:
external: true
name: open-webui
### Logs & Screenshots
2025-08-12 13:35:14.822 | ERROR | open_webui.routers.files:upload_file:192 - 400: An error occurred (ValidationException) when calling the PutVectors operation: Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2048 bytes
Traceback (most recent call last):
File "/app/backend/open_webui/routers/retrieval.py", line 1509, in process_file
raise e
File "/app/backend/open_webui/routers/retrieval.py", line 1481, in process_file
result = save_docs_to_vector_db(
└ <function save_docs_to_vector_db at 0x7f43410ae8e0>
File "/app/backend/open_webui/routers/retrieval.py", line 1323, in save_docs_to_vector_db
raise e
File "/app/backend/open_webui/routers/retrieval.py", line 1315, in save_docs_to_vector_db
VECTOR_DB_CLIENT.insert(
│ └ <function S3VectorClient.insert at 0x7f43446cc680>
└ <open_webui.retrieval.vector.dbs.s3vector.S3VectorClient object at 0x7f4344af9250>
File "/app/backend/open_webui/retrieval/vector/dbs/s3vector.py", line 201, in insert
self.client.put_vectors(
│ │ └ <function ClientCreator._create_api_method.<locals>._api_call at 0x7f43446ce980>
│ └ <botocore.client.S3Vectors object at 0x7f4344020450>
└ <open_webui.retrieval.vector.dbs.s3vector.S3VectorClient object at 0x7f4344af9250>
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 602, in _api_call
return self._make_api_call(operation_name, kwargs)
│ │ │ └ {'vectorBucketName': 'openwebui', 'indexName': 'file-041ac3db-5e5a-4415-9163-8f7bd7a4a66d', 'vectors': [{'key': '5f2fbbd9-d7d...
│ │ └ 'PutVectors'
│ └ <function BaseClient._make_api_call at 0x7f434481c900>
└ <botocore.client.S3Vectors object at 0x7f4344020450>
File "/usr/local/lib/python3.11/site-packages/botocore/context.py", line 123, in wrapper
return func(*args, **kwargs)
│ │ └ {}
│ └ (<botocore.client.S3Vectors object at 0x7f4344020450>, 'PutVectors', {'vectorBucketName': 'openwebui', 'indexName': 'file-041...
└ <function BaseClient._make_api_call at 0x7f434481c860>
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 1078, in _make_api_call
raise error_class(parsed_response, operation_name)
│ │ └ 'PutVectors'
│ └ {'Error': {'Message': "Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2...
└ <class 'botocore.errorfactory.ValidationException'>
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the PutVectors operation: Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2048 bytes
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
│ └ <function Thread._bootstrap_inner at 0x7f437a4749a0>
└ <WorkerThread(AnyIO worker thread, started 139926653867712)>
File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
self.run()
│ └ <function WorkerThread.run at 0x7f433bffb1a0>
└ <WorkerThread(AnyIO worker thread, started 139926653867712)>
File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run
result = context.run(func, *args)
│ │ │ └ ()
│ │ └ functools.partial(<function upload_file at 0x7f4343e98180>, user=UserModel(id='ec30dbab-f0cb-4e08-a1b1-550da2623356', name='C...
│ └ <method 'run' of '_contextvars.Context' objects>
└ <_contextvars.Context object at 0x7f433abb6ec0>
> File "/app/backend/open_webui/routers/files.py", line 183, in upload_file
process_file(request, ProcessFileForm(file_id=id), user=user)
│ │ │ │ └ UserModel(id='ec30dbab-f0cb-4e08-a1b1-550da2623356', name='Cqban', email='cqban@cqban.xyz', role='admin', profile_image_url='...
│ │ │ └ '041ac3db-5e5a-4415-9163-8f7bd7a4a66d'
│ │ └ <class 'open_webui.routers.retrieval.ProcessFileForm'>
│ └ <starlette.requests.Request object at 0x7f433ab85490>
└ <function process_file at 0x7f43410af600>
File "/app/backend/open_webui/routers/retrieval.py", line 1526, in process_file
raise HTTPException(
└ <class 'fastapi.exceptions.HTTPException'>
fastapi.exceptions.HTTPException: 400: An error occurred (ValidationException) when calling the PutVectors operation: Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2048 bytes
2025-08-12 13:35:14.932 | ERROR | open_webui.routers.files:upload_file:193 - Error processing file: 041ac3db-5e5a-4415-9163-8f7bd7a4a66d
### Additional Information
It might just be because of S3 specifications and limitations but if i can be enhanced then it's worth looking at i guess
GiteaMirror
added the bug label 2026-04-25 07:23:53 -05:00
Thanks @Cqban, I'll take a closer look at this. I've never used Tika before, do you happen to know if it adds/ extends existing metadata on vectors? I can do some testing and research to figure it out if you're not sure.
<!-- gh-comment-id:3179551237 -->
@westbrook-ai commented on GitHub (Aug 12, 2025):
Thanks @Cqban, I'll take a closer look at this. I've never used Tika before, do you happen to know if it adds/ extends existing metadata on vectors? I can do some testing and research to figure it out if you're not sure.
I never ran into that issue during my testing using the Open WebUI defaults for chunk size and chunk overlap, but I'll still take a closer look as soon as I can to try to understand what the total size of the metadata per vector was for me using those defaults.
<!-- gh-comment-id:3181911344 -->
@westbrook-ai commented on GitHub (Aug 13, 2025):
Confirmed that the filterable metadata per vector is up to 2KB per the S3 vectors limitations docs: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-limitations.html
I never ran into that issue during my testing using the Open WebUI defaults for [chunk size](https://docs.openwebui.com/getting-started/env-configuration/#chunk_size) and [chunk overlap](https://docs.openwebui.com/getting-started/env-configuration/#chunk_overlap), but I'll still take a closer look as soon as I can to try to understand what the total size of the metadata per vector was for me using those defaults.
@joshrenshaw12 commented on GitHub (Aug 13, 2025):
I've seen similar errors like below:
2025-08-13 08:31:56.497 | ERROR | open_webui.routers.files:upload_file:192 - 400: An error occurred (ValidationException) when calling the PutVectors operation: Request body exceeds max allowed size
This aligns with functionality i have seen in testing where a smaller file with less vectors uploads fine, but a larger one with more vectors does not.
Certainly a different issue to that mentioned above but shows the need for some batching logic applied to backend/open_webui/retrieval/vector/dbs/s3vector.py to match limitations of the AWS API.
<!-- gh-comment-id:3182884663 -->
@joshrenshaw12 commented on GitHub (Aug 13, 2025):
I've seen similar errors like below:
```
2025-08-13 08:31:56.497 | ERROR | open_webui.routers.files:upload_file:192 - 400: An error occurred (ValidationException) when calling the PutVectors operation: Request body exceeds max allowed size
```
From my research this is due to the number of vectors being sent in the single `PutVectors` operation
https://www.perplexity.ai/search/botocore-errorfactory-validati-ZmuvRNkxQeal7FvCWXHTNQ
From docs mentioned above, we know API is limited to 500 vectors per PutVectors API call.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_PutVectors.html
This aligns with functionality i have seen in testing where a smaller file with less vectors uploads fine, but a larger one with more vectors does not.
Certainly a different issue to that mentioned above but shows the need for some batching logic applied to `backend/open_webui/retrieval/vector/dbs/s3vector.py` to match limitations of the AWS API.
@joshrenshaw12 your issue popped up when you tried to upload 500+ files to a collection at once right? I think I have a fix for that now, I can ping you when it ends up in dev if so.
<!-- gh-comment-id:3204081777 -->
@westbrook-ai commented on GitHub (Aug 20, 2025):
@joshrenshaw12 your issue popped up when you tried to upload 500+ files to a collection at once right? I think I have a fix for that now, I can ping you when it ends up in `dev` if so.
<img width="393" height="87" alt="Image" src="https://github.com/user-attachments/assets/9e3f0685-fff3-40b0-8827-eab9f7fc26b8" />
@joshrenshaw12 commented on GitHub (Aug 20, 2025):
@joshrenshaw12 your issue popped up when you tried to upload 500+ files to a collection at once right? I think I have a fix for that now, I can ping you when it ends up in dev if so.
not 500+ individual files, just a single large file that was chunked into 500+ vector chunks that all were trying to get uploaded at once
<!-- gh-comment-id:3204868075 -->
@joshrenshaw12 commented on GitHub (Aug 20, 2025):
> [@joshrenshaw12](https://github.com/joshrenshaw12) your issue popped up when you tried to upload 500+ files to a collection at once right? I think I have a fix for that now, I can ping you when it ends up in `dev` if so.
> <img alt="Image" width="393" height="87" src="https://private-user-images.githubusercontent.com/104535511/479805725-9e3f0685-fff3-40b0-8827-eab9f7fc26b8.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTU2Nzk2MDYsIm5iZiI6MTc1NTY3OTMwNiwicGF0aCI6Ii8xMDQ1MzU1MTEvNDc5ODA1NzI1LTllM2YwNjg1LWZmZjMtNDBiMC04ODI3LWVhYjlmN2ZjMjZiOC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwODIwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDgyMFQwODQxNDZaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT04YWY1MDJkYTM5ZDUxMWQ2NmM1ZDYwYWEwZTg3N2ZlNzFkMDkwMzQyNTg2MmUxNDdiOThhNGFjZDMxZmI4NDQ3JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.Q9_r7C7MUpA6qUbXh_nn4hE89i4ZXNDbrMrUUGGlEtk">
not 500+ individual files, just a single large file that was chunked into 500+ vector chunks that all were trying to get uploaded at once
I'm getting the same thing but I'm confused about how this is related to file size/chunk/overlap. Isn't this a tika-config.xml issue, like re-configuring maxTotalEstimatedSize? Or maybe an issue between filterable metadata and non-filterable metadata?
Also getting this
WARNING | open_webui.retrieval.vector.dbs.s3vector:_filter_metadata:113 - Metadata for key '69a35f84-5f6a-4801-adc5-c75b2948c170' had 11 keys, limited to 10 keys
Is there a log level or a lazy way to be able to tail the s3 put open_webui.retrieval.vector.dbs.s3vector:insert:218? I'm trying to avoid a custom configuration for Tika because it's annoying if testing another provider. Maybe there's a simple way to sanitize/filter from the open_webui.retrieval.vector.dbs.s3vector provider side of things.
<!-- gh-comment-id:3458239165 -->
@rromanchuk commented on GitHub (Oct 28, 2025):
I'm getting the same thing but I'm confused about how this is related to file size/chunk/overlap. Isn't this a `tika-config.xml` issue, like re-configuring `maxTotalEstimatedSize`? Or maybe an issue between filterable metadata and non-filterable metadata?
Also getting this
> WARNING | open_webui.retrieval.vector.dbs.s3vector:_filter_metadata:113 - Metadata for key '69a35f84-5f6a-4801-adc5-c75b2948c170' had 11 keys, limited to 10 keys
Is there a log level or a lazy way to be able to tail the s3 put open_webui.retrieval.vector.dbs.s3vector:insert:218? I'm trying to avoid a custom configuration for Tika because it's annoying if testing another provider. Maybe there's a simple way to sanitize/filter from the open_webui.retrieval.vector.dbs.s3vector provider side of things.
<!-- gh-comment-id:3458830104 -->
@rromanchuk commented on GitHub (Oct 28, 2025):
Weird, it looks like this is already being done, including max key management.
https://github.com/open-webui/open-webui/blob/main/backend/open_webui/retrieval/vector/dbs/s3vector.py#L74
I should probably reset
The following example code generates 1024-dimensional vector embeddings with the Titan Text Embeddings V2 model from Amazon Bedrock by using the AWS SDK for Python (Boto3), and stores them in your vector index by using the PutVectors API. Along with each vector, we attach key-value pairs as filterable metadata. Additionally, we use a non-filterable metadata key that's named source_text to preserve the original text from which each vector is derived. To maximize request throughput and optimize for speed and efficiency, we recommend that you insert and delete vectors in batches. For more information, see S3 Vectors best practices.
This is confusing, are they saying source_text is a reserved keyword that makes it unfilterable and everything else is filterable? So every other kv pair is filterable, like metadata["text"] = item["text"]?
<!-- gh-comment-id:3458922978 -->
@rromanchuk commented on GitHub (Oct 28, 2025):
https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-getting-started.html
> The following example code generates 1024-dimensional vector embeddings with the Titan Text Embeddings V2 model from Amazon Bedrock by using the AWS SDK for Python (Boto3), and stores them in your vector index by using the PutVectors API. Along with each vector, we attach key-value pairs as filterable metadata. Additionally, we use a non-filterable metadata key that's named source_text to preserve the original text from which each vector is derived. To maximize request throughput and optimize for speed and efficiency, we recommend that you insert and delete vectors in batches. For more information, see [S3 Vectors best practices](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-best-practices.html).
This is confusing, are they saying `source_text` is a reserved keyword that makes it unfilterable and everything else is filterable? So every other kv pair is filterable, like `metadata["text"] = item["text"]`?
This says absolutely nothing about it too https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_PutInputVector.html
**ignore me/update** ohh, the keys are assigned on index creation ☹️
It doesn't look like this is being [assigned](https://github.com/open-webui/open-webui/blob/7a83e7dfa367d19f762ec17cac5e4a94ea2bd97d/backend/open_webui/retrieval/vector/dbs/s3vector.py#L60) though
```python
metadataConfiguration={
'nonFilterableMetadataKeys': [
'text',
]
}
```
Full signature
```python
response = client.create_index(
vectorBucketName='string',
vectorBucketArn='string',
indexName='string',
dataType='float32',
dimension=123,
distanceMetric='euclidean'|'cosine',
metadataConfiguration={
'nonFilterableMetadataKeys': [
'string',
]
}
)
```
@rromanchuk was marking the text metadata as non-filterable enough to solve an issue for you? I may have missed that, but can definitely test that out and help get that merged to main if so
<!-- gh-comment-id:3459389145 -->
@westbrook-ai commented on GitHub (Oct 29, 2025):
@rromanchuk was marking the text metadata as non-filterable enough to solve an issue for you? I may have missed that, but can definitely test that out and help get that merged to main if so
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @Cqban on GitHub (Aug 12, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/16527
Check Existing Issues
Installation Method
Docker
Open WebUI Version
v0.6.22
Ollama Version (if applicable)
No response
Operating System
Debian 12
Browser (if applicable)
No response
Confirmation
README.md.Expected Behavior
Documents should upload sucessfuly
Actual Behavior
I get an error if the chunk size is to big (it works with 300 30 overlap , but not with 400 40 overlap)
Steps to Reproduce
name: open-webui
services:
open-webui:
container_name: open-webui
image: ghcr.io/open-webui/open-webui:main
ports:
- 3000:8080
volumes:
- open-webui:/app/backend/data
restart: unless-stopped
environment:
VECTOR_DB: s3vector
S3_VECTOR_BUCKET_NAME: Hidden
AWS_ACCESS_KEY_ID: Hidden
AWS_SECRET_ACCESS_KEY: Hidden
AWS_REGION: Hidden
S3_VECTOR_REGION: Hidden
volumes:
open-webui:
external: true
name: open-webui
Logs & Screenshots
2025-08-12 13:35:14.822 | ERROR | open_webui.routers.files:upload_file:192 - 400: An error occurred (ValidationException) when calling the PutVectors operation: Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2048 bytes
Traceback (most recent call last):
File "/app/backend/open_webui/routers/retrieval.py", line 1509, in process_file
File "/app/backend/open_webui/routers/retrieval.py", line 1481, in process_file
File "/app/backend/open_webui/routers/retrieval.py", line 1323, in save_docs_to_vector_db
File "/app/backend/open_webui/routers/retrieval.py", line 1315, in save_docs_to_vector_db
File "/app/backend/open_webui/retrieval/vector/dbs/s3vector.py", line 201, in insert
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 602, in _api_call
File "/usr/local/lib/python3.11/site-packages/botocore/context.py", line 123, in wrapper
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 1078, in _make_api_call
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the PutVectors operation: Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2048 bytes
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap
File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run
File "/app/backend/open_webui/routers/retrieval.py", line 1526, in process_file
fastapi.exceptions.HTTPException: 400: An error occurred (ValidationException) when calling the PutVectors operation: Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2048 bytes
2025-08-12 13:35:14.932 | ERROR | open_webui.routers.files:upload_file:193 - Error processing file: 041ac3db-5e5a-4415-9163-8f7bd7a4a66d
Additional Information
It might just be because of S3 specifications and limitations but if i can be enhanced then it's worth looking at i guess
@westbrook-ai commented on GitHub (Aug 12, 2025):
Thanks @Cqban, I'll take a closer look at this. I've never used Tika before, do you happen to know if it adds/ extends existing metadata on vectors? I can do some testing and research to figure it out if you're not sure.
@westbrook-ai commented on GitHub (Aug 13, 2025):
Confirmed that the filterable metadata per vector is up to 2KB per the S3 vectors limitations docs: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-limitations.html
I never ran into that issue during my testing using the Open WebUI defaults for chunk size and chunk overlap, but I'll still take a closer look as soon as I can to try to understand what the total size of the metadata per vector was for me using those defaults.
@joshrenshaw12 commented on GitHub (Aug 13, 2025):
I've seen similar errors like below:
From my research this is due to the number of vectors being sent in the single
PutVectorsoperationhttps://www.perplexity.ai/search/botocore-errorfactory-validati-ZmuvRNkxQeal7FvCWXHTNQ
From docs mentioned above, we know API is limited to 500 vectors per PutVectors API call.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_PutVectors.html
This aligns with functionality i have seen in testing where a smaller file with less vectors uploads fine, but a larger one with more vectors does not.
Certainly a different issue to that mentioned above but shows the need for some batching logic applied to
backend/open_webui/retrieval/vector/dbs/s3vector.pyto match limitations of the AWS API.@westbrook-ai commented on GitHub (Aug 20, 2025):
@joshrenshaw12 your issue popped up when you tried to upload 500+ files to a collection at once right? I think I have a fix for that now, I can ping you when it ends up in
devif so.@westbrook-ai commented on GitHub (Aug 20, 2025):
@Cqban how big was the file you hit this issue on?
@joshrenshaw12 commented on GitHub (Aug 20, 2025):
not 500+ individual files, just a single large file that was chunked into 500+ vector chunks that all were trying to get uploaded at once
@rromanchuk commented on GitHub (Oct 28, 2025):
I'm getting the same thing but I'm confused about how this is related to file size/chunk/overlap. Isn't this a
tika-config.xmlissue, like re-configuringmaxTotalEstimatedSize? Or maybe an issue between filterable metadata and non-filterable metadata?Also getting this
Is there a log level or a lazy way to be able to tail the s3 put open_webui.retrieval.vector.dbs.s3vector:insert:218? I'm trying to avoid a custom configuration for Tika because it's annoying if testing another provider. Maybe there's a simple way to sanitize/filter from the open_webui.retrieval.vector.dbs.s3vector provider side of things.
@rromanchuk commented on GitHub (Oct 28, 2025):
Weird, it looks like this is already being done, including max key management.
https://github.com/open-webui/open-webui/blob/main/backend/open_webui/retrieval/vector/dbs/s3vector.py#L74
I should probably reset
@rromanchuk commented on GitHub (Oct 28, 2025):
https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-getting-started.html
This is confusing, are they saying
source_textis a reserved keyword that makes it unfilterable and everything else is filterable? So every other kv pair is filterable, likemetadata["text"] = item["text"]?This says absolutely nothing about it too https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_PutInputVector.html
ignore me/update ohh, the keys are assigned on index creation ☹️
It doesn't look like this is being assigned though
Full signature
@westbrook-ai commented on GitHub (Oct 29, 2025):
@rromanchuk was marking the text metadata as non-filterable enough to solve an issue for you? I may have missed that, but can definitely test that out and help get that merged to main if so