[GH-ISSUE #16527] issue: Limited chunk size with S3 Vectors #17944

New Issue

GiteaMirror · 2026-04-19T23:50:40-05:00

GiteaMirror commented

2026-04-19 23:50:40 -05:00

Originally created by @Cqban on GitHub (Aug 12, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/16527

Check Existing Issues

I have searched the existing issues and discussions.
I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

v0.6.22

Ollama Version (if applicable)

No response

Operating System

Debian 12

Browser (if applicable)

No response

Confirmation

I have read and followed all instructions in README.md.
I am using the latest version of both Open WebUI and Ollama.
I have included the browser console logs.
I have included the Docker container logs.
I have provided every relevant configuration, setting, and environment variable used in my setup.
I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
Start with the initial platform/version/OS and dependencies used,
Specify exact install/launch/configure commands,
List URLs visited, user input (incl. example values/emails/passwords if needed),
Describe all options and toggles enabled or changed,
Include any files or environmental changes,
Identify the expected and actual result at each stage,
Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

Documents should upload sucessfuly

Actual Behavior

I get an error if the chunk size is to big (it works with 300 30 overlap , but not with 400 40 overlap)

Steps to Reproduce

Deploy portainer
Deploy OpenWebUI with this stack configuration on portainer :

name: open-webui
services:
open-webui:
container_name: open-webui
image: ghcr.io/open-webui/open-webui:main
ports:
- 3000:8080
volumes:
- open-webui:/app/backend/data
restart: unless-stopped
environment:
VECTOR_DB: s3vector
S3_VECTOR_BUCKET_NAME: Hidden
AWS_ACCESS_KEY_ID: Hidden
AWS_SECRET_ACCESS_KEY: Hidden
AWS_REGION: Hidden
S3_VECTOR_REGION: Hidden

tika:
    container_name: tika
    image: apache/tika:latest-full
    ports:
        - 9998:9998
    restart: unless-stopped

volumes:
open-webui:
external: true
name: open-webui

Logs & Screenshots

2025-08-12 13:35:14.822 | ERROR | open_webui.routers.files:upload_file:192 - 400: An error occurred (ValidationException) when calling the PutVectors operation: Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2048 bytes

Traceback (most recent call last):

File "/app/backend/open_webui/routers/retrieval.py", line 1509, in process_file

raise e

File "/app/backend/open_webui/routers/retrieval.py", line 1481, in process_file

result = save_docs_to_vector_db(

         └ <function save_docs_to_vector_db at 0x7f43410ae8e0>

File "/app/backend/open_webui/routers/retrieval.py", line 1323, in save_docs_to_vector_db

raise e

File "/app/backend/open_webui/routers/retrieval.py", line 1315, in save_docs_to_vector_db

VECTOR_DB_CLIENT.insert(

│                └ <function S3VectorClient.insert at 0x7f43446cc680>

└ <open_webui.retrieval.vector.dbs.s3vector.S3VectorClient object at 0x7f4344af9250>

File "/app/backend/open_webui/retrieval/vector/dbs/s3vector.py", line 201, in insert

self.client.put_vectors(

│    │      └ <function ClientCreator._create_api_method.<locals>._api_call at 0x7f43446ce980>

│    └ <botocore.client.S3Vectors object at 0x7f4344020450>

└ <open_webui.retrieval.vector.dbs.s3vector.S3VectorClient object at 0x7f4344af9250>

File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 602, in _api_call

return self._make_api_call(operation_name, kwargs)

       │    │              │               └ {'vectorBucketName': 'openwebui', 'indexName': 'file-041ac3db-5e5a-4415-9163-8f7bd7a4a66d', 'vectors': [{'key': '5f2fbbd9-d7d...

       │    │              └ 'PutVectors'

       │    └ <function BaseClient._make_api_call at 0x7f434481c900>

       └ <botocore.client.S3Vectors object at 0x7f4344020450>

File "/usr/local/lib/python3.11/site-packages/botocore/context.py", line 123, in wrapper

return func(*args, **kwargs)

       │     │       └ {}

       │     └ (<botocore.client.S3Vectors object at 0x7f4344020450>, 'PutVectors', {'vectorBucketName': 'openwebui', 'indexName': 'file-041...

       └ <function BaseClient._make_api_call at 0x7f434481c860>

File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 1078, in _make_api_call

raise error_class(parsed_response, operation_name)

      │           │                └ 'PutVectors'

      │           └ {'Error': {'Message': "Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2...

      └ <class 'botocore.errorfactory.ValidationException'>

botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the PutVectors operation: Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2048 bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap

self._bootstrap_inner()

│    └ <function Thread._bootstrap_inner at 0x7f437a4749a0>

└ <WorkerThread(AnyIO worker thread, started 139926653867712)>

File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner

self.run()

│    └ <function WorkerThread.run at 0x7f433bffb1a0>

└ <WorkerThread(AnyIO worker thread, started 139926653867712)>

File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run

result = context.run(func, *args)

         │       │   │      └ ()

         │       │   └ functools.partial(<function upload_file at 0x7f4343e98180>, user=UserModel(id='ec30dbab-f0cb-4e08-a1b1-550da2623356', name='C...

         │       └ <method 'run' of '_contextvars.Context' objects>

         └ <_contextvars.Context object at 0x7f433abb6ec0>

File "/app/backend/open_webui/routers/files.py", line 183, in upload_file

process_file(request, ProcessFileForm(file_id=id), user=user)

│            │        │                       │         └ UserModel(id='ec30dbab-f0cb-4e08-a1b1-550da2623356', name='Cqban', email='cqban@cqban.xyz', role='admin', profile_image_url='...

│            │        │                       └ '041ac3db-5e5a-4415-9163-8f7bd7a4a66d'

│            │        └ <class 'open_webui.routers.retrieval.ProcessFileForm'>

│            └ <starlette.requests.Request object at 0x7f433ab85490>

└ <function process_file at 0x7f43410af600>

File "/app/backend/open_webui/routers/retrieval.py", line 1526, in process_file

raise HTTPException(

      └ <class 'fastapi.exceptions.HTTPException'>

fastapi.exceptions.HTTPException: 400: An error occurred (ValidationException) when calling the PutVectors operation: Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2048 bytes

2025-08-12 13:35:14.932 | ERROR | open_webui.routers.files:upload_file:193 - Error processing file: 041ac3db-5e5a-4415-9163-8f7bd7a4a66d

Additional Information

It might just be because of S3 specifications and limitations but if i can be enhanced then it's worth looking at i guess

Originally created by @Cqban on GitHub (Aug 12, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/16527 ### Check Existing Issues - [x] I have searched the existing issues and discussions. - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version v0.6.22 ### Ollama Version (if applicable) _No response_ ### Operating System Debian 12 ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior Documents should upload sucessfuly ### Actual Behavior I get an error if the chunk size is to big (it works with 300 30 overlap , but not with 400 40 overlap) ### Steps to Reproduce 1. Deploy portainer 2. Deploy OpenWebUI with this stack configuration on portainer : name: open-webui services: open-webui: container_name: open-webui image: ghcr.io/open-webui/open-webui:main ports: - 3000:8080 volumes: - open-webui:/app/backend/data restart: unless-stopped environment: VECTOR_DB: s3vector S3_VECTOR_BUCKET_NAME: Hidden AWS_ACCESS_KEY_ID: Hidden AWS_SECRET_ACCESS_KEY: Hidden AWS_REGION: Hidden S3_VECTOR_REGION: Hidden tika: container_name: tika image: apache/tika:latest-full ports: - 9998:9998 restart: unless-stopped volumes: open-webui: external: true name: open-webui ### Logs & Screenshots 2025-08-12 13:35:14.822 | ERROR | open_webui.routers.files:upload_file:192 - 400: An error occurred (ValidationException) when calling the PutVectors operation: Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2048 bytes Traceback (most recent call last): File "/app/backend/open_webui/routers/retrieval.py", line 1509, in process_file raise e File "/app/backend/open_webui/routers/retrieval.py", line 1481, in process_file result = save_docs_to_vector_db( └ <function save_docs_to_vector_db at 0x7f43410ae8e0> File "/app/backend/open_webui/routers/retrieval.py", line 1323, in save_docs_to_vector_db raise e File "/app/backend/open_webui/routers/retrieval.py", line 1315, in save_docs_to_vector_db VECTOR_DB_CLIENT.insert( │ └ <function S3VectorClient.insert at 0x7f43446cc680> └ <open_webui.retrieval.vector.dbs.s3vector.S3VectorClient object at 0x7f4344af9250> File "/app/backend/open_webui/retrieval/vector/dbs/s3vector.py", line 201, in insert self.client.put_vectors( │ │ └ <function ClientCreator._create_api_method.<locals>._api_call at 0x7f43446ce980> │ └ <botocore.client.S3Vectors object at 0x7f4344020450> └ <open_webui.retrieval.vector.dbs.s3vector.S3VectorClient object at 0x7f4344af9250> File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 602, in _api_call return self._make_api_call(operation_name, kwargs) │ │ │ └ {'vectorBucketName': 'openwebui', 'indexName': 'file-041ac3db-5e5a-4415-9163-8f7bd7a4a66d', 'vectors': [{'key': '5f2fbbd9-d7d... │ │ └ 'PutVectors' │ └ <function BaseClient._make_api_call at 0x7f434481c900> └ <botocore.client.S3Vectors object at 0x7f4344020450> File "/usr/local/lib/python3.11/site-packages/botocore/context.py", line 123, in wrapper return func(*args, **kwargs) │ │ └ {} │ └ (<botocore.client.S3Vectors object at 0x7f4344020450>, 'PutVectors', {'vectorBucketName': 'openwebui', 'indexName': 'file-041... └ <function BaseClient._make_api_call at 0x7f434481c860> File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 1078, in _make_api_call raise error_class(parsed_response, operation_name) │ │ └ 'PutVectors' │ └ {'Error': {'Message': "Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2... └ <class 'botocore.errorfactory.ValidationException'> botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the PutVectors operation: Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2048 bytes During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap self._bootstrap_inner() │ └ <function Thread._bootstrap_inner at 0x7f437a4749a0> └ <WorkerThread(AnyIO worker thread, started 139926653867712)> File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner self.run() │ └ <function WorkerThread.run at 0x7f433bffb1a0> └ <WorkerThread(AnyIO worker thread, started 139926653867712)> File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run result = context.run(func, *args) │ │ │ └ () │ │ └ functools.partial(<function upload_file at 0x7f4343e98180>, user=UserModel(id='ec30dbab-f0cb-4e08-a1b1-550da2623356', name='C... │ └ <method 'run' of '_contextvars.Context' objects> └ <_contextvars.Context object at 0x7f433abb6ec0> > File "/app/backend/open_webui/routers/files.py", line 183, in upload_file process_file(request, ProcessFileForm(file_id=id), user=user) │ │ │ │ └ UserModel(id='ec30dbab-f0cb-4e08-a1b1-550da2623356', name='Cqban', email='cqban@cqban.xyz', role='admin', profile_image_url='... │ │ │ └ '041ac3db-5e5a-4415-9163-8f7bd7a4a66d' │ │ └ <class 'open_webui.routers.retrieval.ProcessFileForm'> │ └ <starlette.requests.Request object at 0x7f433ab85490> └ <function process_file at 0x7f43410af600> File "/app/backend/open_webui/routers/retrieval.py", line 1526, in process_file raise HTTPException( └ <class 'fastapi.exceptions.HTTPException'> fastapi.exceptions.HTTPException: 400: An error occurred (ValidationException) when calling the PutVectors operation: Invalid record for key '5f2fbbd9-d7df-4ec2-ac9d-6c521c6a9e04': Filterable metadata must have at most 2048 bytes 2025-08-12 13:35:14.932 | ERROR | open_webui.routers.files:upload_file:193 - Error processing file: 041ac3db-5e5a-4415-9163-8f7bd7a4a66d ### Additional Information It might just be because of S3 specifications and limitations but if i can be enhanced then it's worth looking at i guess

GiteaMirror added the bug label 2026-04-19 23:50:40 -05:00

GiteaMirror commented

2026-04-19 23:50:41 -05:00

@westbrook-ai commented on GitHub (Aug 12, 2025):

Thanks @Cqban, I'll take a closer look at this. I've never used Tika before, do you happen to know if it adds/ extends existing metadata on vectors? I can do some testing and research to figure it out if you're not sure.

@westbrook-ai commented on GitHub (Aug 12, 2025): Thanks @Cqban, I'll take a closer look at this. I've never used Tika before, do you happen to know if it adds/ extends existing metadata on vectors? I can do some testing and research to figure it out if you're not sure.

GiteaMirror commented

2026-04-19 23:50:42 -05:00

@westbrook-ai commented on GitHub (Aug 13, 2025):

Confirmed that the filterable metadata per vector is up to 2KB per the S3 vectors limitations docs: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-limitations.html

I never ran into that issue during my testing using the Open WebUI defaults for chunk size and chunk overlap, but I'll still take a closer look as soon as I can to try to understand what the total size of the metadata per vector was for me using those defaults.

@westbrook-ai commented on GitHub (Aug 13, 2025): Confirmed that the filterable metadata per vector is up to 2KB per the S3 vectors limitations docs: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-limitations.html I never ran into that issue during my testing using the Open WebUI defaults for [chunk size](https://docs.openwebui.com/getting-started/env-configuration/#chunk_size) and [chunk overlap](https://docs.openwebui.com/getting-started/env-configuration/#chunk_overlap), but I'll still take a closer look as soon as I can to try to understand what the total size of the metadata per vector was for me using those defaults.

GiteaMirror commented

2026-04-19 23:50:43 -05:00

@joshrenshaw12 commented on GitHub (Aug 13, 2025):

I've seen similar errors like below:

2025-08-13 08:31:56.497 | ERROR | open_webui.routers.files:upload_file:192 - 400: An error occurred (ValidationException) when calling the PutVectors operation: Request body exceeds max allowed size

From my research this is due to the number of vectors being sent in the single PutVectors operation
https://www.perplexity.ai/search/botocore-errorfactory-validati-ZmuvRNkxQeal7FvCWXHTNQ

From docs mentioned above, we know API is limited to 500 vectors per PutVectors API call.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_PutVectors.html

This aligns with functionality i have seen in testing where a smaller file with less vectors uploads fine, but a larger one with more vectors does not.
Certainly a different issue to that mentioned above but shows the need for some batching logic applied to backend/open_webui/retrieval/vector/dbs/s3vector.py to match limitations of the AWS API.

@joshrenshaw12 commented on GitHub (Aug 13, 2025): I've seen similar errors like below: ``` 2025-08-13 08:31:56.497 | ERROR | open_webui.routers.files:upload_file:192 - 400: An error occurred (ValidationException) when calling the PutVectors operation: Request body exceeds max allowed size ``` From my research this is due to the number of vectors being sent in the single `PutVectors` operation https://www.perplexity.ai/search/botocore-errorfactory-validati-ZmuvRNkxQeal7FvCWXHTNQ From docs mentioned above, we know API is limited to 500 vectors per PutVectors API call. https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_PutVectors.html This aligns with functionality i have seen in testing where a smaller file with less vectors uploads fine, but a larger one with more vectors does not. Certainly a different issue to that mentioned above but shows the need for some batching logic applied to `backend/open_webui/retrieval/vector/dbs/s3vector.py` to match limitations of the AWS API.

GiteaMirror commented

2026-04-19 23:50:43 -05:00

@westbrook-ai commented on GitHub (Aug 20, 2025):

@joshrenshaw12 your issue popped up when you tried to upload 500+ files to a collection at once right? I think I have a fix for that now, I can ping you when it ends up in dev if so.

@westbrook-ai commented on GitHub (Aug 20, 2025): @joshrenshaw12 your issue popped up when you tried to upload 500+ files to a collection at once right? I think I have a fix for that now, I can ping you when it ends up in `dev` if so. <img width="393" height="87" alt="Image" src="https://github.com/user-attachments/assets/9e3f0685-fff3-40b0-8827-eab9f7fc26b8" />

GiteaMirror commented

2026-04-19 23:50:44 -05:00

@westbrook-ai commented on GitHub (Aug 20, 2025):

@Cqban how big was the file you hit this issue on?

@westbrook-ai commented on GitHub (Aug 20, 2025): @Cqban how big was the file you hit this issue on?

GiteaMirror commented

2026-04-19 23:50:44 -05:00

@joshrenshaw12 commented on GitHub (Aug 20, 2025):

@joshrenshaw12 your issue popped up when you tried to upload 500+ files to a collection at once right? I think I have a fix for that now, I can ping you when it ends up in dev if so.

not 500+ individual files, just a single large file that was chunked into 500+ vector chunks that all were trying to get uploaded at once

@joshrenshaw12 commented on GitHub (Aug 20, 2025): > [@joshrenshaw12](https://github.com/joshrenshaw12) your issue popped up when you tried to upload 500+ files to a collection at once right? I think I have a fix for that now, I can ping you when it ends up in `dev` if so. > <img alt="Image" width="393" height="87" src="https://private-user-images.githubusercontent.com/104535511/479805725-9e3f0685-fff3-40b0-8827-eab9f7fc26b8.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTU2Nzk2MDYsIm5iZiI6MTc1NTY3OTMwNiwicGF0aCI6Ii8xMDQ1MzU1MTEvNDc5ODA1NzI1LTllM2YwNjg1LWZmZjMtNDBiMC04ODI3LWVhYjlmN2ZjMjZiOC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwODIwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDgyMFQwODQxNDZaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT04YWY1MDJkYTM5ZDUxMWQ2NmM1ZDYwYWEwZTg3N2ZlNzFkMDkwMzQyNTg2MmUxNDdiOThhNGFjZDMxZmI4NDQ3JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.Q9_r7C7MUpA6qUbXh_nn4hE89i4ZXNDbrMrUUGGlEtk"> not 500+ individual files, just a single large file that was chunked into 500+ vector chunks that all were trying to get uploaded at once

GiteaMirror commented

2026-04-19 23:50:45 -05:00

@rromanchuk commented on GitHub (Oct 28, 2025):

I'm getting the same thing but I'm confused about how this is related to file size/chunk/overlap. Isn't this a tika-config.xml issue, like re-configuring maxTotalEstimatedSize? Or maybe an issue between filterable metadata and non-filterable metadata?

Also getting this

WARNING | open_webui.retrieval.vector.dbs.s3vector:_filter_metadata:113 - Metadata for key '69a35f84-5f6a-4801-adc5-c75b2948c170' had 11 keys, limited to 10 keys

Is there a log level or a lazy way to be able to tail the s3 put open_webui.retrieval.vector.dbs.s3vector:insert:218? I'm trying to avoid a custom configuration for Tika because it's annoying if testing another provider. Maybe there's a simple way to sanitize/filter from the open_webui.retrieval.vector.dbs.s3vector provider side of things.

@rromanchuk commented on GitHub (Oct 28, 2025): I'm getting the same thing but I'm confused about how this is related to file size/chunk/overlap. Isn't this a `tika-config.xml` issue, like re-configuring `maxTotalEstimatedSize`? Or maybe an issue between filterable metadata and non-filterable metadata? Also getting this > WARNING | open_webui.retrieval.vector.dbs.s3vector:_filter_metadata:113 - Metadata for key '69a35f84-5f6a-4801-adc5-c75b2948c170' had 11 keys, limited to 10 keys Is there a log level or a lazy way to be able to tail the s3 put open_webui.retrieval.vector.dbs.s3vector:insert:218? I'm trying to avoid a custom configuration for Tika because it's annoying if testing another provider. Maybe there's a simple way to sanitize/filter from the open_webui.retrieval.vector.dbs.s3vector provider side of things.

GiteaMirror commented

2026-04-19 23:50:45 -05:00

@rromanchuk commented on GitHub (Oct 28, 2025):

Weird, it looks like this is already being done, including max key management.

https://github.com/open-webui/open-webui/blob/main/backend/open_webui/retrieval/vector/dbs/s3vector.py#L74

I should probably reset

@rromanchuk commented on GitHub (Oct 28, 2025): Weird, it looks like this is already being done, including max key management. https://github.com/open-webui/open-webui/blob/main/backend/open_webui/retrieval/vector/dbs/s3vector.py#L74 I should probably reset

GiteaMirror commented

2026-04-19 23:50:46 -05:00

@rromanchuk commented on GitHub (Oct 28, 2025):

https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-getting-started.html

The following example code generates 1024-dimensional vector embeddings with the Titan Text Embeddings V2 model from Amazon Bedrock by using the AWS SDK for Python (Boto3), and stores them in your vector index by using the PutVectors API. Along with each vector, we attach key-value pairs as filterable metadata. Additionally, we use a non-filterable metadata key that's named source_text to preserve the original text from which each vector is derived. To maximize request throughput and optimize for speed and efficiency, we recommend that you insert and delete vectors in batches. For more information, see S3 Vectors best practices.

This is confusing, are they saying source_text is a reserved keyword that makes it unfilterable and everything else is filterable? So every other kv pair is filterable, like metadata["text"] = item["text"]?

This says absolutely nothing about it too https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_PutInputVector.html

ignore me/update ohh, the keys are assigned on index creation ☹️

It doesn't look like this is being assigned though

metadataConfiguration={
        'nonFilterableMetadataKeys': [
            'text',
        ]
    }

Full signature

response = client.create_index(
    vectorBucketName='string',
    vectorBucketArn='string',
    indexName='string',
    dataType='float32',
    dimension=123,
    distanceMetric='euclidean'|'cosine',
    metadataConfiguration={
        'nonFilterableMetadataKeys': [
            'string',
        ]
    }
)

@rromanchuk commented on GitHub (Oct 28, 2025): https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-getting-started.html > The following example code generates 1024-dimensional vector embeddings with the Titan Text Embeddings V2 model from Amazon Bedrock by using the AWS SDK for Python (Boto3), and stores them in your vector index by using the PutVectors API. Along with each vector, we attach key-value pairs as filterable metadata. Additionally, we use a non-filterable metadata key that's named source_text to preserve the original text from which each vector is derived. To maximize request throughput and optimize for speed and efficiency, we recommend that you insert and delete vectors in batches. For more information, see [S3 Vectors best practices](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-best-practices.html). This is confusing, are they saying `source_text` is a reserved keyword that makes it unfilterable and everything else is filterable? So every other kv pair is filterable, like `metadata["text"] = item["text"]`? This says absolutely nothing about it too https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_PutInputVector.html **ignore me/update** ohh, the keys are assigned on index creation ☹️ It doesn't look like this is being [assigned](https://github.com/open-webui/open-webui/blob/7a83e7dfa367d19f762ec17cac5e4a94ea2bd97d/backend/open_webui/retrieval/vector/dbs/s3vector.py#L60) though ```python metadataConfiguration={ 'nonFilterableMetadataKeys': [ 'text', ] } ``` Full signature ```python response = client.create_index( vectorBucketName='string', vectorBucketArn='string', indexName='string', dataType='float32', dimension=123, distanceMetric='euclidean'|'cosine', metadataConfiguration={ 'nonFilterableMetadataKeys': [ 'string', ] } ) ```

GiteaMirror commented

2026-04-19 23:50:46 -05:00

@westbrook-ai commented on GitHub (Oct 29, 2025):

@rromanchuk was marking the text metadata as non-filterable enough to solve an issue for you? I may have missed that, but can definitely test that out and help get that merged to main if so

@westbrook-ai commented on GitHub (Oct 29, 2025): @rromanchuk was marking the text metadata as non-filterable enough to solve an issue for you? I may have missed that, but can definitely test that out and help get that merged to main if so

GiteaMirror referenced this issue

2026-04-20 05:29:45 -05:00

[PR #17944] [MERGED] build(deps): bump pydantic from 2.11.7 to 2.11.9 in /backend #24612

GiteaMirror referenced this issue

2026-04-25 12:38:56 -05:00

[PR #17944] [MERGED] build(deps): bump pydantic from 2.11.7 to 2.11.9 in /backend #40242

GiteaMirror referenced this issue

2026-04-29 23:00:07 -05:00

[PR #17944] [MERGED] build(deps): bump pydantic from 2.11.7 to 2.11.9 in /backend #47660

Sign in to join this conversation.