mirror of
https://github.com/open-webui/open-webui.git
synced 2026-07-20 20:52:23 -05:00
[GH-ISSUE #19233] issue: S3 Vectors has_collection() fails for indexes beyond first page due to missing pagination #57483
Reference in New Issue
Block a user
Originally created by @shargyle on GitHub (Nov 17, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/19233
Check Existing Issues
Installation Method
Docker
Open WebUI Version
v0.6.36
Ollama Version (if applicable)
No response
Operating System
AWS ECS
Browser (if applicable)
No response
Confirmation
README.md.Expected Behavior
When using S3 Vectors for RAG storage, the
has_collection()method should correctly detect if a vector index exists, regardless of how many total indexes exist in the bucket.When uploading a file and then querying it:
Actual Behavior
When the S3 Vectors bucket contains more than approximately 1,000 indexes:
WARNING | Collection 'file-<uuid>' does not existPattern observed:
Steps to Reproduce
Environment Setup
Platform: AWS ECS Fargate (or any Docker deployment)
Open WebUI Version: v0.6.36
Docker Image:
ghcr.io/open-webui/open-webui:v0.6.36Required Environment Variables:
Docker Run Command:
AWS S3 Vectors Bucket Setup:
IAM Permissions Required:
Creating the Prerequisite State (1,000+ Indexes)
Option 1: Upload 1,000 files through UI (time-consuming but realistic)
Option 2: Script to create test indexes (faster for reproduction)
Reproduction Steps
Step 1: Verify container is running
Step 2: Access Open WebUI
http://localhost:3000(or your deployment URL)Step 3: Create test file
Create a test file
test-document.txtwith content:Step 4: Upload file to workspace
test-document.txtStep 5: Monitor logs during upload
Expected:
Actual: Same as expected ✓
Step 6: Wait for indexing to complete
Step 7: Verify index exists via AWS CLI
Expected: Error (index doesn't exist)
Actual: Returns index details - index EXISTS! ✓
Step 8: Query the uploaded file
What does the document say about quantum computing?Step 9: Check logs for warnings
Expected: No warnings
Actual: Multiple warnings ❌
Step 10: Verify total index count
Expected: Any number
Actual: Returns
1000(or close to it), and response includes"nextToken": "..."indicating pagination ❌Key Observations
has_collection()returns Falsehas_collection()only checks first page oflist_indexes()results (~1,000 indexes)Root Cause (Technical Details)
The
has_collection()method inbackend/open_webui/retrieval/vector/dbs/s3vector.py(line ~126) uses boto3'slist_indexes()without pagination handling:The Problem:
list_indexes()returns only ~1,000 indexes by default with anextTokenfor paginationEvidence from boto3 testing:
With more than ~1,000 indexes in the bucket, boto3 returns approximately 1,000 indexes with a
nextTokenindicating more pages exist. The code doesn't handle pagination, so indexes beyond the first page are invisible to the application.Logs & Screenshots
Index Creation (successful):
Query Attempt (minutes later):
AWS CLI verification (proves index exists):
Additional Information
Proposed Solution
Replace
list_indexes()with directget_index()lookup:Benefits:
Testing Results:
get_index()call: 0.19 seconds, found correctly ✅list_indexes()call: 1.53 seconds, not found (beyond page 1) ❌Impact:
Workaround:
No reliable workaround available. The issue affects all new file uploads once the bucket contains more than ~1,000 indexes.
Willing to Submit PR:
Yes, I have identified the root cause and proposed a fix with supporting evidence. I can submit a PR to address this issue, though full integration testing would require a test environment with 1,000+ indexes.