The Google PSE search returns maximum 10 results through 1 query. The Google PSE api returns an error if num is set to a number larger than 10. We need to use the start parameter to send multiple queries to get more than 10 results from google PSE
Before submitting, make sure you've checked the following:
Target branch: Please verify that the pull request targets the dev branch.
Description: Provide a concise description of the changes made in this pull request.
Changelog: Ensure a changelog entry following the format of Keep a Changelog is added at the bottom of the PR description.
Documentation: No need to update doc, this is an enhencement of previous feature.
Dependencies: No new dependencies
Testing: code is tested Have you written and run sufficient tests for validating the changes?
Code review: code is reviewed (this is a small pr)
Prefix:
feat: Introduces a new feature or enhancement to the codebase
Changelog Entry
Added
Pagination Support for count > 10:
Implemented pagination logic to handle requests for more than 10 search results from Google Programmable Search Engine.
When count is greater than 10, the function now automatically makes multiple API calls using the start parameter to retrieve results in pages of 10.
This allows users to retrieve up to 100 search results (the maximum practical limit due to Google PSE's page limitations).
Changed
count Parameter Behavior:
Updated the function to correctly handle the count parameter for values greater than 10.
The function now iteratively fetches results until the desired count is reached or no more results are available from Google PSE.
Clarified in the function docstring that count can now exceed 10 and up to a practical maximum of 100, due to Google PSE limitations.
Internal
Introduced start_index variable:
Added a start_index variable to manage the start parameter for Google PSE API calls during pagination.
start_index is incremented by 10 in each iteration to fetch the next page of results.
Looping for Pagination:
Implemented a while count > 0 loop to handle pagination logic.
The loop continues making API requests until the requested count is fulfilled or Google PSE returns no more results.
Results Accumulation:
Modified the function to accumulate results from each page into the all_results list before filtering and returning.
Added check for empty results:
Included a check if results: to break the pagination loop when Google PSE returns an empty list of results, indicating no more pages are available.
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.
## 📋 Pull Request Information
**Original PR:** https://github.com/open-webui/open-webui/pull/9837
**Author:** [@silverriver](https://github.com/silverriver)
**Created:** 2/12/2025
**Status:** ✅ Merged
**Merged:** 2/12/2025
**Merged by:** [@tjbck](https://github.com/tjbck)
**Base:** `dev` ← **Head:** `patch-1`
---
### 📝 Commits (1)
- [`7e08373`](https://github.com/open-webui/open-webui/commit/7e08373ae5976e8b3d73d4f82f40ca75692f764e) Update google_pse.py to return results more than 10
### 📊 Changes
**1 file changed** (+32 additions, -16 deletions)
<details>
<summary>View changed files</summary>
📝 `backend/open_webui/retrieval/web/google_pse.py` (+32 -16)
</details>
### 📄 Description
# Pull Request Checklist
The Google PSE search returns maximum 10 results through 1 query. The Google PSE api returns an error if num is set to a number larger than 10. We need to use the start parameter to send multiple queries to get more than 10 results from google PSE
[discussion](https://github.com/open-webui/open-webui/discussions/9836)
**Before submitting, make sure you've checked the following:**
- [x] **Target branch:** Please verify that the pull request targets the `dev` branch.
- [x] **Description:** Provide a concise description of the changes made in this pull request.
- [x] **Changelog:** Ensure a changelog entry following the format of [Keep a Changelog](https://keepachangelog.com/) is added at the bottom of the PR description.
- [ ] **Documentation**: No need to update doc, this is an enhencement of previous feature.
- [ ] **Dependencies**: No new dependencies
- [x] **Testing: code is tested** Have you written and run sufficient tests for validating the changes?
- [x] **Code review**: code is reviewed (this is a small pr)
- [x] **Prefix**:
- **feat**: Introduces a new feature or enhancement to the codebase
# Changelog Entry
### Added
- **Pagination Support for `count > 10`:**
- Implemented pagination logic to handle requests for more than 10 search results from Google Programmable Search Engine.
- When `count` is greater than 10, the function now automatically makes multiple API calls using the `start` parameter to retrieve results in pages of 10.
- This allows users to retrieve up to 100 search results (the maximum practical limit due to Google PSE's page limitations).
### Changed
- **`count` Parameter Behavior:**
- Updated the function to correctly handle the `count` parameter for values greater than 10.
- The function now iteratively fetches results until the desired `count` is reached or no more results are available from Google PSE.
- Clarified in the function docstring that `count` can now exceed 10 and up to a practical maximum of 100, due to Google PSE limitations.
### Internal
- **Introduced `start_index` variable:**
- Added a `start_index` variable to manage the `start` parameter for Google PSE API calls during pagination.
- `start_index` is incremented by 10 in each iteration to fetch the next page of results.
- **Looping for Pagination:**
- Implemented a `while count > 0` loop to handle pagination logic.
- The loop continues making API requests until the requested `count` is fulfilled or Google PSE returns no more results.
- **Results Accumulation:**
- Modified the function to accumulate results from each page into the `all_results` list before filtering and returning.
- **Added check for empty results:**
- Included a check `if results:` to break the pagination loop when Google PSE returns an empty list of results, indicating no more pages are available.
---
<sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
📋 Pull Request Information
Original PR: https://github.com/open-webui/open-webui/pull/9837
Author: @silverriver
Created: 2/12/2025
Status: ✅ Merged
Merged: 2/12/2025
Merged by: @tjbck
Base:
dev← Head:patch-1📝 Commits (1)
7e08373Update google_pse.py to return results more than 10📊 Changes
1 file changed (+32 additions, -16 deletions)
View changed files
📝
backend/open_webui/retrieval/web/google_pse.py(+32 -16)📄 Description
Pull Request Checklist
The Google PSE search returns maximum 10 results through 1 query. The Google PSE api returns an error if num is set to a number larger than 10. We need to use the start parameter to send multiple queries to get more than 10 results from google PSE
discussion
Before submitting, make sure you've checked the following:
devbranch.Changelog Entry
Added
count > 10:countis greater than 10, the function now automatically makes multiple API calls using thestartparameter to retrieve results in pages of 10.Changed
countParameter Behavior:countparameter for values greater than 10.countis reached or no more results are available from Google PSE.countcan now exceed 10 and up to a practical maximum of 100, due to Google PSE limitations.Internal
start_indexvariable:start_indexvariable to manage thestartparameter for Google PSE API calls during pagination.start_indexis incremented by 10 in each iteration to fetch the next page of results.while count > 0loop to handle pagination logic.countis fulfilled or Google PSE returns no more results.all_resultslist before filtering and returning.if results:to break the pagination loop when Google PSE returns an empty list of results, indicating no more pages are available.🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.