[PR #22921] [CLOSED] fix: prepare-pyodide lock file handling and add openpyxl #97967

Closed
opened 2026-05-16 00:39:11 -05:00 by GiteaMirror · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/22921
Author: @EliotGodard
Created: 3/21/2026
Status: Closed

Base: devHead: dev-pyodide-libs


📝 Commits (2)

  • 8cb6573 fix: prepare-pyodide script lock file and version comparison
  • dee7a11 feat: add openpyxl to pyodide default packages

📊 Changes

4 files changed (+36 additions, -8 deletions)

View changed files

📝 scripts/prepare-pyodide.js (+31 -5)
📝 src/lib/components/chat/Messages/CodeBlock.svelte (+2 -1)
📝 src/routes/+layout.svelte (+2 -1)
📝 static/pyodide/pyodide-lock.json (+1 -1)

📄 Description

Pull Request Checklist

  • Target branch: Verify that the pull request targets the dev branch.
  • Description: Provided below.
  • Changelog: Provided below.
  • Testing: Manually tested npm run pyodide:fetch and verified openpyxl imports work via both code blocks and native tool calling.
  • Code review: Self-reviewed.
  • Git Hygiene: Two atomic commits, rebased on dev.
  • Title Prefix: fix:

Changelog Entry

Description

The prepare-pyodide.js script was only caching Pyodide's built-in packages (numpy, pandas, scipy, etc.) to static/pyodide/. Packages that are not part of Pyodide's distribution — even ones already listed in the script like seaborn or black — were never cached locally. Their wheels were fetched by the browser directly from PyPI at runtime every time.

This is because packageCacheDir only applies to packages loaded via pyodide.loadPackage(), not to packages installed via micropip.install() (except if they are part of the Pyodide's built-in packages list). micropip downloads wheels into pyodide's in-memory virtual filesystem without writing them to disk.

To fix this, the script now:

  1. Prevents copyPyodide() from overwriting pyodide-lock.json — the generated lock file from micropip.freeze() was being overwritten by the stock one from node_modules/pyodide/. This in turn made it necessary to normalize file_name paths in the lock file: micropip.freeze() produces absolute local paths and remote PyPI URLs, but the browser resolves wheels relative to indexURL and expects bare filenames. This wasn't an issue before because the stock lock file from node_modules/ already had bare filenames.

  2. Downloads PyPI-only wheels locally — wheels for packages not in Pyodide's distribution are now fetched during the build and saved to static/pyodide/, so they are served locally instead of from PyPI at runtime.

  3. Fixes version comparison — the script compared the semver range from package.json (e.g. ^0.28.20.28.2 after stripping ^) against the actual installed version (0.28.3). Since the current latest available version is 0.28.3, this mismatch caused the script to nuke the static/pyodide/ cache directory on every run. The comparison now uses the actual installed version from node_modules/pyodide/package.json.

  4. Adds openpyxl as an example of a package that can now be properly pre-bundled with this refactor, since it's not part of Pyodide's built-in distribution. openpyxl is useful for manipulating Excel files in the code interpreter.

Added

  • openpyxl to the default pyodide package list and import detection in CodeBlock.svelte and +layout.svelte
  • Build step to download PyPI-only wheels into static/pyodide/

Fixed

  • Version comparison uses actual installed version from node_modules/pyodide/package.json
  • Lock file file_name entries normalized to bare filenames
  • copyPyodide() skips pyodide-lock.json to prevent overwriting the generated lock file

Screenshots or Videos

N/A — build script and runtime package loading fix.

Contributor License Agreement


🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/22921 **Author:** [@EliotGodard](https://github.com/EliotGodard) **Created:** 3/21/2026 **Status:** ❌ Closed **Base:** `dev` ← **Head:** `dev-pyodide-libs` --- ### 📝 Commits (2) - [`8cb6573`](https://github.com/open-webui/open-webui/commit/8cb65736958460ebb52b0ae8e48cf338a0b7c76b) fix: prepare-pyodide script lock file and version comparison - [`dee7a11`](https://github.com/open-webui/open-webui/commit/dee7a117be01658ffdaf8b5bbfb51c5a2c9506f8) feat: add openpyxl to pyodide default packages ### 📊 Changes **4 files changed** (+36 additions, -8 deletions) <details> <summary>View changed files</summary> 📝 `scripts/prepare-pyodide.js` (+31 -5) 📝 `src/lib/components/chat/Messages/CodeBlock.svelte` (+2 -1) 📝 `src/routes/+layout.svelte` (+2 -1) 📝 `static/pyodide/pyodide-lock.json` (+1 -1) </details> ### 📄 Description <!-- ⚠️ CRITICAL CHECKS FOR CONTRIBUTORS (READ, DON'T DELETE) ⚠️ 1. Target the `dev` branch. PRs targeting `main` will be automatically closed. 2. Do NOT delete the CLA section at the bottom. It is required for the bot to accept your PR. --> # Pull Request Checklist - [x] **Target branch:** Verify that the pull request targets the `dev` branch. - [x] **Description:** Provided below. - [x] **Changelog:** Provided below. - [x] **Testing:** Manually tested `npm run pyodide:fetch` and verified openpyxl imports work via both code blocks and native tool calling. - [x] **Code review:** Self-reviewed. - [x] **Git Hygiene:** Two atomic commits, rebased on `dev`. - [x] **Title Prefix:** `fix:` # Changelog Entry ### Description The `prepare-pyodide.js` script was only caching Pyodide's built-in packages (numpy, pandas, scipy, etc.) to `static/pyodide/`. Packages that are not part of Pyodide's distribution — even ones already listed in the script like seaborn or black — were never cached locally. Their wheels were fetched by the browser directly from PyPI at runtime every time. This is because `packageCacheDir` only applies to packages loaded via `pyodide.loadPackage()`, not to packages installed via `micropip.install()` (except if they are part of the Pyodide's built-in packages list). micropip downloads wheels into pyodide's in-memory virtual filesystem without writing them to disk. To fix this, the script now: 1. **Prevents `copyPyodide()` from overwriting `pyodide-lock.json`** — the generated lock file from `micropip.freeze()` was being overwritten by the stock one from `node_modules/pyodide/`. This in turn made it necessary to **normalize `file_name` paths** in the lock file: `micropip.freeze()` produces absolute local paths and remote PyPI URLs, but the browser resolves wheels relative to `indexURL` and expects bare filenames. This wasn't an issue before because the stock lock file from `node_modules/` already had bare filenames. 2. **Downloads PyPI-only wheels locally** — wheels for packages not in Pyodide's distribution are now fetched during the build and saved to `static/pyodide/`, so they are served locally instead of from PyPI at runtime. 3. **Fixes version comparison** — the script compared the semver range from `package.json` (e.g. `^0.28.2` → `0.28.2` after stripping `^`) against the actual installed version (`0.28.3`). Since the current latest available version is `0.28.3`, this mismatch caused the script to nuke the `static/pyodide/` cache directory on every run. The comparison now uses the actual installed version from `node_modules/pyodide/package.json`. 4. **Adds openpyxl** as an example of a package that can now be properly pre-bundled with this refactor, since it's not part of Pyodide's built-in distribution. openpyxl is useful for manipulating Excel files in the code interpreter. ### Added - openpyxl to the default pyodide package list and import detection in `CodeBlock.svelte` and `+layout.svelte` - Build step to download PyPI-only wheels into `static/pyodide/` ### Fixed - Version comparison uses actual installed version from `node_modules/pyodide/package.json` - Lock file `file_name` entries normalized to bare filenames - `copyPyodide()` skips `pyodide-lock.json` to prevent overwriting the generated lock file ### Screenshots or Videos N/A — build script and runtime package loading fix. ### Contributor License Agreement - [x] By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](https://github.com/open-webui/open-webui/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-05-16 00:39:11 -05:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#97967