feat: streamline Python package management and code quality with uv and pre-commit #6298

New Issue

GiteaMirror · 2025-11-11T16:50:23-06:00

GiteaMirror commented

2025-11-11 16:50:23 -06:00

Originally created by @khamaileon on GitHub (Sep 2, 2025).

Check Existing Issues

I have searched the existing issues and discussions.

Problem Description

Currently, Open-webui manages Python dependencies in two separate files: pyproject.toml and requirements.txt. This duplication often leads to formatting issues and conflicts, especially during rebases or merges. Maintaining consistency between these files is error-prone and can slow down development.

Desired Solution you'd like

I propose switching to uv as the primary Python dependency management tool. uv can generate and synchronize lockfiles, ensuring that dependencies are always up-to-date and consistent. Additionally, I recommend installing pre-commit to automate code quality checks and dependency file management.

Here is an example of a .pre-commit-config.yaml file:

repos:
  - repo: "https://github.com/pre-commit/pre-commit-hooks"
    rev: v6.0.0
    hooks:
      - id: check-yaml
        args: ["--unsafe"]
      - id: check-toml
      - id: end-of-file-fixer
      - id: trailing-whitespace

  - repo: https://github.com/astral-sh/uv-pre-commit
    rev: 0.8.14
    hooks:
      # Ensure the lockfile is up-to-date
      - id: uv-lock
      # Autoexport uv.lock to requirements.txt
      - id: uv-export
        args: ["--output-file=backend/requirements.txt"]

  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.12.11
    hooks:
      # Run the formatter.
      - id: ruff-format

What pre-commit does:

Automatically checks YAML and TOML files for syntax errors.
Fixes end-of-file and trailing whitespace issues.
Ensures the dependency lockfile is always up-to-date.
Exports the lockfile to backend/requirements.txt automatically.
Runs code formatting tools to maintain code style consistency.

I can submit a PR to implement this setup and update the documentation to reflect these changes.

Alternatives Considered

No response

Additional Context

No response

Originally created by @khamaileon on GitHub (Sep 2, 2025). ### Check Existing Issues - [x] I have searched the existing issues and discussions. ### Problem Description Currently, Open-webui manages Python dependencies in two separate files: pyproject.toml and requirements.txt. This duplication often leads to formatting issues and conflicts, especially during rebases or merges. Maintaining consistency between these files is error-prone and can slow down development. ### Desired Solution you'd like I propose switching to uv as the primary Python dependency management tool. uv can generate and synchronize lockfiles, ensuring that dependencies are always up-to-date and consistent. Additionally, I recommend installing pre-commit to automate code quality checks and dependency file management. Here is an example of a .pre-commit-config.yaml file: ```yaml repos: - repo: "https://github.com/pre-commit/pre-commit-hooks" rev: v6.0.0 hooks: - id: check-yaml args: ["--unsafe"] - id: check-toml - id: end-of-file-fixer - id: trailing-whitespace - repo: https://github.com/astral-sh/uv-pre-commit rev: 0.8.14 hooks: # Ensure the lockfile is up-to-date - id: uv-lock # Autoexport uv.lock to requirements.txt - id: uv-export args: ["--output-file=backend/requirements.txt"] - repo: https://github.com/astral-sh/ruff-pre-commit rev: v0.12.11 hooks: # Run the formatter. - id: ruff-format ``` What pre-commit does: Automatically checks YAML and TOML files for syntax errors. Fixes end-of-file and trailing whitespace issues. Ensures the dependency lockfile is always up-to-date. Exports the lockfile to backend/requirements.txt automatically. Runs code formatting tools to maintain code style consistency. I can submit a PR to implement this setup and update the documentation to reflect these changes. ### Alternatives Considered _No response_ ### Additional Context _No response_

GiteaMirror closed this issue

2025-11-11 16:50:23 -06:00

GiteaMirror commented

2025-11-11 16:50:23 -06:00

@rgaricano commented on GitHub (Sep 2, 2025):

personally I prefer don't use uv and even less for development: https://github.com/pypa/packaging.python.org/issues/685#issuecomment-1321616748

@rgaricano commented on GitHub (Sep 2, 2025): personally I prefer don't use uv and even less for development: https://github.com/pypa/packaging.python.org/issues/685#issuecomment-1321616748

GiteaMirror commented

2025-11-11 16:50:24 -06:00

@khamaileon commented on GitHub (Sep 2, 2025):

@rgaricano

I see your point about uv and I’m open to alternatives. From your perspective, what would be the best way to avoid having to maintain both pyproject.toml and requirements.txt? That duplication is the main issue I was hoping to address.

I’m also curious about your thoughts on adding pre-commit hooks (for YAML/TOML checks, small formatting fixes, and ruff). Even without uv, this could help us keep things consistent and reduce small errors in day-to-day development.

@khamaileon commented on GitHub (Sep 2, 2025): @rgaricano I see your point about uv and I’m open to alternatives. From your perspective, what would be the best way to avoid having to maintain both pyproject.toml and requirements.txt? That duplication is the main issue I was hoping to address. I’m also curious about your thoughts on adding pre-commit hooks (for YAML/TOML checks, small formatting fixes, and ruff). Even without uv, this could help us keep things consistent and reduce small errors in day-to-day development.

GiteaMirror commented

2025-11-11 16:50:24 -06:00

@Tsafaras commented on GitHub (Oct 15, 2025):

personally I prefer don't use uv and even less for development: pypa/packaging.python.org#685 (comment)

I'm not sure what the state of uv was when that was posted (3 years ago), but it holds no merit today.
If you work on a Python app that relies on uv for managing its dependencies, the best practice is to only have a pyproject.toml file.

@Tsafaras commented on GitHub (Oct 15, 2025): > personally I prefer don't use uv and even less for development: [pypa/packaging.python.org#685 (comment)](https://github.com/pypa/packaging.python.org/issues/685#issuecomment-1321616748) I'm not sure what the state of `uv` was when that was posted (3 years ago), but it holds no merit today. If you work on a Python app that relies on `uv` for managing its dependencies, the [best practice](https://docs.astral.sh/uv/pip/dependencies/) is to _only_ have a `pyproject.toml` file.

GiteaMirror commented

2025-11-11 16:50:25 -06:00

@Tsafaras commented on GitHub (Oct 15, 2025):

I’m also curious about your thoughts on adding pre-commit hooks (for YAML/TOML checks, small formatting fixes, and ruff). Even without uv, this could help us keep things consistent and reduce small errors in day-to-day development.

I'm not sure why that was discarded.

Even though the project does use GitHub Actions for such checks, git hooks have not been set up; whether that's through pre-commit, lefthook, husky, etc.
It leads to people committing changes that fail those checks. Git hooks would prevent that.

@Tsafaras commented on GitHub (Oct 15, 2025): > I’m also curious about your thoughts on adding pre-commit hooks (for YAML/TOML checks, small formatting fixes, and ruff). Even without uv, this could help us keep things consistent and reduce small errors in day-to-day development. I'm not sure why that was discarded. Even though the project does use GitHub Actions for such checks, git hooks have not been set up; whether that's through `pre-commit`, `lefthook`, `husky`, etc. It leads to people committing changes that fail those checks. Git hooks would prevent that.

GiteaMirror commented

2025-11-11 16:50:25 -06:00

@rgaricano commented on GitHub (Oct 16, 2025):

@rgaricano

I see your point about uv and I’m open to alternatives. From your perspective, what would be the best way to avoid having to maintain both pyproject.toml and requirements.txt? That duplication is the main issue I was hoping to address.

I’m also curious about your thoughts on adding pre-commit hooks (for YAML/TOML checks, small formatting fixes, and ruff). Even without uv, this could help us keep things consistent and reduce small errors in day-to-day development.

Syncro script
e.g.

#!/usr/bin/env python3

import toml
import subprocess
import sys

def get_installed_packages():
    """Get list of installed packages and their versions."""
    result = subprocess.run([sys.executable, "-m", "pip", "freeze"], 
                             capture_output=True, text=True)
    if result.returncode != 0:
        print("Error running pip freeze:", result.stderr)
        return {}
    
    return {line.split('==')[0]: line.split('==')[1] for line in result.stdout.splitlines()}

def get_pyproject_dependencies():
    """Extract dependencies from pyproject.toml."""
    try:
        with open('pyproject.toml', 'r') as f:
            pyproject = toml.load(f)
    except FileNotFoundError:
        print("Error: pyproject.toml not found")
        return {}
    
    dependencies = {}
    if 'project' in pyproject and 'dependencies' in pyproject['project']:
        dependencies.update(pyproject['project']['dependencies'])
    return dependencies

def update_requirements_txt(dependencies):
    """Update requirements.txt with the given dependencies."""
    try:
        with open('./backend/requirements.txt', 'w') as f:
            for dep in dependencies:
                f.write(f"{dep}\n")
        print("requirements.txt has been updated.")
    except Exception as e:
        print(f"Error updating requirements.txt: {e}")

def main():
    print("Synchronizing pyproject.toml and requirements.txt...")
    
    # Get dependencies from pyproject.toml
    pyproject_deps = get_pyproject_dependencies()
    if not pyproject_deps:
        print("No dependencies found in pyproject.toml")
        return
    
    # Get currently installed packages
    installed_packages = get_installed_packages()
    
    # Update dependencies list with installed versions
    updated_deps = {pkg: installed_packages.get(pkg, '==') for pkg in pyproject_deps}
    
    # Update requirements.txt
    update_requirements_txt(updated_deps)

if __name__ == "__main__":
    main()
``

@rgaricano commented on GitHub (Oct 16, 2025): > [@rgaricano](https://github.com/rgaricano) > > I see your point about uv and I’m open to alternatives. From your perspective, what would be the best way to avoid having to maintain both pyproject.toml and requirements.txt? That duplication is the main issue I was hoping to address. > > I’m also curious about your thoughts on adding pre-commit hooks (for YAML/TOML checks, small formatting fixes, and ruff). Even without uv, this could help us keep things consistent and reduce small errors in day-to-day development. Syncro script e.g. ``` #!/usr/bin/env python3 import toml import subprocess import sys def get_installed_packages(): """Get list of installed packages and their versions.""" result = subprocess.run([sys.executable, "-m", "pip", "freeze"], capture_output=True, text=True) if result.returncode != 0: print("Error running pip freeze:", result.stderr) return {} return {line.split('==')[0]: line.split('==')[1] for line in result.stdout.splitlines()} def get_pyproject_dependencies(): """Extract dependencies from pyproject.toml.""" try: with open('pyproject.toml', 'r') as f: pyproject = toml.load(f) except FileNotFoundError: print("Error: pyproject.toml not found") return {} dependencies = {} if 'project' in pyproject and 'dependencies' in pyproject['project']: dependencies.update(pyproject['project']['dependencies']) return dependencies def update_requirements_txt(dependencies): """Update requirements.txt with the given dependencies.""" try: with open('./backend/requirements.txt', 'w') as f: for dep in dependencies: f.write(f"{dep}\n") print("requirements.txt has been updated.") except Exception as e: print(f"Error updating requirements.txt: {e}") def main(): print("Synchronizing pyproject.toml and requirements.txt...") # Get dependencies from pyproject.toml pyproject_deps = get_pyproject_dependencies() if not pyproject_deps: print("No dependencies found in pyproject.toml") return # Get currently installed packages installed_packages = get_installed_packages() # Update dependencies list with installed versions updated_deps = {pkg: installed_packages.get(pkg, '==') for pkg in pyproject_deps} # Update requirements.txt update_requirements_txt(updated_deps) if __name__ == "__main__": main() ``

GiteaMirror commented

2025-11-11 16:50:25 -06:00

@Tsafaras commented on GitHub (Oct 16, 2025):

@rgaricano regarding your "syncro script", did you see https://github.com/open-webui/open-webui/issues/17145#issuecomment-3408404924?

@Tsafaras commented on GitHub (Oct 16, 2025): @rgaricano regarding your "syncro script", did you see https://github.com/open-webui/open-webui/issues/17145#issuecomment-3408404924?

GiteaMirror referenced this issue

2026-04-19 20:43:03 -05:00

[GH-ISSUE #6298] Models not working with RAG UNLESS imported via Openwebui's experimental feature (but broken response) #14308

GiteaMirror referenced this issue

2026-04-25 04:14:25 -05:00

[GH-ISSUE #6298] Models not working with RAG UNLESS imported via Openwebui's experimental feature (but broken response) #29836

GiteaMirror referenced this issue

2026-05-05 14:11:54 -05:00

[GH-ISSUE #6298] Models not working with RAG UNLESS imported via Openwebui's experimental feature (but broken response) #52974

GiteaMirror referenced this issue

2026-05-13 01:22:04 -05:00

[GH-ISSUE #6298] Models not working with RAG UNLESS imported via Openwebui's experimental feature (but broken response) #68912