[GH-ISSUE #24255] MLX local model support + external endpoint patterns #58911

New Issue

GiteaMirror · 2026-05-06T00:25:50-05:00

GiteaMirror commented

2026-05-06 00:25:50 -05:00

Originally created by @kstabach on GitHub (Apr 30, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/24255

Context

Advanced local setup using MLX (Apple Silicon) + abliterated models + external harness (LHAI).

Current working architecture:
OpenWebUI → OpenAI-compatible endpoint (mlx_lm.server / vllm-mlx) → MLX models

Problem

OpenWebUI does not have a clear first-class pattern for:

Local MLX model usage
External runtime separation (UI vs inference)
Agent-oriented workflows with strict tool + JSON contracts

What works (documented setup)

Models stored locally (~/models/mlx/...)
MLX server exposes OpenAI-compatible API
OpenWebUI connects via custom endpoint

Gaps

No explicit docs for MLX workflows
No guidance for external runtime vs UI separation
No examples for agent-style usage
No mention of abliterated models and constraints

Proposed Improvements

Official MLX guide
Clear UI → API → Runtime architecture docs
Agent workflow examples (JSON, tools)
Sampling guidance for local models

Takeaway

Treat OpenWebUI as a client, not the runtime.

Meta (for agents)

UI is stateless
Runtime is external
Contracts enforced outside UI

Originally created by @kstabach on GitHub (Apr 30, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/24255 ## Context Advanced local setup using MLX (Apple Silicon) + abliterated models + external harness (LHAI). Current working architecture: OpenWebUI → OpenAI-compatible endpoint (mlx_lm.server / vllm-mlx) → MLX models ## Problem OpenWebUI does not have a clear first-class pattern for: - Local MLX model usage - External runtime separation (UI vs inference) - Agent-oriented workflows with strict tool + JSON contracts ## What works (documented setup) - Models stored locally (~/models/mlx/...) - MLX server exposes OpenAI-compatible API - OpenWebUI connects via custom endpoint ## Gaps 1. No explicit docs for MLX workflows 2. No guidance for external runtime vs UI separation 3. No examples for agent-style usage 4. No mention of abliterated models and constraints ## Proposed Improvements 1. Official MLX guide 2. Clear UI → API → Runtime architecture docs 3. Agent workflow examples (JSON, tools) 4. Sampling guidance for local models ## Takeaway Treat OpenWebUI as a client, not the runtime. ## Meta (for agents) - UI is stateless - Runtime is external - Contracts enforced outside UI

GiteaMirror closed this issue

2026-05-06 00:25:53 -05:00

GiteaMirror commented

2026-05-06 00:25:58 -05:00

@pr-validator-bot commented on GitHub (Apr 30, 2026):

⚠️ Missing Issue Title Prefix

@kstabach, your issue title is missing a prefix (e.g., bug:, feat:, docs:).

Please update your issue title to include one of the following prefixes:

bug: Bug report or error you've encountered
feat: Feature request or enhancement suggestion
docs: Documentation issue or improvement request
question: Question about usage or functionality
help: Request for help or support

Example: bug: Login fails when using special characters in password

@pr-validator-bot commented on GitHub (Apr 30, 2026): # ⚠️ Missing Issue Title Prefix @kstabach, your issue title is missing a prefix (e.g., `bug:`, `feat:`, `docs:`). Please update your issue title to include one of the following prefixes: - **bug**: Bug report or error you've encountered - **feat**: Feature request or enhancement suggestion - **docs**: Documentation issue or improvement request - **question**: Question about usage or functionality - **help**: Request for help or support Example: `bug: Login fails when using special characters in password`

GiteaMirror commented

2026-05-06 00:26:00 -05:00

@kstabach commented on GitHub (Apr 30, 2026):

Closing — filed in wrong repo. Moving to internal LHAI tracking.

@kstabach commented on GitHub (Apr 30, 2026): Closing — filed in wrong repo. Moving to internal LHAI tracking.

GiteaMirror commented

2026-05-06 00:26:03 -05:00

@kstabach commented on GitHub (Apr 30, 2026):

Leaving this here for context, but closing loop:\n\nThis was filed from a local agent/MLX workflow perspective and ended up being better tracked in my own stack (LHAI).\n\nThe core point still stands for anyone finding this:\n\n- OpenWebUI works best as a client\n- Local inference should live behind an OpenAI-compatible endpoint (MLX, vLLM, etc.)\n- Agent-style workflows (JSON, tools, strict schemas) are enforced outside the UI\n\nClosing internally and continuing iteration there. Thanks for the project — this is the best UI in this space right now.

@kstabach commented on GitHub (Apr 30, 2026): Leaving this here for context, but closing loop:\n\nThis was filed from a local agent/MLX workflow perspective and ended up being better tracked in my own stack (LHAI).\n\nThe core point still stands for anyone finding this:\n\n- OpenWebUI works best as a **client**\n- Local inference should live behind an **OpenAI-compatible endpoint** (MLX, vLLM, etc.)\n- Agent-style workflows (JSON, tools, strict schemas) are enforced outside the UI\n\nClosing internally and continuing iteration there. Thanks for the project — this is the best UI in this space right now.

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#58911