Not sure if "bug" or "feature" really, but need the ability to load llm models in an air gapped environment #1651

New Issue

GiteaMirror · 2025-11-11T14:49:24-06:00

GiteaMirror commented

2025-11-11 14:49:24 -06:00

Originally created by @mitchty on GitHub (Jul 31, 2024).

Is your feature request related to a problem? Please describe.

In an isolated network that has no internet access. There needs to be some ability or documented way to get llm data loaded without having to get at data on the public internet.

This would also help for reduction in bandwidth needed for new installs by enabling offloading/caching of data.

Another help would be for supply chain security in ensuring that the models are always obtained from a trusted source or location through some vetting process.

Describe the solution you'd like
Ideally a way to via an api call "freeze/tar/zip" any number of models/versions installed and setup in open-webui then to load that data.

Describe alternatives you've considered
I've been digging through the ollama and open-webui code to get a better idea of how I might accomplish this. My only possible thoughts are to tar the models after installation on a system with access and then to somehow restore that after installing it elsewhere. But better options would be appreciated or advice on maybe other ways to accomplish the goal.

Advice on how I might approach this appreciated.
Additional context
Add any other context or screenshots about the feature request here.

Originally created by @mitchty on GitHub (Jul 31, 2024). **Is your feature request related to a problem? Please describe.** In an isolated network that has no internet access. There needs to be some ability or documented way to get llm data loaded without having to get at data on the public internet. This would also help for reduction in bandwidth needed for new installs by enabling offloading/caching of data. Another help would be for supply chain security in ensuring that the models are always obtained from a trusted source or location through some vetting process. **Describe the solution you'd like** Ideally a way to via an api call "freeze/tar/zip" any number of models/versions installed and setup in open-webui then to load that data. **Describe alternatives you've considered** I've been digging through the ollama and open-webui code to get a better idea of how I might accomplish this. My only possible thoughts are to tar the models after installation on a system with access and then to somehow restore that after installing it elsewhere. But better options would be appreciated or advice on maybe other ways to accomplish the goal. Advice on how I might approach this appreciated. **Additional context** Add any other context or screenshots about the feature request here.

GiteaMirror closed this issue

2025-11-11 14:49:24 -06:00

GiteaMirror commented

2025-11-11 14:49:25 -06:00

@dmessano commented on GitHub (Jul 31, 2024):

We also have a need for a truly air-gaped environment, and will be following this

Our current process is:

Spin up Ollama on Docker with /root/.ollama dir mapped to local directory
Download model on Docker Ollama
tar models directory
Transfer to air-gap network
Extract tar and kubectl cp the blobs and manifest directory to the Ollama pod /root/.ollama/models/ in Kubernetes

This is not ideal, but has worked 100% for us

We tried the GGUF upload in Open-webui with mixed results

@dmessano commented on GitHub (Jul 31, 2024): We also have a need for a truly air-gaped environment, and will be following this Our current process is: 1. Spin up Ollama on Docker with `/root/.ollama` dir mapped to local directory 2. Download model on Docker Ollama 3. tar `models` directory 4. Transfer to air-gap network 5. Extract tar and `kubectl cp` the `blobs` and `manifest` directory to the Ollama pod `/root/.ollama/models/` in Kubernetes This is not ideal, but has worked 100% for us We tried the GGUF upload in Open-webui with mixed results