github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 20:54:54 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #1322] ❔ Run a given LLM/model within docker/podman/cloud run 👶 #686

New Issue
Closed
opened 2026-04-12 10:21:59 -05:00 by GiteaMirror · 10 comments
GiteaMirror commented 2026-04-12 10:21:59 -05:00
Owner
Copy Link

Originally created by @adriens on GitHub (Nov 29, 2023).
Original GitHub issue: https://github.com/ollama/ollama/issues/1322

❔ About

I intend to deploy a given LLM (out of the box or customized) thanks to ollama (let's say https://ollama.ai/library/mistral), then call its APIs from various clients.

Here would be an example of my scenario:

  1. Build/customize the image that runs mistral (for example) within ollama
  2. Push the image on any private docker registry (let's says ghcr or dockerhub)
  3. Pull & run the image (local podman on workstation, GCP Google Cloud Run) so the boot the API
  4. (Optional) : publish the resulting API through on any API Management (Google APIGEE in my case, but could also land on Tyk, Kong,...)
  5. Call it from any client (Java/Python, js) so we can build integrations and release next-gen (llama_index or langchain based) solutions (under the hood)

🎯 The question

👉 The question may be answered by providing (or pointing) a howto as for now I did not find a set-by-step (for dummies like) tutorial.

🙏 Thank you in advance for any help or feedback.

🎟️ Related contents

Originally created by @adriens on GitHub (Nov 29, 2023). Original GitHub issue: https://github.com/ollama/ollama/issues/1322 # :grey_question: About I intend to deploy a given LLM (out of the box or customized) thanks to `ollama` (let's say `https://ollama.ai/library/mistral`), then call its APIs from various clients. Here would be an example of my scenario: 1. **Build/customize the image** that runs mistral (for example) within ollama 2. **Push the image on any private docker** registry (let's says `ghcr` or dockerhub) 3. **Pull & run the image** (local [podman](https://podman.io/) on workstation, [GCP Google Cloud Run](https://cloud.google.com/run)) so the boot the API 4. **(Optional) : publish the resulting API through on any API Management** ([Google APIGEE](https://cloud.google.com/apigee/docs/api-platform/get-started/what-apigee) in my case, but could also land on [Tyk](https://tyk.io/), [Kong](https://konghq.com/products/kong-gateway),...) 5. **Call it from any client (Java/Python, js)** so we can build integrations and release next-gen ([llama_index](https://www.llamaindex.ai/) or [langchain](https://www.langchain.com/) based) solutions (under the hood) # :dart: The question :point_right: The question may be answered by **providing (or pointing) a howto** as for now I did not find a set-by-step (for dummies like) tutorial. :pray: Thank you in advance for any help or feedback. # :tickets: Related contents - https://github.com/jmorganca/ollama/issues/1318 - [🦙 Mistral 7B & Ollama: LLMs 💏 Apache 2.0 Open Source on small hardwares](https://dev.to/adriens/mistral-7b-ollama-llms-apache-20-open-source-on-small-hardwares-3ln3) - https://github.com/jmorganca/ollama/issues/546
GiteaMirror added the feature request label 2026-04-12 10:21:59 -05:00
GiteaMirror commented 2026-04-12 10:22:01 -05:00
Author
Owner
Copy Link

@technovangelist commented on GitHub (Dec 4, 2023):

I am not sure what the question is here. Can you clarify? If you you looking for the docker image check out https://hub.docker.com/r/ollama/ollama. There are a number of docs in the docs folder in the repo, and examples in the examples folder which are a good place to start.

<!-- gh-comment-id:1839575237 --> @technovangelist commented on GitHub (Dec 4, 2023): I am not sure what the question is here. Can you clarify? If you you looking for the docker image check out https://hub.docker.com/r/ollama/ollama. There are a number of docs in the docs folder in the repo, and examples in the examples folder which are a good place to start.
GiteaMirror commented 2026-04-12 10:22:02 -05:00
Author
Owner
Copy Link

@adriens commented on GitHub (Dec 6, 2023):

Hi @technovangelist , thanks a lot for your answer.

What I'm looking for is to be able to pull and run a ready to use ollama instance (with let's say neural-chat.
Without any docker compose not additional conf.

A the end it may look like:

docker pull ollama/ollama-neural-chat

Where the image could come from any public registry, or private.

I could not find the proper way to achieve this simply and properly.

I could make a customer Dockerfile, then install ollama, then pull the requested image then boot the whole but I'm about sure something cleaner could be achievied... like pulling existing prebuilt ones 💭

<!-- gh-comment-id:1841990531 --> @adriens commented on GitHub (Dec 6, 2023): Hi @technovangelist , thanks a lot for your answer. What I'm looking for is to be able to pull and run a ready to use ollama instance (with let's say [`neural-chat`](https://ollama.ai/library/neural-chat). Without any docker compose not additional conf. A the end it may look like: ```shell docker pull ollama/ollama-neural-chat ``` Where the image could come from any public registry, or private. I could not find the proper way to achieve this simply and properly. I could make a customer Dockerfile, then install ollama, then pull the requested image then boot the whole but I'm about sure something cleaner could be achievied... like pulling existing prebuilt ones 💭
GiteaMirror commented 2026-04-12 10:22:02 -05:00
Author
Owner
Copy Link

@adriens commented on GitHub (Dec 6, 2023):

... so :

docker exec -it ollama ollama run llama2

would become something like:

docker exec -d ollama-llama2
<!-- gh-comment-id:1841992004 --> @adriens commented on GitHub (Dec 6, 2023): ... so : ```shell docker exec -it ollama ollama run llama2 ``` would become something like: ```shell docker exec -d ollama-llama2 ```
GiteaMirror commented 2026-04-12 10:22:02 -05:00
Author
Owner
Copy Link

@technovangelist commented on GitHub (Dec 9, 2023):

So are you looking for that to be dynamic, or do you just want some guidance on adding a model to a new docker image that you can push to dockerhub. So a user can run it and it already has the model? I'm not sure how well that would work. I think downloading from us after grabbing the container from docker hub, is going to be faster than grabbing a container with the model already in it from docker hub. I think I may still be missing what you are asking.

<!-- gh-comment-id:1848069053 --> @technovangelist commented on GitHub (Dec 9, 2023): So are you looking for that to be dynamic, or do you just want some guidance on adding a model to a new docker image that you can push to dockerhub. So a user can run it and it already has the model? I'm not sure how well that would work. I think downloading from us after grabbing the container from docker hub, is going to be faster than grabbing a container with the model already in it from docker hub. I think I may still be missing what you are asking.
GiteaMirror commented 2026-04-12 10:22:03 -05:00
Author
Owner
Copy Link

@adriens commented on GitHub (Dec 9, 2023):

Hi, @technovangelist , thanks a lot for taking care of my user feature requests.

So are you looking for that to be dynamic, or do you just want some guidance on adding a model to a new docker image that you can push to dockerhub. So a user can run it and it already has the model?

Yup, that's exactly what I would like to achieve:

  • as a end user : be able to pull a ready-to-use/boot image that I could boot then query through ollama rationalized API without any further action (the end user)
  • as image producer/AI dev : be able to build/push a ready to use image on any public or private registry (quay, dockerhub, ghcr,...)

So a user can run it and it already has the model?

Yes, that's it.

I'm not sure how well that would work. I think downloading from us after grabbing the container from docker hub, is going to be faster than grabbing a container with the model already in it from docker hub.

Hmmmm, that's a very good point as well as space consumption on registries 💭

So, according to your remark, I may ask the following slightly changed question:

👉 How to publish a lighweight ready to use image (but parametrized with a specific llm like mistral)... but which may download the target model at boot time so we rather spend time and disk space on the target host rather than on other places

I think I may still be missing what you are asking.

No, I think, you really got it, your remarks help a lot.

Here is what it may look like from the end-user point of view :

podman run -d --name ollama-mistral -p 8080:8080 my-org/ollama-mistral:latest

or (maybe better with uing tags ? 🤔 )

podman run -d --name ollama-mistral -p 8080:8080 my-org/ollama:mistral

Then...

  1. Conteneur boots up
  2. Conteneur check if image i up-to-date... then downloads requested model at runtime
  3. Image is locally stored in host local registry
  4. ollama api is up and running

Does this scenario make sense to you or do I imagine it the wrong way ?

💭 I need to achieve it this way so I can match existing scenario (Tower) deployments... and then use open source LLM onPrem as almost any other image.

<!-- gh-comment-id:1848767399 --> @adriens commented on GitHub (Dec 9, 2023): Hi, @technovangelist , thanks a lot for taking care of my user feature requests. > So are you looking for that to be dynamic, or do you just want some guidance on adding a model to a new docker image that you can push to dockerhub. So a user can run it and it already has the model? Yup, that's exactly what I would like to achieve: - **as a end user** : be able to pull a ready-to-use/boot image that I could boot then query through ollama rationalized API without any further action (the end user) - **as image producer/AI dev** : be able to build/push a ready to use image on any public or private registry (quay, dockerhub, ghcr,...) > So a user can run it and it already has the model? Yes, that's it. > I'm not sure how well that would work. I think downloading from us after grabbing the container from docker hub, is going to be faster than grabbing a container with the model already in it from docker hub. Hmmmm, that's a very good point as well as space consumption on registries :thought_balloon: So, according to your remark, I may ask the following slightly changed question: :point_right: _How to publish a lighweight ready to use image (but parametrized with a specific llm like `mistral`)... but which may download the target model at boot time so we rather spend time and disk space on the target host rather than on other places_ > I think I may still be missing what you are asking. No, I think, you really got it, your remarks help a lot. Here is what it may look like from the end-user point of view : ```shell podman run -d --name ollama-mistral -p 8080:8080 my-org/ollama-mistral:latest ``` or (maybe better with uing tags ? :thinking: ) ```shell podman run -d --name ollama-mistral -p 8080:8080 my-org/ollama:mistral ``` Then... 1. Conteneur boots up 2. Conteneur check if image i up-to-date... then downloads requested model at runtime 3. Image is locally stored in host local registry 4. ollama api is up and running Does this scenario make sense to you or do I imagine it the wrong way ? :thought_balloon: I need to achieve it this way so I can match existing scenario (`Tower`) deployments... and then use open source LLM onPrem as almost any other image.
GiteaMirror commented 2026-04-12 10:22:03 -05:00
Author
Owner
Copy Link

@technovangelist commented on GitHub (Jan 3, 2024):

Ok, thanks for the clarification. I'll add an enhancement label to this so we can refer to as we go along. Thanks so much for being a great part of this amazing community.

<!-- gh-comment-id:1875736951 --> @technovangelist commented on GitHub (Jan 3, 2024): Ok, thanks for the clarification. I'll add an enhancement label to this so we can refer to as we go along. Thanks so much for being a great part of this amazing community.
GiteaMirror commented 2026-04-12 10:22:03 -05:00
Author
Owner
Copy Link

@adriens commented on GitHub (Jan 3, 2024):

Yup, and thanks to you for making possible, on open source 🤩

Btw, the base model should be downloaded at contener boot time so we don't fullfill our registry 💭 (and save money 💸 )

<!-- gh-comment-id:1876045052 --> @adriens commented on GitHub (Jan 3, 2024): Yup, and thanks to you for making possible, on open source :star_struck: Btw, the base model should be downloaded at contener boot time so we don't fullfill our registry :thought_balloon: (and save money :money_with_wings: )
GiteaMirror commented 2026-04-12 10:22:04 -05:00
Author
Owner
Copy Link

@adriens commented on GitHub (Feb 27, 2024):

🎦 How to run Ollama on Docker, by Matt Williams

<!-- gh-comment-id:1967487968 --> @adriens commented on GitHub (Feb 27, 2024): [:cinema: How to run Ollama on Docker,](https://www.youtube.com/watch?v=ZoxJcPkjirs) by [Matt Williams](https://www.youtube.com/@technovangelist)
GiteaMirror commented 2026-04-12 10:22:04 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (May 9, 2024):

Hi @adriens thanks for the issue. The ollama/ollama Docker image can be used as a base image for which you can download models in and then save a new image - I think this would be the easiest approach at the moment

<!-- gh-comment-id:2103568005 --> @jmorganca commented on GitHub (May 9, 2024): Hi @adriens thanks for the issue. The `ollama/ollama` [Docker image](https://hub.docker.com/r/ollama/ollama) can be used as a base image for which you can download models in and then save a new image - I think this would be the easiest approach at the moment
GiteaMirror commented 2026-04-12 10:22:04 -05:00
Author
Owner
Copy Link

@adriens commented on GitHub (May 9, 2024):

👍 Would you share any Dockerfile that does the job and the client side call ?

<!-- gh-comment-id:2103595511 --> @adriens commented on GitHub (May 9, 2024): :+1: Would you share any Dockerfile that does the job and the client side call ?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#686
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?