github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-08 00:51:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[PR #1082] [CLOSED] Add support for Multimodel models #15732

New Issue
Closed
opened 2026-04-16 05:06:15 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-04-16 05:06:15 -05:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/ollama/ollama/pull/1082
Author: @mattapperson
Created: 11/11/2023
Status: ❌ Closed

Base: main ← Head: multimodel

📝 Commits (4)

📊 Changes

12 files changed (+278 additions, -144 deletions)

View changed files

📝 api/types.go (+60 -29)
📝 cmd/cmd.go (+138 -12)
📝 docs/modelfile.md (+2 -0)
📝 llm/llama.cpp/generate_darwin_amd64.go (+0 -1)
📝 llm/llama.cpp/generate_darwin_arm64.go (+0 -1)
📝 llm/llama.cpp/gguf (+1 -1)
➖ llm/llama.cpp/patches/0001-metal-handle-ggml_scale-for-n-4-0-close-3754.patch (+0 -91)
📝 llm/llama.go (+11 -2)
📝 llm/llm.go (+4 -4)
📝 parser/parser.go (+3 -0)
📝 server/images.go (+33 -0)
📝 server/routes.go (+26 -3)

📄 Description

Interactive cli usage:

/set image add <image id int> <path to image file>
Please tell me what text is in this photo [img-<image id int>]

For the API I added support for the image_data prop with the type of []{id: int, data: string(base64)}for the generate endpoint.

To support this, modelfile now has a MMPROJ key that points to the mmproj file path.

Tested with the following modelfile:

FROM ./ggml-model-q4_k.gguf

TEMPLATE """
USER:{{ .Prompt }}
ASSISTANT:
"""

MMPROJ ./mmproj-model-f16.gguf

And using the following pre-quantatized model - https://huggingface.co/mys/ggml_llava-v1.5-13b.

Screenshot 2023-11-09 at 11 30 10 PM screenshot

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/ollama/ollama/pull/1082 **Author:** [@mattapperson](https://github.com/mattapperson) **Created:** 11/11/2023 **Status:** ❌ Closed **Base:** `main` ← **Head:** `multimodel` --- ### 📝 Commits (4) - [`037a28b`](https://github.com/ollama/ollama/commit/037a28baf30a440b6cc0d5208dfc5f2c3118c753) llama.cpp update - [`0e050a2`](https://github.com/ollama/ollama/commit/0e050a261dc83928895dc66aaa578f60093a2745) working - [`5826b16`](https://github.com/ollama/ollama/commit/5826b16f7fadf70dad71e821dc87fda860a7d2e6) Merge branch 'main' of https://github.com/jmorganca/ollama into multimodel - [`840d651`](https://github.com/ollama/ollama/commit/840d6518cfcb07e51c2e2ef946810d28f539cc11) Update docs/modelfile.md ### 📊 Changes **12 files changed** (+278 additions, -144 deletions) <details> <summary>View changed files</summary> 📝 `api/types.go` (+60 -29) 📝 `cmd/cmd.go` (+138 -12) 📝 `docs/modelfile.md` (+2 -0) 📝 `llm/llama.cpp/generate_darwin_amd64.go` (+0 -1) 📝 `llm/llama.cpp/generate_darwin_arm64.go` (+0 -1) 📝 `llm/llama.cpp/gguf` (+1 -1) ➖ `llm/llama.cpp/patches/0001-metal-handle-ggml_scale-for-n-4-0-close-3754.patch` (+0 -91) 📝 `llm/llama.go` (+11 -2) 📝 `llm/llm.go` (+4 -4) 📝 `parser/parser.go` (+3 -0) 📝 `server/images.go` (+33 -0) 📝 `server/routes.go` (+26 -3) </details> ### 📄 Description Interactive cli usage: ```bash /set image add <image id int> <path to image file> Please tell me what text is in this photo [img-<image id int>] ``` For the API I added support for the `image_data` prop with the type of `[]{id: int, data: string(base64)}`for the generate endpoint. To support this, modelfile now has a `MMPROJ` key that points to the mmproj file path. Tested with the following modelfile: ``` FROM ./ggml-model-q4_k.gguf TEMPLATE """ USER:{{ .Prompt }} ASSISTANT: """ MMPROJ ./mmproj-model-f16.gguf ``` And using the following pre-quantatized model - https://huggingface.co/mys/ggml_llava-v1.5-13b. <img width="1226" alt="Screenshot 2023-11-09 at 11 30 10 PM" src="https://github.com/jmorganca/ollama/assets/192433/ec62e5e7-999b-44f2-9a32-e6b85ca5b404"> <img width="567" alt="screenshot" src="https://github.com/jmorganca/ollama/assets/192433/681c5cd5-9140-43b1-8383-08823cdbb002"> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-04-16 05:06:15 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#15732
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?