github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-05 23:53:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[PR #15166] Draft: api: support multimodal embedding #15060

New Issue
Open
opened 2026-04-13 01:09:32 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-04-13 01:09:32 -05:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/ollama/ollama/pull/15166
Author: @jclab-joseph
Created: 3/31/2026
Status: 🔄 Open

Base: main ← Head: feat/multimodal-embedding

📝 Commits (3)

  • 2847376 api: support multimodal embedding
  • dca9efb docs: add multimodal request
  • 0fbe120 llamarunner: support image embedding

📊 Changes

13 files changed (+894 additions, -21 deletions)

View changed files

📝 api/types.go (+9 -0)
➕ cohere/cohere.go (+387 -0)
📝 docs/api.md (+107 -0)
📝 llm/server.go (+5 -4)
➕ middleware/cohere.go (+94 -0)
➕ middleware/cohere_test.go (+171 -0)
📝 runner/llamarunner/runner.go (+10 -1)
📝 runner/ollamarunner/runner.go (+40 -1)
📝 server/routes.go (+25 -12)
📝 server/routes_cloud_test.go (+43 -0)
📝 server/sched_test.go (+1 -1)
📝 x/imagegen/server.go (+1 -1)
📝 x/mlxrunner/client.go (+1 -1)

📄 Description

Fix https://github.com/ollama/ollama/issues/4296

This is the latest implementation of https://github.com/ollama/ollama/pull/10728.

It supports the cohere embedding API (/v2/embed).

It is not working yet. because of the runner does not support the multimodal embedding model.

Related:

Updated Docs

Generate Embedding

Note: this endpoint has been superseded by /api/embed

POST /api/embeddings

Generate embeddings from a model

Parameters

  • model: name of model to generate embeddings from
  • prompt: text to generate embeddings for

Advanced parameters:

  • options: additional model parameters listed in the documentation for the Modelfile such as temperature
  • keep_alive: controls how long the model will stay loaded into memory following the request (default: 5m)

Examples

Request

curl http://localhost:11434/api/embed -d '{
  "model": "llava",
  "inputs": [
    {
      "text": "Describe this image for retrieval",
      "image": "<base64-encoded-image>"
    }
  ]
}'

Generate Embeddings (Cohere-compatible)

POST /v2/embed

Generate embeddings using a Cohere-compatible API.

Parameters

  • model: name of model to generate embeddings from
  • texts: optional list of text inputs
  • images: optional list of base64-encoded image inputs
  • inputs: optional multimodal list of inputs using Cohere content blocks
  • embedding_types: optional list of output formats. Supported values: float, base64
  • output_dimension: optional output embedding dimension
  • truncate: truncation strategy. Supported values: END and NONE

Notes:

  • Provide exactly one of texts, images, or inputs
  • image_url.url currently supports base64 data URIs, not remote URLs
  • inputs supports text and image_url content blocks

Examples

Request (Text)

curl http://localhost:11434/v2/embed -d '{
  "model": "all-minilm",
  "texts": ["Why is the sky blue?"],
  "embedding_types": ["float"]
}'

Response

{
  "id": "embed-response",
  "embeddings": {
    "float": [
      [
        0.010071029, -0.0017594862, 0.05007221, 0.04692972, 0.054916814,
        0.008599704, 0.105441414, -0.025878139, 0.12958129, 0.031952348
      ]
    ]
  },
  "texts": ["Why is the sky blue?"],
  "meta": {
    "api_version": {
      "version": "2"
    },
    "billed_units": {
      "input_tokens": 8
    }
  },
  "response_type": "embeddings_by_type"
}

Request (Multimodal)

curl http://localhost:11434/v2/embed -d '{
  "model": "llava",
  "inputs": [
    {
      "content": [
        {
          "type": "text",
          "text": "Describe this image for retrieval"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/png;base64,<base64-encoded-image>"
          }
        }
      ]
    }
  ],
  "embedding_types": ["float", "base64"]
}'

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/ollama/ollama/pull/15166 **Author:** [@jclab-joseph](https://github.com/jclab-joseph) **Created:** 3/31/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/multimodal-embedding` --- ### 📝 Commits (3) - [`2847376`](https://github.com/ollama/ollama/commit/2847376790b0fb3728ef7b735a41b8ad328fa3fc) api: support multimodal embedding - [`dca9efb`](https://github.com/ollama/ollama/commit/dca9efbac65a4b7aa2433d51280924c55a05bf35) docs: add multimodal request - [`0fbe120`](https://github.com/ollama/ollama/commit/0fbe1204ce23d6d86fe22d768cacf04295b0ec81) llamarunner: support image embedding ### 📊 Changes **13 files changed** (+894 additions, -21 deletions) <details> <summary>View changed files</summary> 📝 `api/types.go` (+9 -0) ➕ `cohere/cohere.go` (+387 -0) 📝 `docs/api.md` (+107 -0) 📝 `llm/server.go` (+5 -4) ➕ `middleware/cohere.go` (+94 -0) ➕ `middleware/cohere_test.go` (+171 -0) 📝 `runner/llamarunner/runner.go` (+10 -1) 📝 `runner/ollamarunner/runner.go` (+40 -1) 📝 `server/routes.go` (+25 -12) 📝 `server/routes_cloud_test.go` (+43 -0) 📝 `server/sched_test.go` (+1 -1) 📝 `x/imagegen/server.go` (+1 -1) 📝 `x/mlxrunner/client.go` (+1 -1) </details> ### 📄 Description Fix https://github.com/ollama/ollama/issues/4296 This is the latest implementation of https://github.com/ollama/ollama/pull/10728. It supports the cohere embedding API (/v2/embed). It is not working yet. because of the runner does not support the multimodal embedding model. Related: - https://github.com/ollama/ollama/issues/5304 --- # Updated Docs ## Generate Embedding > Note: this endpoint has been superseded by `/api/embed` ``` POST /api/embeddings ``` Generate embeddings from a model ### Parameters - `model`: name of model to generate embeddings from - `prompt`: text to generate embeddings for Advanced parameters: - `options`: additional model parameters listed in the documentation for the [Modelfile](./modelfile.mdx#valid-parameters-and-values) such as `temperature` - `keep_alive`: controls how long the model will stay loaded into memory following the request (default: `5m`) ### Examples #### Request ```shell curl http://localhost:11434/api/embed -d '{ "model": "llava", "inputs": [ { "text": "Describe this image for retrieval", "image": "<base64-encoded-image>" } ] }' ``` ## Generate Embeddings (Cohere-compatible) ``` POST /v2/embed ``` Generate embeddings using a Cohere-compatible API. ### Parameters - `model`: name of model to generate embeddings from - `texts`: optional list of text inputs - `images`: optional list of base64-encoded image inputs - `inputs`: optional multimodal list of inputs using Cohere content blocks - `embedding_types`: optional list of output formats. Supported values: `float`, `base64` - `output_dimension`: optional output embedding dimension - `truncate`: truncation strategy. Supported values: `END` and `NONE` Notes: - Provide exactly one of `texts`, `images`, or `inputs` - `image_url.url` currently supports base64 data URIs, not remote URLs - `inputs` supports `text` and `image_url` content blocks ### Examples #### Request (Text) ```shell curl http://localhost:11434/v2/embed -d '{ "model": "all-minilm", "texts": ["Why is the sky blue?"], "embedding_types": ["float"] }' ``` #### Response ```json { "id": "embed-response", "embeddings": { "float": [ [ 0.010071029, -0.0017594862, 0.05007221, 0.04692972, 0.054916814, 0.008599704, 0.105441414, -0.025878139, 0.12958129, 0.031952348 ] ] }, "texts": ["Why is the sky blue?"], "meta": { "api_version": { "version": "2" }, "billed_units": { "input_tokens": 8 } }, "response_type": "embeddings_by_type" } ``` #### Request (Multimodal) ```shell curl http://localhost:11434/v2/embed -d '{ "model": "llava", "inputs": [ { "content": [ { "type": "text", "text": "Describe this image for retrieval" }, { "type": "image_url", "image_url": { "url": "data:image/png;base64,<base64-encoded-image>" } } ] } ], "embedding_types": ["float", "base64"] }' ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-04-13 01:09:32 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#15060
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?