[GH-ISSUE #653] external model connection using litellm #131890

Closed
opened 2026-05-24 19:21:41 -05:00 by GiteaMirror · 1 comment
Owner

Originally created by @mafrasiabi on GitHub (Feb 6, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/653

Hi @justinh-rahb

I've been struggling to make the litellm work and I was wondering if you could help with some details.

here is my config.yaml file:

model_list:

  • model_name: mistral-7b
    litellm_params:
    model: ollama/llama2
    api_base: http://localhost:11434/api
    headers: {
    "HTTP-Referer": "litellm.ai",
    "X-Title": "LiteLLM Server"
    }
  • model_name: mistral-small
    litellm_params:
    model: mistral/mistral-small
    api_base: https://api.mistral.ai/v1
    api_key: "K"
  • model_name: mistral-medium
    litellm_params:
    model: mistral/mistral-medium
    api_base: https://api.mistral.ai/v1
    api_key: "K"

and the docker-compose.yaml file I'm using:
version: '3.8'

services:
ollama:
volumes:
- ollama:/root/.ollama
container_name: ollama
pull_policy: always
tty: true
restart: unless-stopped
image: ollama/ollama:latest

ollama-webui:
build:
context: .
args:
OLLAMA_API_BASE_URL: '/ollama/api'
dockerfile: Dockerfile
image: ghcr.io/ollama-webui/ollama-webui:main
container_name: ollama-webui
volumes:
- ollama-webui:/app/backend/data
depends_on:
- ollama
ports:
- ${OLLAMA_WEBUI_PORT-3000}:8080
environment:
- 'OLLAMA_API_BASE_URL=http://ollama:11434/api'
# - "OLLAMA_API_BASE_URL=${OLLAMA_API_BASE_URL}"
- "OPENAI_API_BASE_URL=http://litellm:8000/v1"
- "OPENAI_API_KEY=sk-123456"
- 'WEBUI_SECRET_KEY='
extra_hosts:
- host.docker.internal:host-gateway
restart: unless-stopped

litellm:
image: ghcr.io/berriai/litellm:main-latest
environment:
- "MASTER_KEY=sk-123456"
- "OPENAI_API_KEY=sk-123456"
- "MISTRAL_API_KEY="
- "ANTHROPIC_API_KEY="
ports:
- 4000:8000
volumes:
- ./litellm/config.yaml:/app/config.yaml
command: [ "--config", "/app/config.yaml", "--port", "8000" ]
restart: unless-stopped

volumes:
ollama: {}
ollama-webui: {}

The problem is with when I try to use ollama via litellm. that is where I get below error which I think it has somethig to do with api-base of ollama :
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 170, in try_connect
stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 2258, in connect_tcp
await get_running_loop().create_connection(
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1065, in create_connection
raise exceptions[0]
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1050, in create_connection
sock = await self._connect_sock(
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 961, in _connect_sock
await self.sock_connect(sock, address)
File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 500, in sock_connect
return await fut
File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('0.0.0.0', 11434)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
yield
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 114, in connect_tcp
stream: anyio.abc.ByteStream = await anyio.connect_tcp(
File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 232, in connect_tcp
raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
yield
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 268, in handle_async_request
raise exc
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 251, in handle_async_request
response = await connection.handle_async_request(request)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request
raise exc
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request
stream = await self._connect(request)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 124, in _connect
stream = await self._network_backend.connect_tcp(**kwargs)
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/auto.py", line 30, in connect_tcp
return await self._backend.connect_tcp(
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 121, in connect_tcp
stream._raw_socket.setsockopt(*option) # type: ignore[attr-defined] # pragma: no cover
File "/usr/local/lib/python3.9/contextlib.py", line 137, in exit
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/litellm/llms/ollama.py", line 252, in ollama_async_streaming
async with client.stream(
File "/usr/local/lib/python3.9/contextlib.py", line 181, in aenter
return await self.gen.anext()
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1573, in stream
response = await self.send(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1617, in send
response = await self._send_handling_auth(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1719, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.9/contextlib.py", line 137, in exit
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed

Am I doing something wrong?

Originally created by @mafrasiabi on GitHub (Feb 6, 2024). Original GitHub issue: https://github.com/open-webui/open-webui/issues/653 Hi @justinh-rahb I've been struggling to make the litellm work and I was wondering if you could help with some details. here is my config.yaml file: model_list: - model_name: mistral-7b litellm_params: model: ollama/llama2 api_base: http://localhost:11434/api headers: { "HTTP-Referer": "litellm.ai", "X-Title": "LiteLLM Server" } - model_name: mistral-small litellm_params: model: mistral/mistral-small api_base: https://api.mistral.ai/v1 api_key: "K" - model_name: mistral-medium litellm_params: model: mistral/mistral-medium api_base: https://api.mistral.ai/v1 api_key: "K" and the docker-compose.yaml file I'm using: version: '3.8' services: ollama: volumes: - ollama:/root/.ollama container_name: ollama pull_policy: always tty: true restart: unless-stopped image: ollama/ollama:latest ollama-webui: build: context: . args: OLLAMA_API_BASE_URL: '/ollama/api' dockerfile: Dockerfile image: ghcr.io/ollama-webui/ollama-webui:main container_name: ollama-webui volumes: - ollama-webui:/app/backend/data depends_on: - ollama ports: - ${OLLAMA_WEBUI_PORT-3000}:8080 environment: - 'OLLAMA_API_BASE_URL=http://ollama:11434/api' # - "OLLAMA_API_BASE_URL=${OLLAMA_API_BASE_URL}" - "OPENAI_API_BASE_URL=http://litellm:8000/v1" - "OPENAI_API_KEY=sk-123456" - 'WEBUI_SECRET_KEY=' extra_hosts: - host.docker.internal:host-gateway restart: unless-stopped litellm: image: ghcr.io/berriai/litellm:main-latest environment: - "MASTER_KEY=sk-123456" - "OPENAI_API_KEY=sk-123456" - "MISTRAL_API_KEY=" - "ANTHROPIC_API_KEY=" ports: - 4000:8000 volumes: - ./litellm/config.yaml:/app/config.yaml command: [ "--config", "/app/config.yaml", "--port", "8000" ] restart: unless-stopped volumes: ollama: {} ollama-webui: {} The problem is with when I try to use ollama via litellm. that is where I get below error which I think it has somethig to do with api-base of ollama : Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 170, in try_connect stream = await asynclib.connect_tcp(remote_host, remote_port, local_address) File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 2258, in connect_tcp await get_running_loop().create_connection( File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1065, in create_connection raise exceptions[0] File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1050, in create_connection sock = await self._connect_sock( File "/usr/local/lib/python3.9/asyncio/base_events.py", line 961, in _connect_sock await self.sock_connect(sock, address) File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 500, in sock_connect return await fut File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb raise OSError(err, f'Connect call failed {address}') ConnectionRefusedError: [Errno 111] Connect call failed ('0.0.0.0', 11434) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions yield File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 114, in connect_tcp stream: anyio.abc.ByteStream = await anyio.connect_tcp( File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 232, in connect_tcp raise OSError("All connection attempts failed") from cause OSError: All connection attempts failed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions yield File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 366, in handle_async_request resp = await self._pool.handle_async_request(req) File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 268, in handle_async_request raise exc File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 251, in handle_async_request response = await connection.handle_async_request(request) File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request raise exc File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request stream = await self._connect(request) File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 124, in _connect stream = await self._network_backend.connect_tcp(**kwargs) File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/auto.py", line 30, in connect_tcp return await self._backend.connect_tcp( File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 121, in connect_tcp stream._raw_socket.setsockopt(*option) # type: ignore[attr-defined] # pragma: no cover File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__ self.gen.throw(typ, value, traceback) File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions raise to_exc(exc) from exc httpcore.ConnectError: All connection attempts failed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/litellm/llms/ollama.py", line 252, in ollama_async_streaming async with client.stream( File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__ return await self.gen.__anext__() File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1573, in stream response = await self.send( File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1617, in send response = await self._send_handling_auth( File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1645, in _send_handling_auth response = await self._send_handling_redirects( File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects response = await self._send_single_request(request) File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1719, in _send_single_request response = await transport.handle_async_request(request) File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 366, in handle_async_request resp = await self._pool.handle_async_request(req) File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__ self.gen.throw(typ, value, traceback) File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.ConnectError: All connection attempts failed Am I doing something wrong?
Author
Owner

@justinh-rahb commented on GitHub (Feb 6, 2024):

model_list:

  • model_name: mistral-7b
    litellm_params:
    model: ollama/llama2
    api_base: http://localhost:11434/api
    headers: {
    "HTTP-Referer": "litellm.ai",
    "X-Title": "LiteLLM Server"
    }

On mobile right now so I may have missed something in your logs, but at a glance I'd try changing the api_base: http://localhost:11434/api to api_base: http://ollama:11434/api.

Also, while you've named the model "mistral-7b" the actual model you're serving is llama2, you'd want to put in model: ollama/mistral instead, unless this is intentional.

<!-- gh-comment-id:1928562080 --> @justinh-rahb commented on GitHub (Feb 6, 2024): > model_list: > - model_name: mistral-7b > litellm_params: > model: ollama/llama2 > api_base: http://localhost:11434/api > headers: { > "HTTP-Referer": "litellm.ai", > "X-Title": "LiteLLM Server" > } On mobile right now so I may have missed something in your logs, but at a glance I'd try changing the `api_base: http://localhost:11434/api` to `api_base: http://ollama:11434/api`. Also, while you've named the model "mistral-7b" the actual model you're serving is llama2, you'd want to put in `model: ollama/mistral` instead, unless this is intentional.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#131890