The problem is with when I try to use ollama via litellm. that is where I get below error which I think it has somethig to do with api-base of ollama :
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 170, in try_connect
stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 2258, in connect_tcp
await get_running_loop().create_connection(
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1065, in create_connection
raise exceptions[0]
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1050, in create_connection
sock = await self._connect_sock(
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 961, in _connect_sock
await self.sock_connect(sock, address)
File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 500, in sock_connect
return await fut
File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('0.0.0.0', 11434)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
yield
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 114, in connect_tcp
stream: anyio.abc.ByteStream = await anyio.connect_tcp(
File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 232, in connect_tcp
raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
yield
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 268, in handle_async_request
raise exc
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 251, in handle_async_request
response = await connection.handle_async_request(request)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request
raise exc
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request
stream = await self._connect(request)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 124, in _connect
stream = await self._network_backend.connect_tcp(**kwargs)
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/auto.py", line 30, in connect_tcp
return await self._backend.connect_tcp(
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 121, in connect_tcp
stream._raw_socket.setsockopt(*option) # type: ignore[attr-defined] # pragma: no cover
File "/usr/local/lib/python3.9/contextlib.py", line 137, in exit
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectError: All connection attempts failed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/litellm/llms/ollama.py", line 252, in ollama_async_streaming
async with client.stream(
File "/usr/local/lib/python3.9/contextlib.py", line 181, in aenter
return await self.gen.anext()
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1573, in stream
response = await self.send(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1617, in send
response = await self._send_handling_auth(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1719, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.9/contextlib.py", line 137, in exit
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed
Am I doing something wrong?
Originally created by @mafrasiabi on GitHub (Feb 6, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/653
Hi @justinh-rahb
I've been struggling to make the litellm work and I was wondering if you could help with some details.
here is my config.yaml file:
model_list:
- model_name: mistral-7b
litellm_params:
model: ollama/llama2
api_base: http://localhost:11434/api
headers: {
"HTTP-Referer": "litellm.ai",
"X-Title": "LiteLLM Server"
}
- model_name: mistral-small
litellm_params:
model: mistral/mistral-small
api_base: https://api.mistral.ai/v1
api_key: "K"
- model_name: mistral-medium
litellm_params:
model: mistral/mistral-medium
api_base: https://api.mistral.ai/v1
api_key: "K"
and the docker-compose.yaml file I'm using:
version: '3.8'
services:
ollama:
volumes:
- ollama:/root/.ollama
container_name: ollama
pull_policy: always
tty: true
restart: unless-stopped
image: ollama/ollama:latest
ollama-webui:
build:
context: .
args:
OLLAMA_API_BASE_URL: '/ollama/api'
dockerfile: Dockerfile
image: ghcr.io/ollama-webui/ollama-webui:main
container_name: ollama-webui
volumes:
- ollama-webui:/app/backend/data
depends_on:
- ollama
ports:
- ${OLLAMA_WEBUI_PORT-3000}:8080
environment:
- 'OLLAMA_API_BASE_URL=http://ollama:11434/api'
# - "OLLAMA_API_BASE_URL=${OLLAMA_API_BASE_URL}"
- "OPENAI_API_BASE_URL=http://litellm:8000/v1"
- "OPENAI_API_KEY=sk-123456"
- 'WEBUI_SECRET_KEY='
extra_hosts:
- host.docker.internal:host-gateway
restart: unless-stopped
litellm:
image: ghcr.io/berriai/litellm:main-latest
environment:
- "MASTER_KEY=sk-123456"
- "OPENAI_API_KEY=sk-123456"
- "MISTRAL_API_KEY="
- "ANTHROPIC_API_KEY="
ports:
- 4000:8000
volumes:
- ./litellm/config.yaml:/app/config.yaml
command: [ "--config", "/app/config.yaml", "--port", "8000" ]
restart: unless-stopped
volumes:
ollama: {}
ollama-webui: {}
The problem is with when I try to use ollama via litellm. that is where I get below error which I think it has somethig to do with api-base of ollama :
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 170, in try_connect
stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 2258, in connect_tcp
await get_running_loop().create_connection(
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1065, in create_connection
raise exceptions[0]
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1050, in create_connection
sock = await self._connect_sock(
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 961, in _connect_sock
await self.sock_connect(sock, address)
File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 500, in sock_connect
return await fut
File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('0.0.0.0', 11434)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
yield
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 114, in connect_tcp
stream: anyio.abc.ByteStream = await anyio.connect_tcp(
File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 232, in connect_tcp
raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
yield
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 268, in handle_async_request
raise exc
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 251, in handle_async_request
response = await connection.handle_async_request(request)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request
raise exc
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request
stream = await self._connect(request)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 124, in _connect
stream = await self._network_backend.connect_tcp(**kwargs)
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/auto.py", line 30, in connect_tcp
return await self._backend.connect_tcp(
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 121, in connect_tcp
stream._raw_socket.setsockopt(*option) # type: ignore[attr-defined] # pragma: no cover
File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectError: All connection attempts failed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/litellm/llms/ollama.py", line 252, in ollama_async_streaming
async with client.stream(
File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
return await self.gen.__anext__()
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1573, in stream
response = await self.send(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1617, in send
response = await self._send_handling_auth(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1719, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed
Am I doing something wrong?
On mobile right now so I may have missed something in your logs, but at a glance I'd try changing the api_base: http://localhost:11434/api to api_base: http://ollama:11434/api.
Also, while you've named the model "mistral-7b" the actual model you're serving is llama2, you'd want to put in model: ollama/mistral instead, unless this is intentional.
<!-- gh-comment-id:1928562080 -->
@justinh-rahb commented on GitHub (Feb 6, 2024):
> model_list:
> - model_name: mistral-7b
> litellm_params:
> model: ollama/llama2
> api_base: http://localhost:11434/api
> headers: {
> "HTTP-Referer": "litellm.ai",
> "X-Title": "LiteLLM Server"
> }
On mobile right now so I may have missed something in your logs, but at a glance I'd try changing the `api_base: http://localhost:11434/api` to `api_base: http://ollama:11434/api`.
Also, while you've named the model "mistral-7b" the actual model you're serving is llama2, you'd want to put in `model: ollama/mistral` instead, unless this is intentional.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @mafrasiabi on GitHub (Feb 6, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/653
Hi @justinh-rahb
I've been struggling to make the litellm work and I was wondering if you could help with some details.
here is my config.yaml file:
model_list:
litellm_params:
model: ollama/llama2
api_base: http://localhost:11434/api
headers: {
"HTTP-Referer": "litellm.ai",
"X-Title": "LiteLLM Server"
}
litellm_params:
model: mistral/mistral-small
api_base: https://api.mistral.ai/v1
api_key: "K"
litellm_params:
model: mistral/mistral-medium
api_base: https://api.mistral.ai/v1
api_key: "K"
and the docker-compose.yaml file I'm using:
version: '3.8'
services:
ollama:
volumes:
- ollama:/root/.ollama
container_name: ollama
pull_policy: always
tty: true
restart: unless-stopped
image: ollama/ollama:latest
ollama-webui:
build:
context: .
args:
OLLAMA_API_BASE_URL: '/ollama/api'
dockerfile: Dockerfile
image: ghcr.io/ollama-webui/ollama-webui:main
container_name: ollama-webui
volumes:
- ollama-webui:/app/backend/data
depends_on:
- ollama
ports:
- ${OLLAMA_WEBUI_PORT-3000}:8080
environment:
- 'OLLAMA_API_BASE_URL=http://ollama:11434/api'
# - "OLLAMA_API_BASE_URL=${OLLAMA_API_BASE_URL}"
- "OPENAI_API_BASE_URL=http://litellm:8000/v1"
- "OPENAI_API_KEY=sk-123456"
- 'WEBUI_SECRET_KEY='
extra_hosts:
- host.docker.internal:host-gateway
restart: unless-stopped
litellm:
image: ghcr.io/berriai/litellm:main-latest
environment:
- "MASTER_KEY=sk-123456"
- "OPENAI_API_KEY=sk-123456"
- "MISTRAL_API_KEY="
- "ANTHROPIC_API_KEY="
ports:
- 4000:8000
volumes:
- ./litellm/config.yaml:/app/config.yaml
command: [ "--config", "/app/config.yaml", "--port", "8000" ]
restart: unless-stopped
volumes:
ollama: {}
ollama-webui: {}
The problem is with when I try to use ollama via litellm. that is where I get below error which I think it has somethig to do with api-base of ollama :
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 170, in try_connect
stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 2258, in connect_tcp
await get_running_loop().create_connection(
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1065, in create_connection
raise exceptions[0]
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1050, in create_connection
sock = await self._connect_sock(
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 961, in _connect_sock
await self.sock_connect(sock, address)
File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 500, in sock_connect
return await fut
File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('0.0.0.0', 11434)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
yield
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 114, in connect_tcp
stream: anyio.abc.ByteStream = await anyio.connect_tcp(
File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 232, in connect_tcp
raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
yield
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 268, in handle_async_request
raise exc
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 251, in handle_async_request
response = await connection.handle_async_request(request)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request
raise exc
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request
stream = await self._connect(request)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 124, in _connect
stream = await self._network_backend.connect_tcp(**kwargs)
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/auto.py", line 30, in connect_tcp
return await self._backend.connect_tcp(
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 121, in connect_tcp
stream._raw_socket.setsockopt(*option) # type: ignore[attr-defined] # pragma: no cover
File "/usr/local/lib/python3.9/contextlib.py", line 137, in exit
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectError: All connection attempts failed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/litellm/llms/ollama.py", line 252, in ollama_async_streaming
async with client.stream(
File "/usr/local/lib/python3.9/contextlib.py", line 181, in aenter
return await self.gen.anext()
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1573, in stream
response = await self.send(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1617, in send
response = await self._send_handling_auth(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1719, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.9/contextlib.py", line 137, in exit
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed
Am I doing something wrong?
@justinh-rahb commented on GitHub (Feb 6, 2024):
On mobile right now so I may have missed something in your logs, but at a glance I'd try changing the
api_base: http://localhost:11434/apitoapi_base: http://ollama:11434/api.Also, while you've named the model "mistral-7b" the actual model you're serving is llama2, you'd want to put in
model: ollama/mistralinstead, unless this is intentional.