github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 12:48:05 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #6843] Help Needed! Connecting Ollama’s llama3:8b to External Platforms and Connection Refused Error #4321

New Issue
Open
opened 2026-04-12 15:15:03 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-04-12 15:15:03 -05:00
Owner
Copy Link

Originally created by @camilleconte8 on GitHub (Sep 17, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6843

Help Needed! Connecting Ollama’s llama3:8b to External Platforms and Connection Refused Error

I am new to development, and have a windows machine where I have set up a WSL2 environment with Ubuntu 22.04.4 LTS (GNU/Linux 5.15.153.1-microsoft-standard-WSL2 x86_64). I’ve been working on setting up Ollama, specifically running the llama3:8b model on my local machine.

For more context, I have been following this tutorial (https://blog.getwren.ai/how-to-use-meta-llama-3-to-query-mysql-database-using-ollama-on-your-machine-2c087b204e41). (I know they say to use the 70B llama3 model but my machine isn't powerful enough. I thought that was my problem at first, but later discovered it's not which I will get into).

So ollama works great when I interact with it via localhost and even within a Docker container. However, I’ve encountered significant challenges when attempting to connect it with external services like Wren AI and Replit, where I consistently run into connection errors, more specifically [Errno 111] Connection refused. Below is a detailed account my the setup and the issues I’m experiencing.

1. Ollama Setup on My Local Machine

First, I managed to successfully install and run Ollama's llama3:8b model on my local machine. Here’s a quick overview of my environment:

  • Ollama is running on localhost (127.0.0.1) on port 11434 (default). When i visit http://localhost:11434/, it says "Ollama is running."
  • The model is llama3:8b, an 8-billion parameter model from Meta AI, integrated through Ollama by the command ollama pull llama3:8b.
  • Here are status logs after saying sudo systemctl status ollama:
● ollama.service - Ollama Service Loaded: loaded (/etc/systemd/system/ollama.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2024-09-16 16:37:13 EDT; 19h ago Main PID: 58329 (ollama) Tasks: 15 (limit: 14999) Memory: 20.9M CGroup: /system.slice/ollama.service └─58329 /usr/local/bin/ollama serve Sep 16 16:37:13 camillecore ollama[58329]: time=2024-09-16T16:37:13.683-04:00 level=INFO source=images.go:782 msg="total blobs: 10" Sep 16 16:37:13 camillecore ollama[58329]: time=2024-09-16T16:37:13.683-04:00 level=INFO source=images.go:790 msg="total unused blobs removed: 0" Sep 16 16:37:13 camillecore ollama[58329]: time=2024-09-16T16:37:13.683-04:00 level=INFO source=routes.go:1172 msg="Listening on 127.0.0.1:11434 (version 0.3.6)" Sep 16 16:37:13 camillecore ollama[58329]: time=2024-09-16T16:37:13.684-04:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama53233405/runners Sep 16 16:37:16 camillecore ollama[58329]: time=2024-09-16T16:37:16.542-04:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [rocm_v60102 cpu cpu_avx cpu_avx2 cuda_v11]" Sep 16 16:37:16 camillecore ollama[58329]: time=2024-09-16T16:37:16.542-04:00 level=INFO source=gpu.go:204 msg="looking for compatible GPUs" Sep 16 16:37:21 camillecore ollama[58329]: time=2024-09-16T16:37:21.239-04:00 level=INFO source=gpu.go:350 msg="no compatible GPUs were discovered" Sep 16 16:37:21 camillecore ollama[58329]: time=2024-09-16T16:37:21.239-04:00 level=INFO source=types.go:105 msg="inference compute" id=0 library=cpu compute="" driver=0.0 name="" total="19.4 GiB" available="17.1 GiB" Sep 16 16:37:28 camillecore ollama[58329]: [GIN] 2024/09/16 - 16:37:28 | 200 | 821.899µs | 127.0.0.1 | GET "/api/tags" Sep 16 16:37:29 camillecore ollama[58329]: [GIN] 2024/09/16 - 16:37:29 | 200 | 1.216507046s | 127.0.0.1 | POST "/api/pull"
  • This is the configuration of my ollama.service file:
[Unit] Description=Ollama Service After=network-online.target [Service] ExecStart=/usr/local/bin/ollama serve User=ollama Group=ollama Restart=always RestartSec=3 Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0/:/mnt/c/Windows/System32/OpenSSH/:/Docker/host/bin:/mnt/c/Users/camil/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/camil/AppData/Local/Programs/Microsoft VS Code/bin:/snap/bin" Environment="OLLAMA_MODELS=/usr/share/ollama/.ollama/models" Environment="OLLAMA_HOST=127.0.0.1:11434" Environment="OLLAMA_ORIGINS=http://127.0.0.1:11434" Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0" Environment="CUDA_VISIBLE_DEVICES=" [Install] WantedBy=default.target

Modifications I've made to ollama.service:

a. Added Environment="OLLAMA_HOST=127.0.0.1:11434" and Environment="OLLAMA_ORIGINS=http://127.0.0.1:11434". I found this (https://github.com/ollama/ollama/issues/2132) issue where one person had the solution of adding these, but with 0.0.0.0 IP address instead of 127.0.0.1. I tried that as well and have had the same errors. b. added Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0" and Environment="CUDA_VISIBLE_DEVICES=", which were suggestions from another person in that chat (the override part) and ChatGPT (the visible devices part) Again, same error. c. I also broke down the 11434 firewall with sudo ufw allow 11434, then sudo ufw reload, then sudo systemctl restart ollama.service.
  • Here are various commands I've run to confirm Ollama is functioning on my machine:
camilleconte@camillecore:~$ sudo lsof -i :11434 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ollama 34328 ollama 3u IPv4 728857 0t0 TCP localhost:11434 (LISTEN) camilleconte@camillecore:~$ ollama ps NAME ID SIZE PROCESSOR UNTIL llama3:8b 365c0bd3c000 6.2 GB 100% CPU 4 minutes from now camilleconte@camillecore:~$ sudo lsof -i :11434 [sudo] password for camilleconte: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ollama 58329 ollama 3u IPv4 2813691 0t0 TCP localhost:11434 (LISTEN)
  • Here I show that I can interact with Ollama through HTTP requests in my terminal using curl:
camilleconte@camillecore:~$ curl -X POST http://localhost:11434/api/generate \ -H "Content-Type: application/json" \ -d '{ "model": "llama3:8b", "prompt": "What is the capital of France?", "temperature": 0.7 }' Response: {"model":"llama3:8b","created_at":"2024-09-17T16:26:43.660429962Z","response":"The","done":false} {"model":"llama3:8b","created_at":"2024-09-17T16:26:43.895378292Z","response":" capital","done":false} {"model":"llama3:8b","created_at":"2024-09-17T16:26:44.14832767Z","response":" of","done":false} {"model":"llama3:8b","created_at":"2024-09-17T16:26:44.403461964Z","response":" France","done":false} {"model":"llama3:8b","created_at":"2024-09-17T16:26:44.657318964Z","response":" is","done":false} {"model":"llama3:8b","created_at":"2024-09-17T16:26:44.910058883Z","response":" Paris","done":false} {"model":"llama3:8b","created_at":"2024-09-17T16:26:45.176020402Z","response":".","done":false} {"model":"llama3:8b","created_at":"2024-09-17T16:26:45.44038917Z","response":"","done":true,"done_reason":"stop","context":[128006,882,128007,271,3923,374,279,6864,315,9822,30,128009,128006,78191,128007,271,791,6864,315,9822,374,12366,13],"total_duration":8302556474,"load_duration":4573576269,"prompt_eval_count":17,"prompt_eval_duration":1893859000,"eval_count":8,"eval_duration":1780122000}

As you can see This interaction works perfectly and returns the correct responses directly in the terminal. Now I will show the external connections I've been trying to make that have failed.

2. Issues with Connecting to external services

While interacting with Ollama locally works well, I’ve been facing persistent issues when trying to integrate it with external services, namely Wren AI and Replit.

  • Wren AI: Wren AI is intended to integrate with a local or external LLM host, in this case, my Ollama instance. However, whenever I try to connect Wren AI to the localhost:11434 port where Ollama is running, I receive connection errors indicating that the API cannot be reached. These errors typically take the form of "Connection refused" errors, as Wren AI repeatedly fails to establish communication with the Ollama host. Following this (https://blog.getwren.ai/how-to-use-meta-llama-3-to-query-mysql-database-using-ollama-on-your-machine-2c087b204e41) tutorial, here are the relevant files in the docker project directory, where a container called wren-ai-service-1 forwards my ai service to port 5555:
  1. .env.ai
## LLM LLM_PROVIDER=ollama_llm # openai_llm, azure_openai_llm, ollama_llm GENERATION_MODEL=llama3:8b GENERATION_MODEL_KWARGS='{"temperature": 0}' # openai or openai-api-compatible #LLM_OPENAI_API_KEY=sk-xxxx #LLM_OPENAI_API_BASE=https://api.openai.com/v1 # azure_openai #LLM_AZURE_OPENAI_API_KEY= #LLM_AZURE_OPENAI_API_BASE= #LLM_AZURE_OPENAI_VERSION= # ollama LLM_OLLAMA_URL=http://host.docker.internal:11434 ## EMBEDDER EMBEDDER_PROVIDER=ollama_embedder # openai_embedder, azure_openai_embedder, ollama_embedder # supported embedding models providers by qdrant: https://qdrant.tech/documentation/embeddings/ EMBEDDING_MODEL=nomic-embed-text EMBEDDING_MODEL_DIMENSION=768 # openai or openai-api-compatible #EMBEDDER_OPENAI_API_KEY=sk-xxxx #EMBEDDER_OPENAI_API_BASE=https://api.openai.com/v1 # azure_openai #EMBEDDER_AZURE_OPENAI_API_KEY= #EMBEDDER_AZURE_OPENAI_API_BASE= #EMBEDDER_AZURE_OPENAI_VERSION= # ollama EMBEDDER_OLLAMA_URL=http://host.docker.internal:11434 ## DOCUMENT_STORE DOCUMENT_STORE_PROVIDER=qdrant QDRANT_HOST=qdrant
  1. .env
COMPOSE_PROJECT_NAME=wrenai PLATFORM=linux/amd64 PROJECT_DIR=/home/camilleconte/.wrenai # service port WREN_ENGINE_PORT=8080 WREN_ENGINE_SQL_PORT=7432 WREN_AI_SERVICE_PORT=5555 WREN_UI_PORT=3000 IBIS_SERVER_PORT=8000 # service endpoint (for docker-compose-dev.yaml file) WREN_UI_ENDPOINT=http://wren-ui:3000 # LLM #LLM_OPENAI_API_KEY= #EMBEDDER_OPENAI_API_KEY= LLM_PROVIDER=ollama_llm # openai_llm, azure_openai_llm, ollama_llm GENERATION_MODEL_KWARGS='{"temperature": 0}' GENERATION_MODEL=llama3:8b # gpt-4o-mini, gpt-4o, gpt-4-turbo, gpt-3.5-turbo LLM_OLLAMA_URL=http://host.docker.internal:11434 ## EMBEDDER EMBEDDER_PROVIDER=ollama_embedder # openai_embedder, azure_openai_embedder, ollama_embedder # supported embedding models providers by qdrant: https://qdrant.tech/documentation/embeddings/ EMBEDDING_MODEL=nomic-embed-text EMBEDDING_MODEL_DIMENSION=768 # version # CHANGE THIS TO THE LATEST VERSION WREN_PRODUCT_VERSION=0.7.5 WREN_ENGINE_VERSION=0.9.0 WREN_AI_SERVICE_VERSION=0.8.0 IBIS_SERVER_VERSION=0.9.0 WREN_UI_VERSION=0.9.2 WREN_BOOTSTRAP_VERSION=0.1.5 # AI service related env variables AI_SERVICE_ENABLE_TIMER= AI_SERVICE_LOGGING_LEVEL=INFO SHOULD_FORCE_DEPLOY= QDRANT_HOST=qdrant # user id (uuid v4) USER_UUID=1aa9be21-4235-4723-9632-6d1645571769 # for other services POSTHOG_API_KEY=phc_nhF32aj4xHXOZb0oqr2cn4Oy9uiWzz6CCP4KZmRq9aE POSTHOG_HOST=https://app.posthog.com TELEMETRY_ENABLED=true # the port exposes to the host # OPTIONAL: change the port if you have a conflict HOST_PORT=3000 AI_SERVICE_FORWARD_PORT=5555
  1. docker-compose.yaml
version: "3.8" volumes: data: networks: wren: driver: bridge services: bootstrap: image: ghcr.io/canner/wren-bootstrap:${WREN_BOOTSTRAP_VERSION} restart: on-failure platform: ${PLATFORM} environment: DATA_PATH: /app/data volumes: - data:/app/data command: /bin/sh /app/init.sh wren-engine: image: ghcr.io/canner/wren-engine:${WREN_ENGINE_VERSION} restart: on-failure platform: ${PLATFORM} expose: - ${WREN_ENGINE_PORT} - ${WREN_ENGINE_SQL_PORT} volumes: - data:/usr/src/app/etc networks: - wren depends_on: - bootstrap ibis-server: image: ghcr.io/canner/wren-engine-ibis:${IBIS_SERVER_VERSION} restart: on-failure platform: ${PLATFORM} expose: - ${IBIS_SERVER_PORT} environment: WREN_ENGINE_ENDPOINT: http://wren-engine:${WREN_ENGINE_PORT} networks: - wren wren-ai-service: image: ghcr.io/canner/wren-ai-service:${WREN_AI_SERVICE_VERSION} restart: on-failure platform: ${PLATFORM} expose: - ${WREN_AI_SERVICE_PORT} ports: - ${AI_SERVICE_FORWARD_PORT}:${WREN_AI_SERVICE_PORT} environment: LLM_PROVIDER: ollama_llm LLM_OLLAMA_URL: http://host.docker.internal:11434 GENERATION_MODEL: llama3:8b GENERATION_MODEL_KWARGS: '{"temperature": 0}' EMBEDDER_PROVIDER: ollama_embedder EMBEDDING_MODEL: nomic-embed-text EMBEDDING_MODEL_DIMENSION: "768" #EMBEDDER_OPENAI_API_KEY: "" #EMBEDDER_AZURE_OPENAI_API_KEY: "" WREN_UI_PORT: "3000" WREN_UI_ENDPOINT: http://wren-ui:3000 WREN_AI_SERVICE_PORT: "5555" ENABLE_TIMER: "" LOGGING_LEVEL: INFO PYTHONUNBUFFERED: 1 networks: - wren depends_on: - wren-engine - qdrant qdrant: image: qdrant/qdrant:v1.7.4 restart: on-failure expose: - 6333 - 6334 volumes: - data:/qdrant/storage networks: - wren wren-ui: image: ghcr.io/canner/wren-ui:${WREN_UI_VERSION} restart: on-failure platform: ${PLATFORM} environment: DB_TYPE: sqlite # /app is the working directory in the container SQLITE_FILE: /app/data/db.sqlite3 WREN_ENGINE_ENDPOINT: http://wren-engine:${WREN_ENGINE_PORT} WREN_AI_ENDPOINT: http://wren-ai-service:${WREN_AI_SERVICE_PORT} IBIS_SERVER_ENDPOINT: http://ibis-server:${IBIS_SERVER_PORT} EMBEDDING_MODEL: ${EMBEDDING_MODEL} EMBEDDING_MODEL_DIMENSION: ${EMBEDDING_MODEL_DIMENSION} GENERATION_MODEL: ${GENERATION_MODEL} # telemetry WREN_ENGINE_PORT: ${WREN_ENGINE_PORT} WREN_AI_SERVICE_VERSION: ${WREN_AI_SERVICE_VERSION} WREN_UI_VERSION: ${WREN_UI_VERSION} WREN_ENGINE_VERSION: ${WREN_ENGINE_VERSION} USER_UUID: ${USER_UUID} POSTHOG_API_KEY: ${POSTHOG_API_KEY} POSTHOG_HOST: ${POSTHOG_HOST} TELEMETRY_ENABLED: ${TELEMETRY_ENABLED} # client side NEXT_PUBLIC_USER_UUID: ${USER_UUID} NEXT_PUBLIC_POSTHOG_API_KEY: ${POSTHOG_API_KEY} NEXT_PUBLIC_POSTHOG_HOST: ${POSTHOG_HOST} NEXT_PUBLIC_TELEMETRY_ENABLED: ${TELEMETRY_ENABLED} # configs WREN_PRODUCT_VERSION: ${WREN_PRODUCT_VERSION} ports: # HOST_PORT is the port you want to expose to the host machine - ${HOST_PORT}:3000 volumes: - data:/app/data networks: - wren depends_on: - wren-ai-service - wren-engine
  • When start wrenai on docker, it looks like this:
    wren_running_docker

I can visit localhost3000 and it looks great, but then localhost5555 always says ERR_EMPTY_RESPONSE. The docker logs for the wren-ai-service-1 container look like this:

2024-09-16 14:56:01 This module is deprecated and will be removed in Hamilton 2.0 Please use `hamilton.async_driver` instead. 2024-09-16 14:56:02 INFO: Started server process [7] 2024-09-16 14:56:02 INFO: Waiting for application startup. 2024-09-16 14:56:02 2024-09-16 18:56:02,058 - wren-ai-service - INFO - Initializing providers... (utils.py:64) 2024-09-16 14:56:03 2024-09-16 18:56:03,095 - wren-ai-service - INFO - Registering provider: qdrant (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,356 - wren-ai-service - INFO - Registering provider: azure_openai_embedder (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,359 - wren-ai-service - INFO - Registering provider: ollama_embedder (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,360 - wren-ai-service - INFO - Registering provider: openai_embedder (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,361 - wren-ai-service - INFO - Registering provider: wren_ui (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,361 - wren-ai-service - INFO - Registering provider: wren_ibis (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,361 - wren-ai-service - INFO - Registering provider: wren_engine (loader.py:66) 2024-09-16 11:42:03 Waiting for wren-ai-service to start... 2024-09-16 14:55:59 Waiting for wren-ai-service to start... 2024-09-16 14:56:03 2024-09-16 18:56:03,368 - wren-ai-service - INFO - Registering provider: azure_openai_llm (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,372 - wren-ai-service - INFO - Registering provider: ollama_llm (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,372 - wren-ai-service - INFO - Registering provider: openai_llm (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,402 - wren-ai-service - INFO - Pulling Ollama model llama3:8b (loader.py:109) 2024-09-16 14:56:05 2024-09-16 18:56:05,211 - wren-ai-service - INFO - Pulling Ollama model llama3:8b: 100% (loader.py:116) 2024-09-16 14:56:05 2024-09-16 18:56:05,217 - wren-ai-service - INFO - Using Ollama LLM: llama3:8b (ollama.py:135) 2024-09-16 14:56:05 2024-09-16 18:56:05,217 - wren-ai-service - INFO - Using Ollama URL: http://host.docker.internal:11434 (ollama.py:136) 2024-09-16 14:56:05 ERROR: Traceback (most recent call last): 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions 2024-09-16 14:56:05 yield 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 233, in handle_request 2024-09-16 14:56:05 resp = self._pool.handle_request(req) 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request 2024-09-16 14:56:05 raise exc from None 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request 2024-09-16 14:56:05 response = connection.handle_request( 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 99, in handle_request 2024-09-16 14:56:05 raise exc 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 76, in handle_request 2024-09-16 14:56:05 stream = self._connect(request) 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 122, in _connect 2024-09-16 14:56:05 stream = self._network_backend.connect_tcp(**kwargs) 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp 2024-09-16 14:56:05 with map_exceptions(exc_map): 2024-09-16 14:56:05 File "/usr/local/lib/python3.12/contextlib.py", line 155, in __exit__ 2024-09-16 14:56:05 self.gen.throw(value) 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions 2024-09-16 14:56:05 raise to_exc(exc) from exc 2024-09-16 14:56:05 httpcore.ConnectError: [Errno 111] Connection refused 2024-09-16 14:56:05 2024-09-16 14:56:05 The above exception was the direct cause of the following exception: 2024-09-16 14:56:05 2024-09-16 14:56:05 Traceback (most recent call last): 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/starlette/routing.py", line 734, in lifespan 2024-09-16 14:56:05 async with self.lifespan_context(app) as maybe_state: 2024-09-16 14:56:05 File "/usr/local/lib/python3.12/contextlib.py", line 204, in __aenter__ 2024-09-16 14:56:05 return await anext(self.gen) 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/src/__main__.py", line 28, in lifespan 2024-09-16 14:56:05 container.init_globals() 2024-09-16 14:56:05 File "/src/globals.py", line 53, in init_globals 2024-09-16 14:56:05 llm_provider, embedder_provider, document_store_provider, engine = init_providers( 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/src/utils.py", line 68, in init_providers 2024-09-16 14:56:05 embedder_provider = loader.get_provider( 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/src/providers/embedder/ollama.py", line 170, in __init__ 2024-09-16 14:56:05 pull_ollama_model(self._url, self._embedding_model) 2024-09-16 14:56:05 File "/src/providers/loader.py", line 107, in pull_ollama_model 2024-09-16 14:56:05 models = client.list()["models"] 2024-09-16 14:56:05 ^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/ollama/_client.py", line 333, in list 2024-09-16 14:56:05 return self._request('GET', '/api/tags').json() 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/ollama/_client.py", line 69, in _request 2024-09-16 14:56:05 response = self._client.request(method, url, **kwargs) 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_client.py", line 827, in request 2024-09-16 14:56:05 return self.send(request, auth=auth, follow_redirects=follow_redirects) 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_client.py", line 914, in send 2024-09-16 14:56:05 response = self._send_handling_auth( 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth 2024-09-16 14:56:05 response = self._send_handling_redirects( 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_client.py", line 979, in _send_handling_redirects 2024-09-16 14:56:05 response = self._send_single_request(request) 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1015, in _send_single_request 2024-09-16 14:56:05 response = transport.handle_request(request) 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 232, in handle_request 2024-09-16 14:56:05 with map_httpcore_exceptions(): 2024-09-16 14:56:05 File "/usr/local/lib/python3.12/contextlib.py", line 155, in __exit__ 2024-09-16 14:56:05 self.gen.throw(value) 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions 2024-09-16 14:56:05 raise mapped_exc(message) from exc 2024-09-16 14:56:05 httpx.ConnectError: [Errno 111] Connection refused 2024-09-16 14:56:05 2024-09-16 14:56:05 ERROR: Application startup failed. Exiting.

At first, I thought it was an issue with docker communicating with ollama, so I entered the wren-ai-service-1 container in my terminal( docker exec -it wrenai-wren-ai-service-1 /bin/bash). I wanted to see if i could verify the docker connection in there, so I did this and succeeded as you can see here:

root@1773b32d1441:/# curl http://host.docker.internal:11434/v1/models {"object":"list","data":[{"id":"llama3:8b","object":"model","created":1726592861,"owned_by":"library"},{"id":"llama3:70b","object":"model","created":1724455596,"owned_by":"library"},{"id":"nomic-embed-text:latest","object":"model","created":1724455596,"owned_by":"library"}]} root@1773b32d1441:/# curl -X POST http://host.docker.internal:11434/api/generate -H "Content-Type: application/json" -d '{ "model": "llama3:8b", "prompt": "What is the capital of France?", "temperature": 0.7 }' {"model":"llama3:8b","created_at":"2024-09-17T17:15:43.734078981Z","response":"The","done":false} {"model":"llama3:8b","created_at":"2024-09-17T17:15:43.990105229Z","response":" capital","done":false} {"model":"llama3:8b","created_at":"2024-09-17T17:15:44.25267107Z","response":" of","done":false} {"model":"llama3:8b","created_at":"2024-09-17T17:15:44.513833801Z","response":" France","done":false} {"model":"llama3:8b","created_at":"2024-09-17T17:15:44.766417457Z","response":" is","done":false} {"model":"llama3:8b","created_at":"2024-09-17T17:15:45.037476835Z","response":" Paris","done":false} {"model":"llama3:8b","created_at":"2024-09-17T17:15:45.306243202Z","response":".","done":false} {"model":"llama3:8b","created_at":"2024-09-17T17:15:45.557952927Z","response":"","done":true,"done_reason":"stop","context":[128006,882,128007,271,3923,374,279,6864,315,9822,30,128009,128006,78191,128007,271,791,6864,315,9822,374,12366,13],"total_duration":8320738087,"load_duration":4567994305,"prompt_eval_count":17,"prompt_eval_duration":1882077000,"eval_count":8,"eval_duration":1823834000}

Sure enough, the curl request to Ollama worked fine in the docker container through the docker host url. So I confirmed that I could access the ollama service just fine from both my environments; inside the Docker container and directly on the host machine. However, just to be thorough, I should mention that when I am inside the container in terminal and send the request to localhost:11434 instead of host.docker.internal:11434, it says "curl: (7) Failed to connect to localhost port 11434 after 4 ms: Couldn't connect to server".

So next, I thought the problem was with wrenAI, but then this happened:

  • Replit: This was a random sidequest that ended up being pretty informative in the end. Basically I didn't like the output of the llama3:8b model in my terminal before I knew about the "ollama run llama3:8b" command (LOL) So I tried to use replit (a cloud-based development environment) to host its responses in a nicer format. These are the steps I took:
Step 1: Create a Replit Account Step 2: Create a New Repl, Click "Create" in the top left corner of the dashboard. Choose Python as the programming language. Name the project (e.g., Ollama Chat), then click Create Repl. Step 3: Set Up the API Call Install Dependencies: In the Repl’s file explorer, create a requirements.txt file and add: 'requests' Then I Created a main.py file and added this code to interact with Ollama: import requests import json url = "http://localhost:11434/api/generate" data = { "model": "llama3:8b", "prompt": "What is the capital of France?", "temperature": 0.7 } response = requests.post(url, headers={"Content-Type": "application/json"}, data=json.dumps(data)) print(response.json())

I ran the code, and I got these errors:

Traceback (most recent call last): File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/connection.py", line 199, in _new_conn sock = connection.create_connection( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/connectionpool.py", line 789, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/connectionpool.py", line 495, in _make_request conn.request( File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/connection.py", line 441, in request self.endheaders() File "/nix/store/f98g7xbckgqbkagdvpzc2r6lv3h1p9ki-python3-3.11.9/lib/python3.11/http/client.py", line 1298, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/nix/store/f98g7xbckgqbkagdvpzc2r6lv3h1p9ki-python3-3.11.9/lib/python3.11/http/client.py", line 1058, in _send_output self.send(msg) File "/nix/store/f98g7xbckgqbkagdvpzc2r6lv3h1p9ki-python3-3.11.9/lib/python3.11/http/client.py", line 996, in send self.connect() File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/connection.py", line 279, in connect self.sock = self._new_conn() ^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/connection.py", line 214, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/connectionpool.py", line 843, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=11434): Max retries exceeded with url: /api/generate (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/runner/ollama-8b-chat/main.py", line 15, in response = requests.post(url, headers={"Content-Type": "application/json"}, data=json.dumps(data)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/requests/api.py", line 115, in post return request("post", url, data=data, json=json, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/requests/adapters.py", line 700, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=11434): Max retries exceeded with url: /api/generate (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))

As you can see, it threw the same [Errno 111] Connection refused. This error indicates that localhost:11434 cannot be reached from Replit, which makes sense because Replit runs in the cloud and doesn’t have access to my local machine’s localhost. The same problem occurs in Wren AI because, like Replit, it's not running on my local machine but rather trying to connect to my local instance over HTTP. So this is what has convinced me that there's something wrong with my ollama service, as it does not seem to allow any external services to interact with it despite how it is functioning just fine on my machine locally. Note: it also gave the same errors when I changed the python script to http://host.docker.internal:11434/api/generate instead of localhost:11434.

3. Suspicions and Additional Troubleshooting Attempts

  1. One of the first things I suspected was that Ollama’s default CORS (Cross-Origin Resource Sharing) policy might be preventing access from external sources like Replit and Wren AI, as by default, Ollama’s CORS policy is restricted to localhost. To resolve this, I tried editing the Ollama systemd service configuration like I described earlier. But again, despite making these changes, both Wren AI and Replit continued to throw the same connection refused errors.

  2. I noticed that a lot of the error files are coming from a python "httpx" directory, and I have read on ollama's documentation faq (https://github.com/ollama/ollama/blob/main/docs/faq.md#setting-environment-variables-on-linux) that one should "Avoid setting HTTP_PROXY. Ollama does not use HTTP for model pulls, only HTTPS. Setting HTTP_PROXY may interrupt client connections to the server." No idea id this is a rabbithole of relevance though.
    On the subject of proxies, I know that there's an option to expose my ollama- like using a tool like ngrok to expose my locally running Ollama instance to the internet so that Wren AI and Replit can access it via an externally accessible URL, or deploying Ollama on a cloud server or a VPS. I do not want to expose my ollama server, and I feel like I shouldn't have to, right? Again, I am a beginner, so if any of you experts think this is my only option I would love to hear it from you.

Okay that is all! Any help, suggestions, comments etc would be greatly appreciated because I am feeling very stuck. Thanks!

Originally created by @camilleconte8 on GitHub (Sep 17, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6843 ### Help Needed! Connecting Ollama’s llama3:8b to External Platforms and Connection Refused Error I am new to development, and have a windows machine where I have set up a WSL2 environment with Ubuntu 22.04.4 LTS (GNU/Linux 5.15.153.1-microsoft-standard-WSL2 x86_64). I’ve been working on setting up Ollama, specifically running the **llama3:8b model** on my local machine. For more context, I have been following this tutorial (https://blog.getwren.ai/how-to-use-meta-llama-3-to-query-mysql-database-using-ollama-on-your-machine-2c087b204e41). (I know they say to use the 70B llama3 model but my machine isn't powerful enough. I thought that was my problem at first, but later discovered it's not which I will get into). So ollama works great when I interact with it via **localhost** and even within a **Docker container**. However, I’ve encountered significant challenges when attempting to connect it with external services like **Wren AI** and **Replit**, where I consistently run into connection errors, more specifically [Errno 111] Connection refused. Below is a detailed account my the setup and the issues I’m experiencing. ### 1. **Ollama Setup on My Local Machine** First, I managed to successfully install and run Ollama's llama3:8b model on my local machine. Here’s a quick overview of my environment: - **Ollama is running on localhost** (127.0.0.1) on port **11434** (default). When i visit http://localhost:11434/, it says "Ollama is running." - The model is **llama3:8b**, an 8-billion parameter model from Meta AI, integrated through Ollama by the command ollama pull llama3:8b. - Here are status logs after saying sudo systemctl status ollama: <details> ● ollama.service - Ollama Service Loaded: loaded (/etc/systemd/system/ollama.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2024-09-16 16:37:13 EDT; 19h ago Main PID: 58329 (ollama) Tasks: 15 (limit: 14999) Memory: 20.9M CGroup: /system.slice/ollama.service └─58329 /usr/local/bin/ollama serve Sep 16 16:37:13 camillecore ollama[58329]: time=2024-09-16T16:37:13.683-04:00 level=INFO source=images.go:782 msg="total blobs: 10" Sep 16 16:37:13 camillecore ollama[58329]: time=2024-09-16T16:37:13.683-04:00 level=INFO source=images.go:790 msg="total unused blobs removed: 0" Sep 16 16:37:13 camillecore ollama[58329]: time=2024-09-16T16:37:13.683-04:00 level=INFO source=routes.go:1172 msg="Listening on 127.0.0.1:11434 (version 0.3.6)" Sep 16 16:37:13 camillecore ollama[58329]: time=2024-09-16T16:37:13.684-04:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama53233405/runners Sep 16 16:37:16 camillecore ollama[58329]: time=2024-09-16T16:37:16.542-04:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [rocm_v60102 cpu cpu_avx cpu_avx2 cuda_v11]" Sep 16 16:37:16 camillecore ollama[58329]: time=2024-09-16T16:37:16.542-04:00 level=INFO source=gpu.go:204 msg="looking for compatible GPUs" Sep 16 16:37:21 camillecore ollama[58329]: time=2024-09-16T16:37:21.239-04:00 level=INFO source=gpu.go:350 msg="no compatible GPUs were discovered" Sep 16 16:37:21 camillecore ollama[58329]: time=2024-09-16T16:37:21.239-04:00 level=INFO source=types.go:105 msg="inference compute" id=0 library=cpu compute="" driver=0.0 name="" total="19.4 GiB" available="17.1 GiB" Sep 16 16:37:28 camillecore ollama[58329]: [GIN] 2024/09/16 - 16:37:28 | 200 | 821.899µs | 127.0.0.1 | GET "/api/tags" Sep 16 16:37:29 camillecore ollama[58329]: [GIN] 2024/09/16 - 16:37:29 | 200 | 1.216507046s | 127.0.0.1 | POST "/api/pull" </details> - This is the configuration of my ollama.service file: <details> [Unit] Description=Ollama Service After=network-online.target [Service] ExecStart=/usr/local/bin/ollama serve User=ollama Group=ollama Restart=always RestartSec=3 Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0/:/mnt/c/Windows/System32/OpenSSH/:/Docker/host/bin:/mnt/c/Users/camil/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/camil/AppData/Local/Programs/Microsoft VS Code/bin:/snap/bin" Environment="OLLAMA_MODELS=/usr/share/ollama/.ollama/models" Environment="OLLAMA_HOST=127.0.0.1:11434" Environment="OLLAMA_ORIGINS=http://127.0.0.1:11434" Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0" Environment="CUDA_VISIBLE_DEVICES=" [Install] WantedBy=default.target </details> **Modifications I've made to ollama.service:** <details> a. Added Environment="OLLAMA_HOST=127.0.0.1:11434" and Environment="OLLAMA_ORIGINS=http://127.0.0.1:11434". I found this (https://github.com/ollama/ollama/issues/2132) issue where one person had the solution of adding these, but with 0.0.0.0 IP address instead of 127.0.0.1. I tried that as well and have had the same errors. b. added Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0" and Environment="CUDA_VISIBLE_DEVICES=", which were suggestions from another person in that chat (the override part) and ChatGPT (the visible devices part) Again, same error. c. I also broke down the 11434 firewall with sudo ufw allow 11434, then sudo ufw reload, then sudo systemctl restart ollama.service. </details> - Here are various commands I've run to confirm Ollama is functioning on my machine: <details> camilleconte@camillecore:~$ sudo lsof -i :11434 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ollama 34328 ollama 3u IPv4 728857 0t0 TCP localhost:11434 (LISTEN) camilleconte@camillecore:~$ ollama ps NAME ID SIZE PROCESSOR UNTIL llama3:8b 365c0bd3c000 6.2 GB 100% CPU 4 minutes from now camilleconte@camillecore:~$ sudo lsof -i :11434 [sudo] password for camilleconte: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ollama 58329 ollama 3u IPv4 2813691 0t0 TCP localhost:11434 (LISTEN) </details> - Here I show that I can interact with Ollama through HTTP requests in my terminal using `curl`: <details> camilleconte@camillecore:~$ curl -X POST http://localhost:11434/api/generate \ -H "Content-Type: application/json" \ -d '{ "model": "llama3:8b", "prompt": "What is the capital of France?", "temperature": 0.7 }' Response: {"model":"llama3:8b","created_at":"2024-09-17T16:26:43.660429962Z","response":"The","done":false} {"model":"llama3:8b","created_at":"2024-09-17T16:26:43.895378292Z","response":" capital","done":false} {"model":"llama3:8b","created_at":"2024-09-17T16:26:44.14832767Z","response":" of","done":false} {"model":"llama3:8b","created_at":"2024-09-17T16:26:44.403461964Z","response":" France","done":false} {"model":"llama3:8b","created_at":"2024-09-17T16:26:44.657318964Z","response":" is","done":false} {"model":"llama3:8b","created_at":"2024-09-17T16:26:44.910058883Z","response":" Paris","done":false} {"model":"llama3:8b","created_at":"2024-09-17T16:26:45.176020402Z","response":".","done":false} {"model":"llama3:8b","created_at":"2024-09-17T16:26:45.44038917Z","response":"","done":true,"done_reason":"stop","context":[128006,882,128007,271,3923,374,279,6864,315,9822,30,128009,128006,78191,128007,271,791,6864,315,9822,374,12366,13],"total_duration":8302556474,"load_duration":4573576269,"prompt_eval_count":17,"prompt_eval_duration":1893859000,"eval_count":8,"eval_duration":1780122000} </details> As you can see This interaction works perfectly and returns the correct responses directly in the terminal. Now I will show the external connections I've been trying to make that have failed. ### 2. **Issues with Connecting to external services** While interacting with Ollama locally works well, I’ve been facing persistent issues when trying to integrate it with external services, namely **Wren AI** and **Replit**. - **Wren AI**: Wren AI is intended to integrate with a local or external LLM host, in this case, my Ollama instance. However, whenever I try to connect Wren AI to the **localhost:11434** port where Ollama is running, I receive connection errors indicating that the API cannot be reached. These errors typically take the form of **"Connection refused"** errors, as Wren AI repeatedly fails to establish communication with the Ollama host. Following this (https://blog.getwren.ai/how-to-use-meta-llama-3-to-query-mysql-database-using-ollama-on-your-machine-2c087b204e41) tutorial, here are the **relevant files** in the docker project directory, where a container called wren-ai-service-1 forwards my ai service to port 5555: 1. .env.ai <details> ## LLM LLM_PROVIDER=ollama_llm # openai_llm, azure_openai_llm, ollama_llm GENERATION_MODEL=llama3:8b GENERATION_MODEL_KWARGS='{"temperature": 0}' # openai or openai-api-compatible #LLM_OPENAI_API_KEY=sk-xxxx #LLM_OPENAI_API_BASE=https://api.openai.com/v1 # azure_openai #LLM_AZURE_OPENAI_API_KEY= #LLM_AZURE_OPENAI_API_BASE= #LLM_AZURE_OPENAI_VERSION= # ollama LLM_OLLAMA_URL=http://host.docker.internal:11434 ## EMBEDDER EMBEDDER_PROVIDER=ollama_embedder # openai_embedder, azure_openai_embedder, ollama_embedder # supported embedding models providers by qdrant: https://qdrant.tech/documentation/embeddings/ EMBEDDING_MODEL=nomic-embed-text EMBEDDING_MODEL_DIMENSION=768 # openai or openai-api-compatible #EMBEDDER_OPENAI_API_KEY=sk-xxxx #EMBEDDER_OPENAI_API_BASE=https://api.openai.com/v1 # azure_openai #EMBEDDER_AZURE_OPENAI_API_KEY= #EMBEDDER_AZURE_OPENAI_API_BASE= #EMBEDDER_AZURE_OPENAI_VERSION= # ollama EMBEDDER_OLLAMA_URL=http://host.docker.internal:11434 ## DOCUMENT_STORE DOCUMENT_STORE_PROVIDER=qdrant QDRANT_HOST=qdrant </details> 2. .env <details> COMPOSE_PROJECT_NAME=wrenai PLATFORM=linux/amd64 PROJECT_DIR=/home/camilleconte/.wrenai # service port WREN_ENGINE_PORT=8080 WREN_ENGINE_SQL_PORT=7432 WREN_AI_SERVICE_PORT=5555 WREN_UI_PORT=3000 IBIS_SERVER_PORT=8000 # service endpoint (for docker-compose-dev.yaml file) WREN_UI_ENDPOINT=http://wren-ui:3000 # LLM #LLM_OPENAI_API_KEY= #EMBEDDER_OPENAI_API_KEY= LLM_PROVIDER=ollama_llm # openai_llm, azure_openai_llm, ollama_llm GENERATION_MODEL_KWARGS='{"temperature": 0}' GENERATION_MODEL=llama3:8b # gpt-4o-mini, gpt-4o, gpt-4-turbo, gpt-3.5-turbo LLM_OLLAMA_URL=http://host.docker.internal:11434 ## EMBEDDER EMBEDDER_PROVIDER=ollama_embedder # openai_embedder, azure_openai_embedder, ollama_embedder # supported embedding models providers by qdrant: https://qdrant.tech/documentation/embeddings/ EMBEDDING_MODEL=nomic-embed-text EMBEDDING_MODEL_DIMENSION=768 # version # CHANGE THIS TO THE LATEST VERSION WREN_PRODUCT_VERSION=0.7.5 WREN_ENGINE_VERSION=0.9.0 WREN_AI_SERVICE_VERSION=0.8.0 IBIS_SERVER_VERSION=0.9.0 WREN_UI_VERSION=0.9.2 WREN_BOOTSTRAP_VERSION=0.1.5 # AI service related env variables AI_SERVICE_ENABLE_TIMER= AI_SERVICE_LOGGING_LEVEL=INFO SHOULD_FORCE_DEPLOY= QDRANT_HOST=qdrant # user id (uuid v4) USER_UUID=1aa9be21-4235-4723-9632-6d1645571769 # for other services POSTHOG_API_KEY=phc_nhF32aj4xHXOZb0oqr2cn4Oy9uiWzz6CCP4KZmRq9aE POSTHOG_HOST=https://app.posthog.com TELEMETRY_ENABLED=true # the port exposes to the host # OPTIONAL: change the port if you have a conflict HOST_PORT=3000 AI_SERVICE_FORWARD_PORT=5555 </details> 3. docker-compose.yaml <details> version: "3.8" volumes: data: networks: wren: driver: bridge services: bootstrap: image: ghcr.io/canner/wren-bootstrap:${WREN_BOOTSTRAP_VERSION} restart: on-failure platform: ${PLATFORM} environment: DATA_PATH: /app/data volumes: - data:/app/data command: /bin/sh /app/init.sh wren-engine: image: ghcr.io/canner/wren-engine:${WREN_ENGINE_VERSION} restart: on-failure platform: ${PLATFORM} expose: - ${WREN_ENGINE_PORT} - ${WREN_ENGINE_SQL_PORT} volumes: - data:/usr/src/app/etc networks: - wren depends_on: - bootstrap ibis-server: image: ghcr.io/canner/wren-engine-ibis:${IBIS_SERVER_VERSION} restart: on-failure platform: ${PLATFORM} expose: - ${IBIS_SERVER_PORT} environment: WREN_ENGINE_ENDPOINT: http://wren-engine:${WREN_ENGINE_PORT} networks: - wren wren-ai-service: image: ghcr.io/canner/wren-ai-service:${WREN_AI_SERVICE_VERSION} restart: on-failure platform: ${PLATFORM} expose: - ${WREN_AI_SERVICE_PORT} ports: - ${AI_SERVICE_FORWARD_PORT}:${WREN_AI_SERVICE_PORT} environment: LLM_PROVIDER: ollama_llm LLM_OLLAMA_URL: http://host.docker.internal:11434 GENERATION_MODEL: llama3:8b GENERATION_MODEL_KWARGS: '{"temperature": 0}' EMBEDDER_PROVIDER: ollama_embedder EMBEDDING_MODEL: nomic-embed-text EMBEDDING_MODEL_DIMENSION: "768" #EMBEDDER_OPENAI_API_KEY: "" #EMBEDDER_AZURE_OPENAI_API_KEY: "" WREN_UI_PORT: "3000" WREN_UI_ENDPOINT: http://wren-ui:3000 WREN_AI_SERVICE_PORT: "5555" ENABLE_TIMER: "" LOGGING_LEVEL: INFO PYTHONUNBUFFERED: 1 networks: - wren depends_on: - wren-engine - qdrant qdrant: image: qdrant/qdrant:v1.7.4 restart: on-failure expose: - 6333 - 6334 volumes: - data:/qdrant/storage networks: - wren wren-ui: image: ghcr.io/canner/wren-ui:${WREN_UI_VERSION} restart: on-failure platform: ${PLATFORM} environment: DB_TYPE: sqlite # /app is the working directory in the container SQLITE_FILE: /app/data/db.sqlite3 WREN_ENGINE_ENDPOINT: http://wren-engine:${WREN_ENGINE_PORT} WREN_AI_ENDPOINT: http://wren-ai-service:${WREN_AI_SERVICE_PORT} IBIS_SERVER_ENDPOINT: http://ibis-server:${IBIS_SERVER_PORT} EMBEDDING_MODEL: ${EMBEDDING_MODEL} EMBEDDING_MODEL_DIMENSION: ${EMBEDDING_MODEL_DIMENSION} GENERATION_MODEL: ${GENERATION_MODEL} # telemetry WREN_ENGINE_PORT: ${WREN_ENGINE_PORT} WREN_AI_SERVICE_VERSION: ${WREN_AI_SERVICE_VERSION} WREN_UI_VERSION: ${WREN_UI_VERSION} WREN_ENGINE_VERSION: ${WREN_ENGINE_VERSION} USER_UUID: ${USER_UUID} POSTHOG_API_KEY: ${POSTHOG_API_KEY} POSTHOG_HOST: ${POSTHOG_HOST} TELEMETRY_ENABLED: ${TELEMETRY_ENABLED} # client side NEXT_PUBLIC_USER_UUID: ${USER_UUID} NEXT_PUBLIC_POSTHOG_API_KEY: ${POSTHOG_API_KEY} NEXT_PUBLIC_POSTHOG_HOST: ${POSTHOG_HOST} NEXT_PUBLIC_TELEMETRY_ENABLED: ${TELEMETRY_ENABLED} # configs WREN_PRODUCT_VERSION: ${WREN_PRODUCT_VERSION} ports: # HOST_PORT is the port you want to expose to the host machine - ${HOST_PORT}:3000 volumes: - data:/app/data networks: - wren depends_on: - wren-ai-service - wren-engine </details> - When start wrenai on docker, it looks like this: ![wren_running_docker](https://github.com/user-attachments/assets/27545f38-c9a4-4e0c-a347-4fa414ccde42) I can visit localhost3000 and it looks great, but then localhost5555 always says ERR_EMPTY_RESPONSE. The docker logs for the wren-ai-service-1 container look like this: <details> 2024-09-16 14:56:01 This module is deprecated and will be removed in Hamilton 2.0 Please use `hamilton.async_driver` instead. 2024-09-16 14:56:02 INFO: Started server process [7] 2024-09-16 14:56:02 INFO: Waiting for application startup. 2024-09-16 14:56:02 2024-09-16 18:56:02,058 - wren-ai-service - INFO - Initializing providers... (utils.py:64) 2024-09-16 14:56:03 2024-09-16 18:56:03,095 - wren-ai-service - INFO - Registering provider: qdrant (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,356 - wren-ai-service - INFO - Registering provider: azure_openai_embedder (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,359 - wren-ai-service - INFO - Registering provider: ollama_embedder (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,360 - wren-ai-service - INFO - Registering provider: openai_embedder (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,361 - wren-ai-service - INFO - Registering provider: wren_ui (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,361 - wren-ai-service - INFO - Registering provider: wren_ibis (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,361 - wren-ai-service - INFO - Registering provider: wren_engine (loader.py:66) 2024-09-16 11:42:03 Waiting for wren-ai-service to start... 2024-09-16 14:55:59 Waiting for wren-ai-service to start... 2024-09-16 14:56:03 2024-09-16 18:56:03,368 - wren-ai-service - INFO - Registering provider: azure_openai_llm (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,372 - wren-ai-service - INFO - Registering provider: ollama_llm (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,372 - wren-ai-service - INFO - Registering provider: openai_llm (loader.py:66) 2024-09-16 14:56:03 2024-09-16 18:56:03,402 - wren-ai-service - INFO - Pulling Ollama model llama3:8b (loader.py:109) 2024-09-16 14:56:05 2024-09-16 18:56:05,211 - wren-ai-service - INFO - Pulling Ollama model llama3:8b: 100% (loader.py:116) 2024-09-16 14:56:05 2024-09-16 18:56:05,217 - wren-ai-service - INFO - Using Ollama LLM: llama3:8b (ollama.py:135) 2024-09-16 14:56:05 2024-09-16 18:56:05,217 - wren-ai-service - INFO - Using Ollama URL: http://host.docker.internal:11434 (ollama.py:136) 2024-09-16 14:56:05 ERROR: Traceback (most recent call last): 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions 2024-09-16 14:56:05 yield 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 233, in handle_request 2024-09-16 14:56:05 resp = self._pool.handle_request(req) 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request 2024-09-16 14:56:05 raise exc from None 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request 2024-09-16 14:56:05 response = connection.handle_request( 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 99, in handle_request 2024-09-16 14:56:05 raise exc 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 76, in handle_request 2024-09-16 14:56:05 stream = self._connect(request) 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 122, in _connect 2024-09-16 14:56:05 stream = self._network_backend.connect_tcp(**kwargs) 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp 2024-09-16 14:56:05 with map_exceptions(exc_map): 2024-09-16 14:56:05 File "/usr/local/lib/python3.12/contextlib.py", line 155, in __exit__ 2024-09-16 14:56:05 self.gen.throw(value) 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions 2024-09-16 14:56:05 raise to_exc(exc) from exc 2024-09-16 14:56:05 httpcore.ConnectError: [Errno 111] Connection refused 2024-09-16 14:56:05 2024-09-16 14:56:05 The above exception was the direct cause of the following exception: 2024-09-16 14:56:05 2024-09-16 14:56:05 Traceback (most recent call last): 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/starlette/routing.py", line 734, in lifespan 2024-09-16 14:56:05 async with self.lifespan_context(app) as maybe_state: 2024-09-16 14:56:05 File "/usr/local/lib/python3.12/contextlib.py", line 204, in __aenter__ 2024-09-16 14:56:05 return await anext(self.gen) 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/src/__main__.py", line 28, in lifespan 2024-09-16 14:56:05 container.init_globals() 2024-09-16 14:56:05 File "/src/globals.py", line 53, in init_globals 2024-09-16 14:56:05 llm_provider, embedder_provider, document_store_provider, engine = init_providers( 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/src/utils.py", line 68, in init_providers 2024-09-16 14:56:05 embedder_provider = loader.get_provider( 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/src/providers/embedder/ollama.py", line 170, in __init__ 2024-09-16 14:56:05 pull_ollama_model(self._url, self._embedding_model) 2024-09-16 14:56:05 File "/src/providers/loader.py", line 107, in pull_ollama_model 2024-09-16 14:56:05 models = client.list()["models"] 2024-09-16 14:56:05 ^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/ollama/_client.py", line 333, in list 2024-09-16 14:56:05 return self._request('GET', '/api/tags').json() 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/ollama/_client.py", line 69, in _request 2024-09-16 14:56:05 response = self._client.request(method, url, **kwargs) 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_client.py", line 827, in request 2024-09-16 14:56:05 return self.send(request, auth=auth, follow_redirects=follow_redirects) 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_client.py", line 914, in send 2024-09-16 14:56:05 response = self._send_handling_auth( 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth 2024-09-16 14:56:05 response = self._send_handling_redirects( 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_client.py", line 979, in _send_handling_redirects 2024-09-16 14:56:05 response = self._send_single_request(request) 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1015, in _send_single_request 2024-09-16 14:56:05 response = transport.handle_request(request) 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 232, in handle_request 2024-09-16 14:56:05 with map_httpcore_exceptions(): 2024-09-16 14:56:05 File "/usr/local/lib/python3.12/contextlib.py", line 155, in __exit__ 2024-09-16 14:56:05 self.gen.throw(value) 2024-09-16 14:56:05 File "/app/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions 2024-09-16 14:56:05 raise mapped_exc(message) from exc 2024-09-16 14:56:05 httpx.ConnectError: [Errno 111] Connection refused 2024-09-16 14:56:05 2024-09-16 14:56:05 ERROR: Application startup failed. Exiting. </details> At first, I thought it was an issue with docker communicating with ollama, so I entered the wren-ai-service-1 container in my terminal( docker exec -it wrenai-wren-ai-service-1 /bin/bash). I wanted to see if i could verify the docker connection in there, so I did this and succeeded as you can see here: <details> root@1773b32d1441:/# curl http://host.docker.internal:11434/v1/models {"object":"list","data":[{"id":"llama3:8b","object":"model","created":1726592861,"owned_by":"library"},{"id":"llama3:70b","object":"model","created":1724455596,"owned_by":"library"},{"id":"nomic-embed-text:latest","object":"model","created":1724455596,"owned_by":"library"}]} root@1773b32d1441:/# curl -X POST http://host.docker.internal:11434/api/generate -H "Content-Type: application/json" -d '{ "model": "llama3:8b", "prompt": "What is the capital of France?", "temperature": 0.7 }' {"model":"llama3:8b","created_at":"2024-09-17T17:15:43.734078981Z","response":"The","done":false} {"model":"llama3:8b","created_at":"2024-09-17T17:15:43.990105229Z","response":" capital","done":false} {"model":"llama3:8b","created_at":"2024-09-17T17:15:44.25267107Z","response":" of","done":false} {"model":"llama3:8b","created_at":"2024-09-17T17:15:44.513833801Z","response":" France","done":false} {"model":"llama3:8b","created_at":"2024-09-17T17:15:44.766417457Z","response":" is","done":false} {"model":"llama3:8b","created_at":"2024-09-17T17:15:45.037476835Z","response":" Paris","done":false} {"model":"llama3:8b","created_at":"2024-09-17T17:15:45.306243202Z","response":".","done":false} {"model":"llama3:8b","created_at":"2024-09-17T17:15:45.557952927Z","response":"","done":true,"done_reason":"stop","context":[128006,882,128007,271,3923,374,279,6864,315,9822,30,128009,128006,78191,128007,271,791,6864,315,9822,374,12366,13],"total_duration":8320738087,"load_duration":4567994305,"prompt_eval_count":17,"prompt_eval_duration":1882077000,"eval_count":8,"eval_duration":1823834000} </details> Sure enough, the curl request to Ollama worked fine in the docker container through the docker host url. So I confirmed that I could access the ollama service just fine from both my environments; inside the Docker container and directly on the host machine. However, just to be thorough, I should mention that when I am inside the container in terminal and send the request to localhost:11434 instead of host.docker.internal:11434, it says "curl: (7) Failed to connect to localhost port 11434 after 4 ms: Couldn't connect to server". So next, I thought the problem was with wrenAI, but then this happened: - **Replit**: This was a random sidequest that ended up being pretty informative in the end. Basically I didn't like the output of the llama3:8b model in my terminal before I knew about the "ollama run llama3:8b" command (LOL) So I tried to use replit (a cloud-based development environment) to host its responses in a nicer format. These are the steps I took: <details> Step 1: Create a Replit Account Step 2: Create a New Repl, Click "Create" in the top left corner of the dashboard. Choose Python as the programming language. Name the project (e.g., Ollama Chat), then click Create Repl. Step 3: Set Up the API Call Install Dependencies: In the Repl’s file explorer, create a requirements.txt file and add: 'requests' Then I Created a main.py file and added this code to interact with Ollama: import requests import json url = "http://localhost:11434/api/generate" data = { "model": "llama3:8b", "prompt": "What is the capital of France?", "temperature": 0.7 } response = requests.post(url, headers={"Content-Type": "application/json"}, data=json.dumps(data)) print(response.json()) </details> I ran the code, and I got these errors: <details> Traceback (most recent call last): File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/connection.py", line 199, in _new_conn sock = connection.create_connection( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/connectionpool.py", line 789, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/connectionpool.py", line 495, in _make_request conn.request( File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/connection.py", line 441, in request self.endheaders() File "/nix/store/f98g7xbckgqbkagdvpzc2r6lv3h1p9ki-python3-3.11.9/lib/python3.11/http/client.py", line 1298, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/nix/store/f98g7xbckgqbkagdvpzc2r6lv3h1p9ki-python3-3.11.9/lib/python3.11/http/client.py", line 1058, in _send_output self.send(msg) File "/nix/store/f98g7xbckgqbkagdvpzc2r6lv3h1p9ki-python3-3.11.9/lib/python3.11/http/client.py", line 996, in send self.connect() File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/connection.py", line 279, in connect self.sock = self._new_conn() ^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/connection.py", line 214, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7b2fd58958d0>: Failed to establish a new connection: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/connectionpool.py", line 843, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=11434): Max retries exceeded with url: /api/generate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7b2fd58958d0>: Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/runner/ollama-8b-chat/main.py", line 15, in <module> response = requests.post(url, headers={"Content-Type": "application/json"}, data=json.dumps(data)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/requests/api.py", line 115, in post return request("post", url, data=data, json=json, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/ollama-8b-chat/.pythonlibs/lib/python3.11/site-packages/requests/adapters.py", line 700, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=11434): Max retries exceeded with url: /api/generate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7b2fd58958d0>: Failed to establish a new connection: [Errno 111] Connection refused')) </details> As you can see, it threw the same [Errno 111] Connection refused. This error indicates that **localhost:11434** cannot be reached from Replit, which makes sense because Replit runs in the cloud and doesn’t have access to my local machine’s `localhost`. The same problem occurs in Wren AI because, like Replit, it's not running on my local machine but rather trying to connect to my local instance over HTTP. So this is what has convinced me that there's something wrong with my ollama service, as it does not seem to allow any external services to interact with it despite how it is functioning just fine on my machine locally. Note: it also gave the same errors when I changed the python script to http://host.docker.internal:11434/api/generate instead of localhost:11434. ### 3. **Suspicions and Additional Troubleshooting Attempts** 1. One of the first things I suspected was that Ollama’s default CORS (Cross-Origin Resource Sharing) policy might be preventing access from external sources like Replit and Wren AI, as by default, Ollama’s CORS policy is restricted to **localhost**. To resolve this, I tried editing the **Ollama systemd service** configuration like I described earlier. But again, despite making these changes, both Wren AI and Replit continued to throw the same **connection refused** errors. 2. I noticed that a lot of the error files are coming from a python "httpx" directory, and I have read on ollama's documentation faq (https://github.com/ollama/ollama/blob/main/docs/faq.md#setting-environment-variables-on-linux) that one should "Avoid setting HTTP_PROXY. Ollama does not use HTTP for model pulls, only HTTPS. Setting HTTP_PROXY may interrupt client connections to the server." No idea id this is a rabbithole of relevance though. On the subject of proxies, I know that there's an option to expose my ollama- like using a tool like **ngrok** to expose my locally running Ollama instance to the internet so that Wren AI and Replit can access it via an externally accessible URL, or deploying Ollama on a cloud server or a VPS. I do not want to expose my ollama server, and I feel like I shouldn't have to, right? Again, I am a beginner, so if any of you experts think this is my only option I would love to hear it from you. ### Okay that is all! Any help, suggestions, comments etc would be greatly appreciated because I am feeling very stuck. Thanks!
GiteaMirror commented 2026-04-12 15:15:04 -05:00
Author
Owner
Copy Link

@camilleconte8 commented on GitHub (Sep 17, 2024):

It is easier to read all this in this md file, so here it is attached.
blogpost.md

<!-- gh-comment-id:2356785470 --> @camilleconte8 commented on GitHub (Sep 17, 2024): It is easier to read all this in this md file, so here it is attached. [blogpost.md](https://github.com/user-attachments/files/17034324/blogpost.md)
GiteaMirror commented 2026-04-12 15:15:05 -05:00
Author
Owner
Copy Link

@mxyng commented on GitHub (Sep 19, 2024):

I think the issue is your configuration specifies Ollama as both LLM and embedder providers and sets LLM_OLLAMA_URL but doesn't set EMBEDDER_OLLAMA_URL.

2024-09-16 14:56:05   File "/src/utils.py", line 68, in init_providers
2024-09-16 14:56:05     embedder_provider = loader.get_provider(
2024-09-16 14:56:05                         ^^^^^^^^^^^^^^^^^^^^
2024-09-16 14:56:05   File "/src/providers/embedder/ollama.py", line 170, in __init__
2024-09-16 14:56:05     pull_ollama_model(self._url, self._embedding_model)

        environment:
          LLM_PROVIDER: ollama_llm
          LLM_OLLAMA_URL: http://host.docker.internal:11434
          GENERATION_MODEL: llama3:8b
          GENERATION_MODEL_KWARGS: '{"temperature": 0}'
          EMBEDDER_PROVIDER: ollama_embedder
          EMBEDDING_MODEL: nomic-embed-text
          EMBEDDING_MODEL_DIMENSION: "768"
          #EMBEDDER_OPENAI_API_KEY: ""
          #EMBEDDER_AZURE_OPENAI_API_KEY: ""
          WREN_UI_PORT: "3000"
          WREN_UI_ENDPOINT: http://wren-ui:3000
          WREN_AI_SERVICE_PORT: "5555"
          ENABLE_TIMER: ""
          LOGGING_LEVEL: INFO
          PYTHONUNBUFFERED: 1
<!-- gh-comment-id:2359646307 --> @mxyng commented on GitHub (Sep 19, 2024): I think the issue is your configuration specifies Ollama as both LLM and embedder providers and sets `LLM_OLLAMA_URL` but doesn't set `EMBEDDER_OLLAMA_URL.` ```console 2024-09-16 14:56:05 File "/src/utils.py", line 68, in init_providers 2024-09-16 14:56:05 embedder_provider = loader.get_provider( 2024-09-16 14:56:05 ^^^^^^^^^^^^^^^^^^^^ 2024-09-16 14:56:05 File "/src/providers/embedder/ollama.py", line 170, in __init__ 2024-09-16 14:56:05 pull_ollama_model(self._url, self._embedding_model) ``` ```yaml environment: LLM_PROVIDER: ollama_llm LLM_OLLAMA_URL: http://host.docker.internal:11434 GENERATION_MODEL: llama3:8b GENERATION_MODEL_KWARGS: '{"temperature": 0}' EMBEDDER_PROVIDER: ollama_embedder EMBEDDING_MODEL: nomic-embed-text EMBEDDING_MODEL_DIMENSION: "768" #EMBEDDER_OPENAI_API_KEY: "" #EMBEDDER_AZURE_OPENAI_API_KEY: "" WREN_UI_PORT: "3000" WREN_UI_ENDPOINT: http://wren-ui:3000 WREN_AI_SERVICE_PORT: "5555" ENABLE_TIMER: "" LOGGING_LEVEL: INFO PYTHONUNBUFFERED: 1 ```
GiteaMirror commented 2026-04-12 15:15:05 -05:00
Author
Owner
Copy Link

@SRDP commented on GitHub (Sep 20, 2024):

@camilleconte8: Did you face this server error while connecting to db

('08S01', '[08S01] [FreeTDS][SQL Server] Unable to connect: Adaptive Server is unavailable or does not exist (20009) (SQLDriverConnect)')

<!-- gh-comment-id:2363561357 --> @SRDP commented on GitHub (Sep 20, 2024): @camilleconte8: Did you face this server error while connecting to db ('08S01', '[08S01] [FreeTDS][SQL Server] Unable to connect: Adaptive Server is unavailable or does not exist (20009) (SQLDriverConnect)')
GiteaMirror commented 2026-04-12 15:15:06 -05:00
Author
Owner
Copy Link

@muhdnabil370 commented on GitHub (May 28, 2025):

i setup up wrenai + docker +ollama + mysql

then open this page for wrenai UI http://localhost:3000/setup/connection.

then encounter this , error
Internal server error
Cannot read properties of undefined (reading 'data')

but when i try using wrenai launcher, it does not have this error.

reason setup wrenai in docker instead wrenai launcher, easier for me to scale up and scale down

<!-- gh-comment-id:2914711221 --> @muhdnabil370 commented on GitHub (May 28, 2025): i setup up wrenai + docker +ollama + mysql then open this page for wrenai UI http://localhost:3000/setup/connection. then encounter this , error Internal server error Cannot read properties of undefined (reading 'data') but when i try using wrenai launcher, it does not have this error. reason setup wrenai in docker instead wrenai launcher, easier for me to scale up and scale down
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#4321
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?