feat: configurable OpenAI API endpoint. #95

@justinh-rahb commented on GitHub (Dec 16, 2023):

Would love to see this as well. Supposedly the new Mistral API platform is OpenAI API compatible, so that too would be an option available that I'm personally most interested in right now. In addition to Azure OpenAPI endpoints, LiteLLM, etc.

Bonus if it doesn't need to be an either/or situation, like if we could add endpoints in addition to the default OpenAI one, that would be most beneficial.

@justinh-rahb commented on GitHub (Dec 16, 2023): Would love to see this as well. Supposedly the new Mistral API platform is OpenAI API compatible, so that too would be an option available that I'm personally most interested in right now. In addition to Azure OpenAPI endpoints, LiteLLM, etc. Bonus if it doesn't need to be an either/or situation, like if we could add endpoints in addition to the default OpenAI one, that would be most beneficial.

GiteaMirror commented

@tjbck commented on GitHub (Dec 21, 2023):

Actively taking a look! One question I have is, how would users fetch the list of available models? Do other services offer api endpoints equivalent to https://api.openai.com/v1/models?

@tjbck commented on GitHub (Dec 21, 2023): Actively taking a look! One question I have is, how would users fetch the list of available models? Do other services offer api endpoints equivalent to `https://api.openai.com/v1/models`?

GiteaMirror commented

@Flowm commented on GitHub (Dec 21, 2023):

The Mistral API seems to be very similar (Docs, API spec)

@Flowm commented on GitHub (Dec 21, 2023): The Mistral API seems to be very similar ([Docs](https://docs.mistral.ai/), [API spec](https://docs.mistral.ai/api/))

GiteaMirror commented

@tjbck commented on GitHub (Dec 21, 2023):

@Flowm, Could you try to check if Mistral api supports the /models api route? Thanks!

@tjbck commented on GitHub (Dec 21, 2023): @Flowm, Could you try to check if Mistral api supports the `/models` api route? Thanks!

GiteaMirror commented

@ahkhan03 commented on GitHub (Dec 21, 2023):

I tried it with mistral and anyscale, and both support /models route. I tried several others in the past few days, and all of them supported it, but I have confirmed for these two just now. They have additional fields in the returned JSON, but they all have data.id, data.object, and data.owned_by in common, which should be enough for integrating with OpenAI compatible tools.

@ahkhan03 commented on GitHub (Dec 21, 2023): I tried it with mistral and anyscale, and both support `/models` route. I tried several others in the past few days, and all of them supported it, but I have confirmed for these two just now. They have additional fields in the returned JSON, but they all have `data.id`, `data.object`, and `data.owned_by` in common, which should be enough for integrating with OpenAI compatible tools.

GiteaMirror commented

@justinh-rahb commented on GitHub (Dec 21, 2023):

@tjbck I'd be happy to give you an API key (with an expiry date of course) for Mistral if it'll help you get this feature completed. You can find my contact info in my profile.

@justinh-rahb commented on GitHub (Dec 21, 2023): @tjbck I'd be happy to give you an API key (with an expiry date of course) for Mistral if it'll help you get this feature completed. You can find my contact info in my profile.

GiteaMirror commented

@chymian commented on GitHub (Dec 22, 2023):

to name one more on the ever growing list of openAI compatible APIs with resonable pricing: together.ai

API-endpoint: https://api.together.xyz/
see: https://docs.together.ai/docs/openai-api-compatibility

they offer a $25 entry bonus and prices for 1M token between $0.1 to $0.9.
@Flowm @justinh-rahb , the also offer the new Mixtral Mod. for very reconable prices.

curl -s  https://api.together.xyz/v1/models -H "Content-Type: application/json"  -H "Authorization: Bearer 1bef28e8a23fa3a11xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"|jq
[
  {
    "id": "Austism/chronos-hermes-13b",
    "object": "model",
    "created": 1692896905
  },
  {
    "id": "DiscoResearch/DiscoLM-mixtral-8x7b-v2",
    "object": "model",
    "created": 1702245101
  },
  {
    "id": "EleutherAI/llemma_7b",
    "object": "model",
    "created": 1697837306
  },
  {
    "id": "Gryphe/MythoMax-L2-13b",
    "object": "model",
    "created": 1693943905
  },
  {
    "id": "Meta-Llama/Llama-Guard-7b",
    "object": "model",
    "created": 0
  },
  {
    "id": "Nexusflow/NexusRaven-V2-13B",
    "object": "model",
    "created": 1701796524
  },
  {
    "id": "NousResearch/Nous-Capybara-7B-V1p9",
    "object": "model",
    "created": 1701203533
  },
  {
    "id": "NousResearch/Nous-Hermes-Llama2-13b",
    "object": "model",
    "created": 1691017613
  },
  {
    "id": "NousResearch/Nous-Hermes-Llama2-70b",
    "object": "model",
    "created": 1697837306
  }
  
...

@chymian commented on GitHub (Dec 22, 2023): to name one more on the ever growing list of openAI compatible APIs with resonable pricing: __together.ai__ API-endpoint: https://api.together.xyz/ see: https://docs.together.ai/docs/openai-api-compatibility they offer a $25 entry bonus and prices for 1M token between $0.1 to $0.9. @Flowm @justinh-rahb , the also offer the new Mixtral Mod. for very reconable prices. ```bash curl -s https://api.together.xyz/v1/models -H "Content-Type: application/json" -H "Authorization: Bearer 1bef28e8a23fa3a11xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"|jq [ { "id": "Austism/chronos-hermes-13b", "object": "model", "created": 1692896905 }, { "id": "DiscoResearch/DiscoLM-mixtral-8x7b-v2", "object": "model", "created": 1702245101 }, { "id": "EleutherAI/llemma_7b", "object": "model", "created": 1697837306 }, { "id": "Gryphe/MythoMax-L2-13b", "object": "model", "created": 1693943905 }, { "id": "Meta-Llama/Llama-Guard-7b", "object": "model", "created": 0 }, { "id": "Nexusflow/NexusRaven-V2-13B", "object": "model", "created": 1701796524 }, { "id": "NousResearch/Nous-Capybara-7B-V1p9", "object": "model", "created": 1701203533 }, { "id": "NousResearch/Nous-Hermes-Llama2-13b", "object": "model", "created": 1691017613 }, { "id": "NousResearch/Nous-Hermes-Llama2-70b", "object": "model", "created": 1697837306 } ... ```

GiteaMirror commented

@tjbck commented on GitHub (Dec 23, 2023):

Hey guys! Just added the custom api endpoint feature to #265 branch. If any of you guys could try it out and confirm that it works, I'll merge it to main! Thanks!

@tjbck commented on GitHub (Dec 23, 2023): Hey guys! Just added the custom api endpoint feature to #265 branch. If any of you guys could try it out and confirm that it works, I'll merge it to main! Thanks!

GiteaMirror commented

2025-11-11 14:05:13 -06:00

@justinh-rahb commented on GitHub (Dec 23, 2023):

Hey guys! Just added the custom api endpoint feature to #265 branch. If any of you guys could try it out and confirm that it works, I'll merge it to main! Thanks!

@tjbck Loading the model names from the /models endpoint is working:

However, generation is not responding:

I got the same response with all models. Unfortunately the Docker log doesn't show me anything useful, anywhere else I can look or enable debugging?

@justinh-rahb commented on GitHub (Dec 23, 2023): > Hey guys! Just added the custom api endpoint feature to #265 branch. If any of you guys could try it out and confirm that it works, I'll merge it to main! Thanks! @tjbck Loading the model names from the /models endpoint is working: <img width="778" alt="Screenshot 2023-12-22 at 11 55 51 PM" src="https://github.com/ollama-webui/ollama-webui/assets/52832301/cee2d382-0e7c-4a37-a7da-30e2e7ad0705"> However, generation is not responding: <img width="778" alt="Screenshot 2023-12-22 at 11 58 15 PM" src="https://github.com/ollama-webui/ollama-webui/assets/52832301/d9f4d46d-4daa-4e8d-989f-5eb3f9dc1906"> I got the same response with all models. Unfortunately the Docker log doesn't show me anything useful, anywhere else I can look or enable debugging?

GiteaMirror commented

@ahkhan03 commented on GitHub (Dec 23, 2023):

@tjbck I just tried this branch. The model list loads correctly, but it shows a CORS policy issue in the browser console.

Access to fetch at 'https://api.mistral.ai/v1/chat/completions' from origin 'http://localhost:8080' has been blocked by CORS policy: Request header field x-title is not allowed by Access-Control-Allow-Headers in preflight response.

POST https://api.mistral.ai/v1/chat/completions net::ERR_FAILED

TypeError: Failed to fetch
at window.fetch (start.c756bce8.js:1:1402)
at X (3.92fc93bc.js:4:1895)
at async 3.92fc93bc.js:1:6227
at async Promise.all (index 0)
at async z (3.92fc93bc.js:1:6142)
at async ce (3.92fc93bc.js:7:2070)```

@ahkhan03 commented on GitHub (Dec 23, 2023): @tjbck I just tried this branch. The model list loads correctly, but it shows a CORS policy issue in the browser console. ```Access to fetch at 'https://api.mistral.ai/v1/chat/completions' from origin 'http://localhost:8080' has been blocked by CORS policy: Request header field x-title is not allowed by Access-Control-Allow-Headers in preflight response.``` ```POST https://api.mistral.ai/v1/chat/completions net::ERR_FAILED``` ``` TypeError: Failed to fetch at window.fetch (start.c756bce8.js:1:1402) at X (3.92fc93bc.js:4:1895) at async 3.92fc93bc.js:1:6227 at async Promise.all (index 0) at async z (3.92fc93bc.js:1:6142) at async ce (3.92fc93bc.js:7:2070)```

GiteaMirror commented

2025-11-11 14:05:13 -06:00

@tjbck commented on GitHub (Dec 23, 2023):

Just a header problem it seems like, I'll remove some fields and let you guys know!

@tjbck commented on GitHub (Dec 23, 2023): Just a header problem it seems like, I'll remove some fields and let you guys know!

GiteaMirror commented

@tjbck commented on GitHub (Dec 23, 2023):

Please try again and let me know!

@tjbck commented on GitHub (Dec 23, 2023): Please try again and let me know!

GiteaMirror commented

@ahkhan03 commented on GitHub (Dec 23, 2023):

I tested it for anyscale and mistral models and it is working perfectly now.

@ahkhan03 commented on GitHub (Dec 23, 2023): I tested it for anyscale and mistral models and it is working perfectly now.

GiteaMirror commented

@justinh-rahb commented on GitHub (Dec 23, 2023):

I can archive my chatbot-ui forks now 😂

@justinh-rahb commented on GitHub (Dec 23, 2023): I can archive my chatbot-ui forks now 😂

GiteaMirror commented