github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 20:54:54 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #2812] Allow integration with Slurm #1704

New Issue
Open
opened 2026-04-12 11:40:47 -05:00 by GiteaMirror · 2 comments
GiteaMirror commented 2026-04-12 11:40:47 -05:00
Owner
Copy Link

Originally created by @iamashwin99 on GitHub (Feb 28, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2812

Slurm is a utility to manage and schedule workloads on a cluster of computers.
Many academic institutions use it for distributing computation.

I was wondering if it would be a good idea to implement an interface that allows use of chat interface with the model loaded via Slurm jobs.
This way your request gets queued, and when the computation is done, ollama will pipe the output.

There could be a number of ways to do this :

  • Allow single-response subcommands which start the server, run the query and kill the server when the output is received, for eg:
$ ollama singlerun mistral --message "what is the meaning of life"
[answer]
  • integrate the slurm scheduling within Ollama
  • Write a wrapper around ollama that does point 1.

Im hoping to have a discussion on what the community thinks about this topic.

Originally created by @iamashwin99 on GitHub (Feb 28, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2812 Slurm is a utility to manage and schedule workloads on a cluster of computers. Many academic institutions use it for distributing computation. I was wondering if it would be a good idea to implement an interface that allows use of chat interface with the model loaded via Slurm jobs. This way your request gets queued, and when the computation is done, ollama will pipe the output. There could be a number of ways to do this : - Allow single-response subcommands which start the server, run the query and kill the server when the output is received, for eg: ```console $ ollama singlerun mistral --message "what is the meaning of life" [answer] ``` - integrate the slurm scheduling within Ollama - Write a wrapper around ollama that does point 1. Im hoping to have a discussion on what the community thinks about this topic.
GiteaMirror added the feature request label 2026-04-12 11:40:47 -05:00
GiteaMirror commented 2026-04-12 11:40:48 -05:00
Author
Owner
Copy Link

@stanier commented on GitHub (Mar 2, 2024):

Personally, I would go a different route. I have similarly asked myself repeatedly over the past year or so what the best way to do simple LLM inference in a cluster compute environment is. But I never felt the explicit need to integrate with the job scheduler for a couple reasons:

  • additional complexity with zero inherent value gain
  • resource allocation for a short-lived or very-short-lived job such as running inference against a single query makes absolutely no sense, even for a dedicated interactive queue plentiful in GRES. Even if this were not to matter to the parties involved, I'd imagine sysops being naturally miffed by the overhead on the scheduler. Instead of allocating for such short-lived jobs, you'd get better mileage just invoking ollama from an interactive job
  • single user inference against dedicated hardware for most LLMs is actually inherently wasteful of resources from my understanding, because you will most likely not be exploiting the benefits of inference batching
  • ollama, and the llama.cpp/GGML library it is built upon are more appropriate for enabling inference on consumer-grade hardware rather than a cluster, as I believe is part of the separate intentions of both. Most centers already have HF Transformers, ONNX, and other backends installed as modules for inference against various network architectures, which are built with enterprise hardware and scaling in mind.

Generally I think it is an ill fit and would advise against, only because you have other options that are much more practical which I would highly recommend considering. But I could also be looking at this the wrong way somehow, as admittedly is sometimes characteristic of myself... I'm open to hear what others have to think on this and hear if there are maybe some needs/constraints I'm not taking into account.

edit: On second thought, I may have spoke too hastily about not using llama.cpp/GGML for inference in a cluster environment, as it would be entirely valid in use cases where you have maybe ~4-40GB VRAM to work with. In the same way, ollama would make some sense for convenience. I still can't get behind the use of such short-lived jobs, but if it is a small (less than a dozen nodes) cluster dedicated to one group, it may make sense for them and I have no room to speak in that case.

<!-- gh-comment-id:1974107699 --> @stanier commented on GitHub (Mar 2, 2024): Personally, I would go a different route. I have similarly asked myself repeatedly over the past year or so what the best way to do simple LLM inference in a cluster compute environment is. But I never felt the explicit need to integrate with the job scheduler for a couple reasons: * additional complexity with zero inherent value gain * resource allocation for a short-lived or very-short-lived job such as running inference against a single query makes absolutely no sense, even for a dedicated interactive queue plentiful in GRES. Even if this were not to matter to the parties involved, I'd imagine sysops being naturally miffed by the overhead on the scheduler. Instead of allocating for such short-lived jobs, you'd get better mileage just invoking ollama from an interactive job * single user inference against dedicated hardware for most LLMs is actually inherently wasteful of resources from my understanding, because you will most likely not be exploiting the benefits of inference batching * ollama, and the llama.cpp/GGML library it is built upon are more appropriate for enabling inference on consumer-grade hardware rather than a cluster, as I believe is part of the separate intentions of both. Most centers already have HF Transformers, ONNX, and other backends installed as modules for inference against various network architectures, which are built with enterprise hardware and scaling in mind. Generally I think it is an ill fit and would advise against, only because you have other options that are much more practical which I would highly recommend considering. But I could also be looking at this the wrong way *somehow*, as admittedly is sometimes characteristic of myself... I'm open to hear what others have to think on this and hear if there are maybe some needs/constraints I'm not taking into account. edit: On second thought, I may have spoke too hastily about not using llama.cpp/GGML for inference in a cluster environment, as it would be entirely valid in use cases where you have maybe ~4-40GB VRAM to work with. In the same way, ollama would make some sense for convenience. I still can't get behind the use of such short-lived jobs, but if it is a small (less than a dozen nodes) cluster dedicated to one group, it may make sense for them and I have no room to speak in that case.
GiteaMirror commented 2026-04-12 11:40:48 -05:00
Author
Owner
Copy Link

@wwjCMP commented on GitHub (Apr 27, 2024):

I believe this improvement is valuable.

<!-- gh-comment-id:2080513644 --> @wwjCMP commented on GitHub (Apr 27, 2024): I believe this improvement is valuable.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#1704
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?