github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #5068] please add nvidia/Nemotron-4-340B-Instruct #3200

New Issue
Open
opened 2026-04-12 13:41:32 -05:00 by GiteaMirror · 9 comments
GiteaMirror commented 2026-04-12 13:41:32 -05:00
Owner
Copy Link

Originally created by @gileneusz on GitHub (Jun 15, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/5068

my GPUs are not utilized fully, I need to spin my H200s!!

just kidding, need quantized version of the model ;)

Originally created by @gileneusz on GitHub (Jun 15, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/5068 my GPUs are not utilized fully, I need to spin my H200s!! just kidding, need quantized version of the model ;)
GiteaMirror added the model label 2026-04-12 13:41:32 -05:00
GiteaMirror commented 2026-04-12 13:41:36 -05:00
Author
Owner
Copy Link

@bmj-sys commented on GitHub (Jun 16, 2024):

How to run this model on macOS ? #nemotron

<!-- gh-comment-id:2171676650 --> @bmj-sys commented on GitHub (Jun 16, 2024): How to run this model on macOS ? #nemotron
GiteaMirror commented 2026-04-12 13:41:39 -05:00
Author
Owner
Copy Link

@Nehc commented on GitHub (Jun 17, 2024):

the quantized version, theoretically, will require 170 GB of RAM, right? In general, this is not so hopeless! )

<!-- gh-comment-id:2173541538 --> @Nehc commented on GitHub (Jun 17, 2024): the quantized version, theoretically, will require 170 GB of RAM, right? In general, this is not so hopeless! )
GiteaMirror commented 2026-04-12 13:41:42 -05:00
Author
Owner
Copy Link

@sujumayas commented on GitHub (Jun 17, 2024):

I am tempted to buy a new mac...

<!-- gh-comment-id:2174430995 --> @sujumayas commented on GitHub (Jun 17, 2024): I am tempted to buy a new mac...
GiteaMirror commented 2026-04-12 13:41:42 -05:00
Author
Owner
Copy Link

@kozuch commented on GitHub (Jun 28, 2024):

llama3:70b-instruct-q2_K (26GB in size, cca 2.9bpw) actually does not perform that bad. On Nemotron we could probably test how agresive quantization behaves with large models. 2.9bpw for Nemotron would make about 126 GB. Can still be run for few $s on rented HW. Would be really interesting to test.

<!-- gh-comment-id:2196798878 --> @kozuch commented on GitHub (Jun 28, 2024): llama3:70b-instruct-q2_K (26GB in size, cca 2.9bpw) actually does not perform that bad. On Nemotron we could probably test how agresive quantization behaves with large models. 2.9bpw for Nemotron would make about 126 GB. Can still be run for few $s on rented HW. Would be really interesting to test.
GiteaMirror commented 2026-04-12 13:41:43 -05:00
Author
Owner
Copy Link

@barclaybrown commented on GitHub (Aug 18, 2024):

Agree. In some applications, speed doesn't matter too much. Often I'm building applications that iterate through a dataset and use an LLM to do something on each item. Whether these take a few minutes or a few hours doesn't really matter, so I usually develop and test with a small, fast model, but then run the final with a bigger, smarter model. Would love to have Nemotron as an option, even it it runs slowly.

<!-- gh-comment-id:2295239395 --> @barclaybrown commented on GitHub (Aug 18, 2024): Agree. In some applications, speed doesn't matter too much. Often I'm building applications that iterate through a dataset and use an LLM to do something on each item. Whether these take a few minutes or a few hours doesn't really matter, so I usually develop and test with a small, fast model, but then run the final with a bigger, smarter model. Would love to have Nemotron as an option, even it it runs slowly.
GiteaMirror commented 2026-04-12 13:41:44 -05:00
Author
Owner
Copy Link

@Nehc commented on GitHub (Sep 6, 2024):

there is already guff, I even uploaded it to ollama, but it says that the Nemotron model is not supported...

<!-- gh-comment-id:2333859005 --> @Nehc commented on GitHub (Sep 6, 2024): there is already [guff](https://huggingface.co/mradermacher/Nemotron-4-340B-Instruct-hf-GGUF), I even uploaded it to ollama, but it says that the Nemotron model is not supported...
GiteaMirror commented 2026-04-12 13:41:44 -05:00
Author
Owner
Copy Link

@kozuch commented on GitHub (Sep 6, 2024):

Now that we have the Llama 3.1 405B model I think dealing with Nemotron-340 starts to be obsolete a bit. Surely will be interesting, though the priority for myself went down because of new Llama.

<!-- gh-comment-id:2333872909 --> @kozuch commented on GitHub (Sep 6, 2024): Now that we have the Llama 3.1 405B model I think dealing with Nemotron-340 starts to be obsolete a bit. Surely will be interesting, though the priority for myself went down because of new Llama.
GiteaMirror commented 2026-04-12 13:41:45 -05:00
Author
Owner
Copy Link

@gileneusz commented on GitHub (Sep 6, 2024):

It's still a good model and can be included for testing purposes. Additionally, it has a very favorable license for dataset generation. However, it's not added because most people don't have access to DGX platforms, so they can't really use it...

<!-- gh-comment-id:2333894379 --> @gileneusz commented on GitHub (Sep 6, 2024): It's still a good model and can be included for testing purposes. Additionally, it has a very favorable license for dataset generation. However, it's not added because most people don't have access to DGX platforms, so they can't really use it...
GiteaMirror commented 2026-04-12 13:41:45 -05:00
Author
Owner
Copy Link

@Nehc commented on GitHub (Sep 6, 2024):

Теперь, когда у нас есть модель Llama 3.1 405B, я думаю, что работа с Nemotron-340 начинает немного устаревать. Конечно, будет интересно, хотя приоритет для меня снизился из-за новой Llama.

I understand, yes. However, I had the opportunity to compare these models on some tasks, and I can say for sure that at least on some (perhaps a little specific to my tasks) Nemotron is directly noticeably cooler than the LLama 405b

<!-- gh-comment-id:2333933143 --> @Nehc commented on GitHub (Sep 6, 2024): > Теперь, когда у нас есть модель Llama 3.1 405B, я думаю, что работа с Nemotron-340 начинает немного устаревать. Конечно, будет интересно, хотя приоритет для меня снизился из-за новой Llama. I understand, yes. However, I had the opportunity to compare these models on some tasks, and I can say for sure that at least on some (perhaps a little specific to my tasks) Nemotron is directly noticeably cooler than the LLama 405b
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label model
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#3200
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?