github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 21:25:46 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #10393] Vision - IQ3_XXS - Mistral Small 3.1 24b #32589

New Issue
Closed
opened 2026-04-22 14:02:13 -05:00 by GiteaMirror · 12 comments
GiteaMirror commented 2026-04-22 14:02:13 -05:00
Owner
Copy Link

Originally created by @mirage335 on GitHub (Apr 24, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/10393

Laptop RTX 4090 has only 16GB VRAM instead of 24GB.

Recall like capabilities, using agentic description/transcription from screenshots, could be implemented for computers able to do this 'image-to-text' compression locally. Experiments with the q4_K_M quant currently available are only ~3t/s, putting the laptop under full load for more than a minute, experiments with IQ3_XXS quant shows >>20t/s, a much more acceptable power/heat/etc duty cycle of only several seconds per image processed.

Unfortunately, as many have found out, the quants found at HuggingFace, etc, generally do not include vision support, and quants from ollama that have vision support are q4 and larger.
https://www.ollama.com/library/mistral-small3.1
https://www.ollama.com/JollyLlama/Mistral-Small-3.1-24B

Think of the laptops!

Originally created by @mirage335 on GitHub (Apr 24, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/10393 Laptop RTX 4090 has only 16GB VRAM instead of 24GB. Recall like capabilities, using agentic description/transcription from screenshots, could be implemented for computers able to do this 'image-to-text' compression locally. Experiments with the q4_K_M quant currently available are only ~3t/s, putting the laptop under full load for more than a minute, experiments with IQ3_XXS quant shows >>20t/s, a much more acceptable power/heat/etc duty cycle of only several seconds per image processed. Unfortunately, as many have found out, the quants found at HuggingFace, etc, generally do not include vision support, and quants from ollama that have vision support are q4 and larger. https://www.ollama.com/library/[mistral-small3.1](https://www.ollama.com/library/mistral-small3.1) https://www.ollama.com/JollyLlama/Mistral-Small-3.1-24B Think of the laptops!
GiteaMirror added the model label 2026-04-22 14:02:13 -05:00
GiteaMirror commented 2026-04-22 14:02:18 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 24, 2025):

If you have disk space and bandwidth, you can pull the fp16 and create your own quant. Unfortunately the imatrix is not available so the IQ quants are unsupported.

$ ollama pull mistral-small3.1:24b-instruct-2503-fp16
$ echo FROM mistral-small3.1:24b-instruct-2503-fp16 > Modelfile
$ ollama create -q q3_k_s mistral-small3.1:24b-instruct-2503-q3_k_s
$ ollama list mistral-small3.1:24b-instruct-2503-q3_k_s
NAME                                         ID              SIZE     MODIFIED       
mistral-small3.1:24b-instruct-2503-q3_k_s    c49c6c7a91e8    11 GB    24 seconds ago    
$ ollama run mistral-small3.1:24b-instruct-2503-q3_k_s describe this image ./puppy.jpg 
Added image './puppy.jpg'
This image features a small, white puppy sitting on a stone surface, possibly a set of steps or a patio. The
puppy has a fluffy coat and is wearing a red collar with a small bell attached to it. The puppy appears to be
looking directly at the camera with an alert and curious expression. The background is somewhat blurred,
focusing attention on the puppy, but it seems to be an indoor or semi-indoor setting with a tiled floor visible
in the distance. The overall mood of the image is cute and endearing, highlighting the puppy's adorable features.
<!-- gh-comment-id:2828187937 --> @rick-github commented on GitHub (Apr 24, 2025): If you have disk space and bandwidth, you can pull the fp16 and create your own quant. Unfortunately the imatrix is not available so the IQ quants are unsupported. ```console $ ollama pull mistral-small3.1:24b-instruct-2503-fp16 $ echo FROM mistral-small3.1:24b-instruct-2503-fp16 > Modelfile $ ollama create -q q3_k_s mistral-small3.1:24b-instruct-2503-q3_k_s $ ollama list mistral-small3.1:24b-instruct-2503-q3_k_s NAME ID SIZE MODIFIED mistral-small3.1:24b-instruct-2503-q3_k_s c49c6c7a91e8 11 GB 24 seconds ago $ ollama run mistral-small3.1:24b-instruct-2503-q3_k_s describe this image ./puppy.jpg Added image './puppy.jpg' This image features a small, white puppy sitting on a stone surface, possibly a set of steps or a patio. The puppy has a fluffy coat and is wearing a red collar with a small bell attached to it. The puppy appears to be looking directly at the camera with an alert and curious expression. The background is somewhat blurred, focusing attention on the puppy, but it seems to be an indoor or semi-indoor setting with a tiled floor visible in the distance. The overall mood of the image is cute and endearing, highlighting the puppy's adorable features. ```
GiteaMirror commented 2026-04-22 14:02:19 -05:00
Author
Owner
Copy Link

@mirage335 commented on GitHub (Apr 24, 2025):

Just tried that. Unfortunately, the resulting model so far seems not to work with images (ie. lacks vision). Continuing to test, but could be a new bug.

On the upside, at least it seems q3_k_s is small enough to get that >>20t/s .

EDIT: Seems my previous attempt was with '2501' . Will try with '2503' ...

ollama run mistral-small:24b-instruct-2501-q3_k_s describe this image ./download.png
I'm unable to view or describe images directly. However, if you can provide more details about the image or what
specifically you would like to know about it, I might be able to help with that information!
<!-- gh-comment-id:2828620036 --> @mirage335 commented on GitHub (Apr 24, 2025): Just tried that. Unfortunately, the resulting model so far seems not to work with images (ie. lacks vision). Continuing to test, but could be a new bug. On the upside, at least it seems q3_k_s is small enough to get that >>20t/s . EDIT: Seems my previous attempt was with '2501' . Will try with '2503' ... ```terminal ollama run mistral-small:24b-instruct-2501-q3_k_s describe this image ./download.png I'm unable to view or describe images directly. However, if you can provide more details about the image or what specifically you would like to know about it, I might be able to help with that information! ```
GiteaMirror commented 2026-04-22 14:02:20 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 24, 2025):

What does the following show:

curl -s localhost:11434/api/show -d '{"model":"mistral-small:24b-instruct-2501-q3_k_s"}' | jq .capabilities
<!-- gh-comment-id:2828653650 --> @rick-github commented on GitHub (Apr 24, 2025): What does the following show: ``` curl -s localhost:11434/api/show -d '{"model":"mistral-small:24b-instruct-2501-q3_k_s"}' | jq .capabilities ```
GiteaMirror commented 2026-04-22 14:02:21 -05:00
Author
Owner
Copy Link

@mirage335 commented on GitHub (Apr 24, 2025):

Sorry, already replaced that model.

echo FROM mistral-small3.1:24b-instruct-2503-fp16 > Modelfile
ollama create -q q3_k_s mistral-small3.1:24b-instruct-2503-q3_k_s

Now it seems to almost work as expected.

NAME                                         ID              SIZE     MODIFIED
mistral-small3.1:24b-instruct-2503-q3_k_s    c49c6c7a91e8    11 GB    23 minutes ago

curl -s localhost:11434/api/show -d '{"model":"mistral-small3.1:24b-instruct-2503-q3_k_s"}' | jq .capabilities
[
  "completion",
  "vision",
  "tools"
]

ollama run mistral-small3.1:24b-instruct-2503-q3_k_s describe this image ./download.png
Added image './download.png'
The image shows a text editor window, specifically Kate,

Unfortunately, seems the model is still not small enough, or ollama insists on splitting vision models to the CPU in this situation with 16GB VRAM. Still ~3-4t/s , and the CPU/GPU load, etc, shows it.

 ollama ps
NAME                                         ID              SIZE     PROCESSOR          UNTIL
mistral-small3.1:24b-instruct-2503-q3_k_s    c49c6c7a91e8    22 GB    31%/69% CPU/GPU    4 minutes from now
<!-- gh-comment-id:2828862671 --> @mirage335 commented on GitHub (Apr 24, 2025): Sorry, already replaced that model. ```bash echo FROM mistral-small3.1:24b-instruct-2503-fp16 > Modelfile ollama create -q q3_k_s mistral-small3.1:24b-instruct-2503-q3_k_s ``` Now it seems to almost work as expected. ```terminal NAME ID SIZE MODIFIED mistral-small3.1:24b-instruct-2503-q3_k_s c49c6c7a91e8 11 GB 23 minutes ago ``` ```terminal curl -s localhost:11434/api/show -d '{"model":"mistral-small3.1:24b-instruct-2503-q3_k_s"}' | jq .capabilities [ "completion", "vision", "tools" ] ``` ```terminal ollama run mistral-small3.1:24b-instruct-2503-q3_k_s describe this image ./download.png Added image './download.png' The image shows a text editor window, specifically Kate, ``` Unfortunately, seems the model is still not small enough, or ollama insists on splitting vision models to the CPU in this situation with 16GB VRAM. Still ~3-4t/s , and the CPU/GPU load, etc, shows it. ```terminal ollama ps NAME ID SIZE PROCESSOR UNTIL mistral-small3.1:24b-instruct-2503-q3_k_s c49c6c7a91e8 22 GB 31%/69% CPU/GPU 4 minutes from now ```
GiteaMirror commented 2026-04-22 14:02:22 -05:00
Author
Owner
Copy Link

@mirage335 commented on GitHub (Apr 24, 2025):

Unfortunately restarting ollama did not help, still somewhat slow, although IIRC the previous q4_k_m model did run more 50%/50% CPU/GPU, so, minor improvement. Reducing the context window did not apparently improve speed, just truncated the prompt_tokens to an obviously insufficient 2048...

Hmm... I see 'SIZE' '22GB'. I guess vision capability loads a bunch more stuff?

EDIT: FWIW, two-pass may be more efficient, vision encoder stuff, then language model stuff, if it can fit in the VRAM.

<!-- gh-comment-id:2828881004 --> @mirage335 commented on GitHub (Apr 24, 2025): Unfortunately restarting ollama did not help, still somewhat slow, although IIRC the previous q4_k_m model did run more 50%/50% CPU/GPU, so, minor improvement. Reducing the context window did not apparently improve speed, just truncated the prompt_tokens to an obviously insufficient 2048... Hmm... I see 'SIZE' '22GB'. I guess vision capability loads a bunch more stuff? EDIT: FWIW, two-pass may be more efficient, vision encoder stuff, then language model stuff, if it can fit in the VRAM.
GiteaMirror commented 2026-04-22 14:02:23 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 24, 2025):

I guess vision capability loads a bunch more stuff?

Yes, the projector graph is nearly 9GB. However, looking at the actual memory allocation, it looks like ollama is over-estimating usage, and as a result under-utilizing VRAM. You can work around this by overriding the number of layers that ollama loads into VRAM:

$ echo FROM mistral-small3.1:24b-instruct-2503-q3_k_s > Modelfile
$ echo PARAMETER num_gpu 41 >> Modelfile
$ ollama create mistral-small3.1:24b-instruct-2503-g41-q3_k_s

Original:

$ ollama run  mistral-small3.1:24b-instruct-2503-q3_k_s hello --verbose
Hello! How can I assist you today?

total duration:       4.014324892s
load duration:        1.665869755s
prompt eval count:    359 token(s)
prompt eval duration: 1.038536345s
prompt eval rate:     345.68 tokens/s
eval count:           10 token(s)
eval duration:        1.307108658s
eval rate:            7.65 tokens/s

New:

$ ollama run mistral-small3.1:24b-instruct-2503-g41-q3_k_s hello --verbose
Hello! How can I assist you today?

total duration:       7.717788303s
load duration:        6.752422994s
prompt eval count:    359 token(s)
prompt eval duration: 544.495739ms
prompt eval rate:     659.33 tokens/s
eval count:           10 token(s)
eval duration:        419.513377ms
eval rate:            23.84 tokens/s
<!-- gh-comment-id:2828908201 --> @rick-github commented on GitHub (Apr 24, 2025): > I guess vision capability loads a bunch more stuff? Yes, the projector graph is nearly 9GB. However, looking at the actual memory allocation, it looks like ollama is over-estimating usage, and as a result under-utilizing VRAM. You can work around this by overriding the number of layers that ollama loads into VRAM: ```console $ echo FROM mistral-small3.1:24b-instruct-2503-q3_k_s > Modelfile $ echo PARAMETER num_gpu 41 >> Modelfile $ ollama create mistral-small3.1:24b-instruct-2503-g41-q3_k_s ``` Original: ``` $ ollama run mistral-small3.1:24b-instruct-2503-q3_k_s hello --verbose Hello! How can I assist you today? total duration: 4.014324892s load duration: 1.665869755s prompt eval count: 359 token(s) prompt eval duration: 1.038536345s prompt eval rate: 345.68 tokens/s eval count: 10 token(s) eval duration: 1.307108658s eval rate: 7.65 tokens/s ``` New: ``` $ ollama run mistral-small3.1:24b-instruct-2503-g41-q3_k_s hello --verbose Hello! How can I assist you today? total duration: 7.717788303s load duration: 6.752422994s prompt eval count: 359 token(s) prompt eval duration: 544.495739ms prompt eval rate: 659.33 tokens/s eval count: 10 token(s) eval duration: 419.513377ms eval rate: 23.84 tokens/s ```
GiteaMirror commented 2026-04-22 14:02:24 -05:00
Author
Owner
Copy Link

@mirage335 commented on GitHub (Apr 24, 2025):

AWESOME!!!

21.98t/s, 18s

And the description/transcription is still very much useful!

Though... it seems like it might be possible to push down the 18s runtime a little more, and improve the quality just a little (transcribing code with quoting from a screenshot sometimes gives slightly different results at this quant).

Very promising, should be very usable this way, but maybe an IQ3_XXS quant and a little more tweaking could slightly improve things even more.

Thank you!

<!-- gh-comment-id:2828956407 --> @mirage335 commented on GitHub (Apr 24, 2025): AWESOME!!! 21.98t/s, 18s And the description/transcription is still very much useful! Though... it seems like it might be possible to push down the 18s runtime a little more, and improve the quality just a little (transcribing code with quoting from a screenshot sometimes gives slightly different results at this quant). Very promising, should be very usable this way, but maybe an IQ3_XXS quant and a little more tweaking could slightly improve things even more. Thank you!
GiteaMirror commented 2026-04-22 14:02:25 -05:00
Author
Owner
Copy Link

@mirage335 commented on GitHub (Apr 24, 2025):

Seems somewhat fragile. Still seeing >95% GPU usage (which usually indicates models are not getting partially processed by CPU), but tokens/second is back down to 3-4t/s .

Seems like the margins for having enough VRAM are still a little thin. Making the model even smaller (eg. IQ3_XXS or maybe even smaller) might just help...

Apparently this happens when the laptop is disconnected from external power, though other models are not slowed down as drastically.

<!-- gh-comment-id:2829012137 --> @mirage335 commented on GitHub (Apr 24, 2025): Seems somewhat fragile. Still seeing >95% GPU usage (which usually indicates models are not getting partially processed by CPU), but tokens/second is back down to 3-4t/s . Seems like the margins for having enough VRAM are still a little thin. Making the model even smaller (eg. IQ3_XXS or maybe even smaller) might just help... Apparently this happens when the laptop is disconnected from external power, though other models are not slowed down as drastically.
GiteaMirror commented 2026-04-22 14:02:25 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 24, 2025):

Might be GPU power throttling to reduce battery drain. What's the output of nvidia-smi -q on and off external power?

<!-- gh-comment-id:2829041731 --> @rick-github commented on GitHub (Apr 24, 2025): Might be GPU power throttling to reduce battery drain. What's the output of `nvidia-smi -q` on and off external power?
GiteaMirror commented 2026-04-22 14:02:27 -05:00
Author
Owner
Copy Link

@mirage335 commented on GitHub (Apr 24, 2025):

Pardon, somewhat long...

With external power, and getting ~20t/s from the model:

nvidia-smi -q

==============NVSMI LOG==============

Timestamp                                 : Fri Apr 25 00:40:33 2025
Driver Version                            : 572.83
CUDA Version                              : 12.8

Attached GPUs                             : 1
GPU 00000000:01:00.0
    Product Name                          : NVIDIA GeForce RTX 4090 Laptop GPU
    Product Brand                         : GeForce
    Product Architecture                  : Ada Lovelace
    Display Mode                          : Disabled
    Display Active                        : Disabled
    Persistence Mode                      : N/A
    Addressing Mode                       : N/A
    MIG Mode
        Current                           : N/A
        Pending                           : N/A
    Accounting Mode                       : Disabled
    Accounting Mode Buffer Size           : 4000
    Driver Model
        Current                           : WDDM
        Pending                           : WDDM
    Serial Number                         : N/A
    GPU UUID                              : GPU-9e774f15-e659-849b-aa06-4415cca19573
    Minor Number                          : N/A
    VBIOS Version                         : 95.03.33.00.8b
    MultiGPU Board                        : No
    Board ID                              : 0x100
    Board Part Number                     : N/A
    GPU Part Number                       : 2717-750-A1
    FRU Part Number                       : N/A
    Platform Info
        Chassis Serial Number             : N/A
        Slot Number                       : N/A
        Tray Index                        : N/A
        Host ID                           : N/A
        Peer Type                         : N/A
        Module Id                         : 1
        GPU Fabric GUID                   : N/A
    Inforom Version
        Image Version                     : G002.0000.00.03
        OEM Object                        : 2.0
        ECC Object                        : N/A
        Power Management Object           : N/A
    Inforom BBX Object Flush
        Latest Timestamp                  : N/A
        Latest Duration                   : N/A
    GPU Operation Mode
        Current                           : N/A
        Pending                           : N/A
    GPU C2C Mode                          : N/A
    GPU Virtualization Mode
        Virtualization Mode               : None
        Host VGPU Mode                    : N/A
        vGPU Heterogeneous Mode           : N/A
    GPU Reset Status
        Reset Required                    : Requested functionality has been deprecated
        Drain and Reset Recommended       : Requested functionality has been deprecated
    GPU Recovery Action                   : None
    GSP Firmware Version                  : N/A
    IBMNPU
        Relaxed Ordering Mode             : N/A
    PCI
        Bus                               : 0x01
        Device                            : 0x00
        Domain                            : 0x0000
        Device Id                         : 0x271710DE
        Bus Id                            : 00000000:01:00.0
        Sub System Id                     : 0x231617AA
        GPU Link Info
            PCIe Generation
                Max                       : 4
                Current                   : 1
                Device Current            : 1
                Device Max                : 4
                Host Max                  : 5
            Link Width
                Max                       : 16x
                Current                   : 8x
        Bridge Chip
            Type                          : N/A
            Firmware                      : N/A
        Replays Since Reset               : 0
        Replay Number Rollovers           : 0
        Tx Throughput                     : 4050 KB/s
        Rx Throughput                     : 100 KB/s
        Atomic Caps Outbound              : N/A
        Atomic Caps Inbound               : N/A
    Fan Speed                             : N/A
    Performance State                     : P8
    Clocks Event Reasons
        Idle                              : Active
        Applications Clocks Setting       : Not Active
        SW Power Cap                      : Not Active
        HW Slowdown                       : Not Active
            HW Thermal Slowdown           : Not Active
            HW Power Brake Slowdown       : Not Active
        Sync Boost                        : Not Active
        SW Thermal Slowdown               : Not Active
        Display Clock Setting             : Not Active
    Sparse Operation Mode                 : N/A
    FB Memory Usage
        Total                             : 16376 MiB
        Reserved                          : 326 MiB
        Used                              : 8433 MiB
        Free                              : 7618 MiB
    BAR1 Memory Usage
        Total                             : 16384 MiB
        Used                              : 16356 MiB
        Free                              : 28 MiB
    Conf Compute Protected Memory Usage
        Total                             : N/A
        Used                              : N/A
        Free                              : N/A
    Compute Mode                          : Default
    Utilization
        GPU                               : 0 %
        Memory                            : 0 %
        Encoder                           : 0 %
        Decoder                           : 0 %
        JPEG                              : 0 %
        OFA                               : 0 %
    Encoder Stats
        Active Sessions                   : 0
        Average FPS                       : 0
        Average Latency                   : 0
    FBC Stats
        Active Sessions                   : 0
        Average FPS                       : 0
        Average Latency                   : 0
    DRAM Encryption Mode
        Current                           : N/A
        Pending                           : N/A
    ECC Mode
        Current                           : N/A
        Pending                           : N/A
    ECC Errors
        Volatile
            SRAM Correctable              : N/A
            SRAM Uncorrectable Parity     : N/A
            SRAM Uncorrectable SEC-DED    : N/A
            DRAM Correctable              : N/A
            DRAM Uncorrectable            : N/A
        Aggregate
            SRAM Correctable              : N/A
            SRAM Uncorrectable Parity     : N/A
            SRAM Uncorrectable SEC-DED    : N/A
            DRAM Correctable              : N/A
            DRAM Uncorrectable            : N/A
            SRAM Threshold Exceeded       : N/A
        Aggregate Uncorrectable SRAM Sources
            SRAM L2                       : N/A
            SRAM SM                       : N/A
            SRAM Microcontroller          : N/A
            SRAM PCIE                     : N/A
            SRAM Other                    : N/A
    Retired Pages
        Single Bit ECC                    : N/A
        Double Bit ECC                    : N/A
        Pending Page Blacklist            : N/A
    Remapped Rows
        Correctable Error                 : 0
        Uncorrectable Error               : 0
        Pending                           : No
        Remapping Failure Occurred        : No
        Bank Remap Availability Histogram
            Max                           : 128 bank(s)
            High                          : 0 bank(s)
            Partial                       : 0 bank(s)
            Low                           : 0 bank(s)
            None                          : 0 bank(s)
    Temperature
        GPU Current Temp                  : 49 C
        GPU T.Limit Temp                  : 38 C
        GPU Shutdown T.Limit Temp         : -5 C
        GPU Slowdown T.Limit Temp         : -2 C
        GPU Max Operating T.Limit Temp    : 0 C
        GPU Target Temperature            : 87 C
        Memory Current Temp               : N/A
        Memory Max Operating T.Limit Temp : N/A
    GPU Power Readings
        Average Power Draw                : N/A
        Instantaneous Power Draw          : 10.02 W
        Current Power Limit               : 102.25 W
        Requested Power Limit             : 102.25 W
        Default Power Limit               : 80.00 W
        Min Power Limit                   : 5.00 W
        Max Power Limit                   : 105.00 W
    GPU Memory Power Readings
        Average Power Draw                : N/A
        Instantaneous Power Draw          : N/A
    Module Power Readings
        Average Power Draw                : N/A
        Instantaneous Power Draw          : N/A
        Current Power Limit               : N/A
        Requested Power Limit             : N/A
        Default Power Limit               : N/A
        Min Power Limit                   : N/A
        Max Power Limit                   : N/A
    Power Smoothing                       : N/A
    Workload Power Profiles
        Requested Profiles                : N/A
        Enforced Profiles                 : N/A
    Clocks
        Graphics                          : 210 MHz
        SM                                : 210 MHz
        Memory                            : 405 MHz
        Video                             : 765 MHz
    Applications Clocks
        Graphics                          : N/A
        Memory                            : N/A
    Default Applications Clocks
        Graphics                          : N/A
        Memory                            : N/A
    Deferred Clocks
        Memory                            : N/A
    Max Clocks
        Graphics                          : 3105 MHz
        SM                                : 3105 MHz
        Memory                            : 9001 MHz
        Video                             : 2415 MHz
    Max Customer Boost Clocks
        Graphics                          : N/A
    Clock Policy
        Auto Boost                        : N/A
        Auto Boost Default                : N/A
    Voltage
        Graphics                          : N/A
    Fabric
        State                             : N/A
        Status                            : N/A
        CliqueId                          : N/A
        ClusterUUID                       : N/A
        Health
            Bandwidth                     : N/A
            Route Recovery in progress    : N/A
            Route Unhealthy               : N/A
            Access Timeout Recovery       : N/A
    Processes
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 1048
            Type                          : C+G
            Name                          :
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 3536
            Type                          : C+G
            Name                          : C:\Program Files\Microsoft OneDrive\OneDrive.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 5392
            Type                          : C
            Name                          : C:\Users\mirag\AppData\Local\Programs\Ollama\ollama.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 7576
            Type                          : C+G
            Name                          : C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.22.10731.0_x64__8wekyb3d8bbwe\WindowsTerminal.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 8452
            Type                          : C+G
            Name                          : C:\Program Files\WindowsApps\Microsoft.6365217CE6EB4_102.2503.28002.0_x64__8wekyb3d8bbwe\MicrosoftSecurityApp\MicrosoftSecurityApp.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 8792
            Type                          : C+G
            Name                          : C:\Windows\explorer.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 15300
            Type                          : C+G
            Name                          : C:\Windows\explorer.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 15404
            Type                          : C+G
            Name                          : C:\Program Files (x86)\Microsoft\EdgeWebView\Application\134.0.3124.93\msedgewebview2.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 19872
            Type                          : C+G
            Name                          : C:\Program Files\Elgato\StreamDeck\StreamDeck.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 21216
            Type                          : C+G
            Name                          : C:\Users\mirag\AppData\Local\Programs\Bitwarden\Bitwarden.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 22064
            Type                          : C+G
            Name                          : C:\Program Files\Common Files\Adobe\Adobe Desktop Common\HEX\Creative Cloud UI Helper.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 23248
            Type                          : C+G
            Name                          : C:\Program Files\Docker\Docker\frontend\Docker Desktop.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 24024
            Type                          : C+G
            Name                          : C:\Windows\SystemApps\Microsoft.Windows.AppRep.ChxApp_cw5n1h2txyewy\CHXSmartScreen.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 24600
            Type                          : C+G
            Name                          : C:\Windows\ImmersiveControlPanel\SystemSettings.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 24780
            Type                          : C+G
            Name                          : C:\Program Files\WindowsApps\AcrobatNotificationClient_1.0.4.0_x86__e1rzdqpraam7r\AcrobatNotificationClient.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 27124
            Type                          : C+G
            Name                          : C:\Windows\System32\ApplicationFrameHost.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 27664
            Type                          : C+G
            Name                          : C:\Program Files (x86)\Microsoft\EdgeWebView\Application\134.0.3124.93\msedgewebview2.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 28620
            Type                          : C+G
            Name                          : C:\Program Files\WindowsApps\MSTeams_25060.205.3499.6849_x64__8wekyb3d8bbwe\ms-teams.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 32588
            Type                          : C+G
            Name                          : C:\Windows\SystemApps\ShellExperienceHost_cw5n1h2txyewy\ShellExperienceHost.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 35120
            Type                          : C+G
            Name                          : C:\Windows\explorer.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 37400
            Type                          : C+G
            Name                          : C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 44456
            Type                          : C+G
            Name                          : C:\Windows\explorer.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 48620
            Type                          : C+G
            Name                          : C:\Program Files\WindowsApps\RealtekSemiconductorCorp.RealtekAudioControl_1.47.310.0_x64__dt26b99r8h8gj\RtkUWP.exe
            Used GPU Memory               : Not available in WDDM driver model
    Capabilities
        EGM                               : disabled

Without external power, getting ~4t/s from the model.

nvidia-smi -q

==============NVSMI LOG==============

Timestamp                                 : Fri Apr 25 00:49:04 2025
Driver Version                            : 572.83
CUDA Version                              : 12.8

Attached GPUs                             : 1
GPU 00000000:01:00.0
    Product Name                          : NVIDIA GeForce RTX 4090 Laptop GPU
    Product Brand                         : GeForce
    Product Architecture                  : Ada Lovelace
    Display Mode                          : Disabled
    Display Active                        : Disabled
    Persistence Mode                      : N/A
    Addressing Mode                       : N/A
    MIG Mode
        Current                           : N/A
        Pending                           : N/A
    Accounting Mode                       : Disabled
    Accounting Mode Buffer Size           : 4000
    Driver Model
        Current                           : WDDM
        Pending                           : WDDM
    Serial Number                         : N/A
    GPU UUID                              : GPU-9e774f15-e659-849b-aa06-4415cca19573
    Minor Number                          : N/A
    VBIOS Version                         : 95.03.33.00.8b
    MultiGPU Board                        : No
    Board ID                              : 0x100
    Board Part Number                     : N/A
    GPU Part Number                       : 2717-750-A1
    FRU Part Number                       : N/A
    Platform Info
        Chassis Serial Number             : N/A
        Slot Number                       : N/A
        Tray Index                        : N/A
        Host ID                           : N/A
        Peer Type                         : N/A
        Module Id                         : 1
        GPU Fabric GUID                   : N/A
    Inforom Version
        Image Version                     : G002.0000.00.03
        OEM Object                        : 2.0
        ECC Object                        : N/A
        Power Management Object           : N/A
    Inforom BBX Object Flush
        Latest Timestamp                  : N/A
        Latest Duration                   : N/A
    GPU Operation Mode
        Current                           : N/A
        Pending                           : N/A
    GPU C2C Mode                          : N/A
    GPU Virtualization Mode
        Virtualization Mode               : None
        Host VGPU Mode                    : N/A
        vGPU Heterogeneous Mode           : N/A
    GPU Reset Status
        Reset Required                    : Requested functionality has been deprecated
        Drain and Reset Recommended       : Requested functionality has been deprecated
    GPU Recovery Action                   : None
    GSP Firmware Version                  : N/A
    IBMNPU
        Relaxed Ordering Mode             : N/A
    PCI
        Bus                               : 0x01
        Device                            : 0x00
        Domain                            : 0x0000
        Device Id                         : 0x271710DE
        Bus Id                            : 00000000:01:00.0
        Sub System Id                     : 0x231617AA
        GPU Link Info
            PCIe Generation
                Max                       : 4
                Current                   : 1
                Device Current            : 1
                Device Max                : 4
                Host Max                  : 5
            Link Width
                Max                       : 16x
                Current                   : 8x
        Bridge Chip
            Type                          : N/A
            Firmware                      : N/A
        Replays Since Reset               : 0
        Replay Number Rollovers           : 0
        Tx Throughput                     : 2900 KB/s
        Rx Throughput                     : 100 KB/s
        Atomic Caps Outbound              : N/A
        Atomic Caps Inbound               : N/A
    Fan Speed                             : N/A
    Performance State                     : P8
    Clocks Event Reasons
        Idle                              : Active
        Applications Clocks Setting       : Not Active
        SW Power Cap                      : Not Active
        HW Slowdown                       : Not Active
            HW Thermal Slowdown           : Not Active
            HW Power Brake Slowdown       : Not Active
        Sync Boost                        : Not Active
        SW Thermal Slowdown               : Not Active
        Display Clock Setting             : Not Active
    Sparse Operation Mode                 : N/A
    FB Memory Usage
        Total                             : 16376 MiB
        Reserved                          : 326 MiB
        Used                              : 8415 MiB
        Free                              : 7636 MiB
    BAR1 Memory Usage
        Total                             : 16384 MiB
        Used                              : 16356 MiB
        Free                              : 28 MiB
    Conf Compute Protected Memory Usage
        Total                             : N/A
        Used                              : N/A
        Free                              : N/A
    Compute Mode                          : Default
    Utilization
        GPU                               : 0 %
        Memory                            : 0 %
        Encoder                           : 0 %
        Decoder                           : 0 %
        JPEG                              : 0 %
        OFA                               : 0 %
    Encoder Stats
        Active Sessions                   : 0
        Average FPS                       : 0
        Average Latency                   : 0
    FBC Stats
        Active Sessions                   : 0
        Average FPS                       : 0
        Average Latency                   : 0
    DRAM Encryption Mode
        Current                           : N/A
        Pending                           : N/A
    ECC Mode
        Current                           : N/A
        Pending                           : N/A
    ECC Errors
        Volatile
            SRAM Correctable              : N/A
            SRAM Uncorrectable Parity     : N/A
            SRAM Uncorrectable SEC-DED    : N/A
            DRAM Correctable              : N/A
            DRAM Uncorrectable            : N/A
        Aggregate
            SRAM Correctable              : N/A
            SRAM Uncorrectable Parity     : N/A
            SRAM Uncorrectable SEC-DED    : N/A
            DRAM Correctable              : N/A
            DRAM Uncorrectable            : N/A
            SRAM Threshold Exceeded       : N/A
        Aggregate Uncorrectable SRAM Sources
            SRAM L2                       : N/A
            SRAM SM                       : N/A
            SRAM Microcontroller          : N/A
            SRAM PCIE                     : N/A
            SRAM Other                    : N/A
    Retired Pages
        Single Bit ECC                    : N/A
        Double Bit ECC                    : N/A
        Pending Page Blacklist            : N/A
    Remapped Rows
        Correctable Error                 : 0
        Uncorrectable Error               : 0
        Pending                           : No
        Remapping Failure Occurred        : No
        Bank Remap Availability Histogram
            Max                           : 128 bank(s)
            High                          : 0 bank(s)
            Partial                       : 0 bank(s)
            Low                           : 0 bank(s)
            None                          : 0 bank(s)
    Temperature
        GPU Current Temp                  : 52 C
        GPU T.Limit Temp                  : 35 C
        GPU Shutdown T.Limit Temp         : -5 C
        GPU Slowdown T.Limit Temp         : -2 C
        GPU Max Operating T.Limit Temp    : 0 C
        GPU Target Temperature            : 87 C
        Memory Current Temp               : N/A
        Memory Max Operating T.Limit Temp : N/A
    GPU Power Readings
        Average Power Draw                : N/A
        Instantaneous Power Draw          : 3.99 W
        Current Power Limit               : 40.00 W
        Requested Power Limit             : 40.00 W
        Default Power Limit               : 80.00 W
        Min Power Limit                   : 5.00 W
        Max Power Limit                   : 105.00 W
    GPU Memory Power Readings
        Average Power Draw                : N/A
        Instantaneous Power Draw          : N/A
    Module Power Readings
        Average Power Draw                : N/A
        Instantaneous Power Draw          : N/A
        Current Power Limit               : N/A
        Requested Power Limit             : N/A
        Default Power Limit               : N/A
        Min Power Limit                   : N/A
        Max Power Limit                   : N/A
    Power Smoothing                       : N/A
    Workload Power Profiles
        Requested Profiles                : N/A
        Enforced Profiles                 : N/A
    Clocks
        Graphics                          : 210 MHz
        SM                                : 210 MHz
        Memory                            : 405 MHz
        Video                             : 765 MHz
    Applications Clocks
        Graphics                          : N/A
        Memory                            : N/A
    Default Applications Clocks
        Graphics                          : N/A
        Memory                            : N/A
    Deferred Clocks
        Memory                            : N/A
    Max Clocks
        Graphics                          : 3105 MHz
        SM                                : 3105 MHz
        Memory                            : 9001 MHz
        Video                             : 2415 MHz
    Max Customer Boost Clocks
        Graphics                          : N/A
    Clock Policy
        Auto Boost                        : N/A
        Auto Boost Default                : N/A
    Voltage
        Graphics                          : N/A
    Fabric
        State                             : N/A
        Status                            : N/A
        CliqueId                          : N/A
        ClusterUUID                       : N/A
        Health
            Bandwidth                     : N/A
            Route Recovery in progress    : N/A
            Route Unhealthy               : N/A
            Access Timeout Recovery       : N/A
    Processes
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 1048
            Type                          : C+G
            Name                          :
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 3536
            Type                          : C+G
            Name                          : C:\Program Files\Microsoft OneDrive\OneDrive.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 7576
            Type                          : C+G
            Name                          : C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.22.10731.0_x64__8wekyb3d8bbwe\WindowsTerminal.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 8452
            Type                          : C+G
            Name                          : C:\Program Files\WindowsApps\Microsoft.6365217CE6EB4_102.2503.28002.0_x64__8wekyb3d8bbwe\MicrosoftSecurityApp\MicrosoftSecurityApp.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 8792
            Type                          : C+G
            Name                          : C:\Windows\explorer.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 15300
            Type                          : C+G
            Name                          : C:\Windows\explorer.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 15404
            Type                          : C+G
            Name                          : C:\Program Files (x86)\Microsoft\EdgeWebView\Application\134.0.3124.93\msedgewebview2.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 19872
            Type                          : C+G
            Name                          : C:\Program Files\Elgato\StreamDeck\StreamDeck.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 21216
            Type                          : C+G
            Name                          : C:\Users\mirag\AppData\Local\Programs\Bitwarden\Bitwarden.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 22064
            Type                          : C+G
            Name                          : C:\Program Files\Common Files\Adobe\Adobe Desktop Common\HEX\Creative Cloud UI Helper.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 23248
            Type                          : C+G
            Name                          : C:\Program Files\Docker\Docker\frontend\Docker Desktop.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 24024
            Type                          : C+G
            Name                          : C:\Windows\SystemApps\Microsoft.Windows.AppRep.ChxApp_cw5n1h2txyewy\CHXSmartScreen.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 24780
            Type                          : C+G
            Name                          : C:\Program Files\WindowsApps\AcrobatNotificationClient_1.0.4.0_x86__e1rzdqpraam7r\AcrobatNotificationClient.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 27124
            Type                          : C+G
            Name                          : C:\Windows\System32\ApplicationFrameHost.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 27664
            Type                          : C+G
            Name                          : C:\Program Files (x86)\Microsoft\EdgeWebView\Application\134.0.3124.93\msedgewebview2.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 28620
            Type                          : C+G
            Name                          : C:\Program Files\WindowsApps\MSTeams_25060.205.3499.6849_x64__8wekyb3d8bbwe\ms-teams.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 32588
            Type                          : C+G
            Name                          : C:\Windows\SystemApps\ShellExperienceHost_cw5n1h2txyewy\ShellExperienceHost.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 35120
            Type                          : C+G
            Name                          : C:\Windows\explorer.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 37272
            Type                          : C
            Name                          : C:\Users\mirag\AppData\Local\Programs\Ollama\ollama.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 37400
            Type                          : C+G
            Name                          : C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 44456
            Type                          : C+G
            Name                          : C:\Windows\explorer.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 48620
            Type                          : C+G
            Name                          : C:\Program Files\WindowsApps\RealtekSemiconductorCorp.RealtekAudioControl_1.47.310.0_x64__dt26b99r8h8gj\RtkUWP.exe
            Used GPU Memory               : Not available in WDDM driver model
    Capabilities
        EGM                               : disabled

Lenovo 'Ultra Performance Mode' was supposedly active for this P1 Gen6 i9-13900H 64GB RAM RTX 4090 Laptop GPU laptop, both when external power was and was not connected.

Other AI models were not as affected, a Llama-3.1-8b q4_K_M derived model (ie. Llama-augment) does 48t/s with external power and 20t/s without external model. Though, it does seem odd these are at all affected...

<!-- gh-comment-id:2829087852 --> @mirage335 commented on GitHub (Apr 24, 2025): Pardon, somewhat long... With external power, and getting ~20t/s from the model: ```terminal nvidia-smi -q ==============NVSMI LOG============== Timestamp : Fri Apr 25 00:40:33 2025 Driver Version : 572.83 CUDA Version : 12.8 Attached GPUs : 1 GPU 00000000:01:00.0 Product Name : NVIDIA GeForce RTX 4090 Laptop GPU Product Brand : GeForce Product Architecture : Ada Lovelace Display Mode : Disabled Display Active : Disabled Persistence Mode : N/A Addressing Mode : N/A MIG Mode Current : N/A Pending : N/A Accounting Mode : Disabled Accounting Mode Buffer Size : 4000 Driver Model Current : WDDM Pending : WDDM Serial Number : N/A GPU UUID : GPU-9e774f15-e659-849b-aa06-4415cca19573 Minor Number : N/A VBIOS Version : 95.03.33.00.8b MultiGPU Board : No Board ID : 0x100 Board Part Number : N/A GPU Part Number : 2717-750-A1 FRU Part Number : N/A Platform Info Chassis Serial Number : N/A Slot Number : N/A Tray Index : N/A Host ID : N/A Peer Type : N/A Module Id : 1 GPU Fabric GUID : N/A Inforom Version Image Version : G002.0000.00.03 OEM Object : 2.0 ECC Object : N/A Power Management Object : N/A Inforom BBX Object Flush Latest Timestamp : N/A Latest Duration : N/A GPU Operation Mode Current : N/A Pending : N/A GPU C2C Mode : N/A GPU Virtualization Mode Virtualization Mode : None Host VGPU Mode : N/A vGPU Heterogeneous Mode : N/A GPU Reset Status Reset Required : Requested functionality has been deprecated Drain and Reset Recommended : Requested functionality has been deprecated GPU Recovery Action : None GSP Firmware Version : N/A IBMNPU Relaxed Ordering Mode : N/A PCI Bus : 0x01 Device : 0x00 Domain : 0x0000 Device Id : 0x271710DE Bus Id : 00000000:01:00.0 Sub System Id : 0x231617AA GPU Link Info PCIe Generation Max : 4 Current : 1 Device Current : 1 Device Max : 4 Host Max : 5 Link Width Max : 16x Current : 8x Bridge Chip Type : N/A Firmware : N/A Replays Since Reset : 0 Replay Number Rollovers : 0 Tx Throughput : 4050 KB/s Rx Throughput : 100 KB/s Atomic Caps Outbound : N/A Atomic Caps Inbound : N/A Fan Speed : N/A Performance State : P8 Clocks Event Reasons Idle : Active Applications Clocks Setting : Not Active SW Power Cap : Not Active HW Slowdown : Not Active HW Thermal Slowdown : Not Active HW Power Brake Slowdown : Not Active Sync Boost : Not Active SW Thermal Slowdown : Not Active Display Clock Setting : Not Active Sparse Operation Mode : N/A FB Memory Usage Total : 16376 MiB Reserved : 326 MiB Used : 8433 MiB Free : 7618 MiB BAR1 Memory Usage Total : 16384 MiB Used : 16356 MiB Free : 28 MiB Conf Compute Protected Memory Usage Total : N/A Used : N/A Free : N/A Compute Mode : Default Utilization GPU : 0 % Memory : 0 % Encoder : 0 % Decoder : 0 % JPEG : 0 % OFA : 0 % Encoder Stats Active Sessions : 0 Average FPS : 0 Average Latency : 0 FBC Stats Active Sessions : 0 Average FPS : 0 Average Latency : 0 DRAM Encryption Mode Current : N/A Pending : N/A ECC Mode Current : N/A Pending : N/A ECC Errors Volatile SRAM Correctable : N/A SRAM Uncorrectable Parity : N/A SRAM Uncorrectable SEC-DED : N/A DRAM Correctable : N/A DRAM Uncorrectable : N/A Aggregate SRAM Correctable : N/A SRAM Uncorrectable Parity : N/A SRAM Uncorrectable SEC-DED : N/A DRAM Correctable : N/A DRAM Uncorrectable : N/A SRAM Threshold Exceeded : N/A Aggregate Uncorrectable SRAM Sources SRAM L2 : N/A SRAM SM : N/A SRAM Microcontroller : N/A SRAM PCIE : N/A SRAM Other : N/A Retired Pages Single Bit ECC : N/A Double Bit ECC : N/A Pending Page Blacklist : N/A Remapped Rows Correctable Error : 0 Uncorrectable Error : 0 Pending : No Remapping Failure Occurred : No Bank Remap Availability Histogram Max : 128 bank(s) High : 0 bank(s) Partial : 0 bank(s) Low : 0 bank(s) None : 0 bank(s) Temperature GPU Current Temp : 49 C GPU T.Limit Temp : 38 C GPU Shutdown T.Limit Temp : -5 C GPU Slowdown T.Limit Temp : -2 C GPU Max Operating T.Limit Temp : 0 C GPU Target Temperature : 87 C Memory Current Temp : N/A Memory Max Operating T.Limit Temp : N/A GPU Power Readings Average Power Draw : N/A Instantaneous Power Draw : 10.02 W Current Power Limit : 102.25 W Requested Power Limit : 102.25 W Default Power Limit : 80.00 W Min Power Limit : 5.00 W Max Power Limit : 105.00 W GPU Memory Power Readings Average Power Draw : N/A Instantaneous Power Draw : N/A Module Power Readings Average Power Draw : N/A Instantaneous Power Draw : N/A Current Power Limit : N/A Requested Power Limit : N/A Default Power Limit : N/A Min Power Limit : N/A Max Power Limit : N/A Power Smoothing : N/A Workload Power Profiles Requested Profiles : N/A Enforced Profiles : N/A Clocks Graphics : 210 MHz SM : 210 MHz Memory : 405 MHz Video : 765 MHz Applications Clocks Graphics : N/A Memory : N/A Default Applications Clocks Graphics : N/A Memory : N/A Deferred Clocks Memory : N/A Max Clocks Graphics : 3105 MHz SM : 3105 MHz Memory : 9001 MHz Video : 2415 MHz Max Customer Boost Clocks Graphics : N/A Clock Policy Auto Boost : N/A Auto Boost Default : N/A Voltage Graphics : N/A Fabric State : N/A Status : N/A CliqueId : N/A ClusterUUID : N/A Health Bandwidth : N/A Route Recovery in progress : N/A Route Unhealthy : N/A Access Timeout Recovery : N/A Processes GPU instance ID : N/A Compute instance ID : N/A Process ID : 1048 Type : C+G Name : Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 3536 Type : C+G Name : C:\Program Files\Microsoft OneDrive\OneDrive.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 5392 Type : C Name : C:\Users\mirag\AppData\Local\Programs\Ollama\ollama.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 7576 Type : C+G Name : C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.22.10731.0_x64__8wekyb3d8bbwe\WindowsTerminal.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 8452 Type : C+G Name : C:\Program Files\WindowsApps\Microsoft.6365217CE6EB4_102.2503.28002.0_x64__8wekyb3d8bbwe\MicrosoftSecurityApp\MicrosoftSecurityApp.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 8792 Type : C+G Name : C:\Windows\explorer.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 15300 Type : C+G Name : C:\Windows\explorer.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 15404 Type : C+G Name : C:\Program Files (x86)\Microsoft\EdgeWebView\Application\134.0.3124.93\msedgewebview2.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 19872 Type : C+G Name : C:\Program Files\Elgato\StreamDeck\StreamDeck.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 21216 Type : C+G Name : C:\Users\mirag\AppData\Local\Programs\Bitwarden\Bitwarden.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 22064 Type : C+G Name : C:\Program Files\Common Files\Adobe\Adobe Desktop Common\HEX\Creative Cloud UI Helper.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 23248 Type : C+G Name : C:\Program Files\Docker\Docker\frontend\Docker Desktop.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 24024 Type : C+G Name : C:\Windows\SystemApps\Microsoft.Windows.AppRep.ChxApp_cw5n1h2txyewy\CHXSmartScreen.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 24600 Type : C+G Name : C:\Windows\ImmersiveControlPanel\SystemSettings.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 24780 Type : C+G Name : C:\Program Files\WindowsApps\AcrobatNotificationClient_1.0.4.0_x86__e1rzdqpraam7r\AcrobatNotificationClient.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 27124 Type : C+G Name : C:\Windows\System32\ApplicationFrameHost.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 27664 Type : C+G Name : C:\Program Files (x86)\Microsoft\EdgeWebView\Application\134.0.3124.93\msedgewebview2.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 28620 Type : C+G Name : C:\Program Files\WindowsApps\MSTeams_25060.205.3499.6849_x64__8wekyb3d8bbwe\ms-teams.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 32588 Type : C+G Name : C:\Windows\SystemApps\ShellExperienceHost_cw5n1h2txyewy\ShellExperienceHost.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 35120 Type : C+G Name : C:\Windows\explorer.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 37400 Type : C+G Name : C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 44456 Type : C+G Name : C:\Windows\explorer.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 48620 Type : C+G Name : C:\Program Files\WindowsApps\RealtekSemiconductorCorp.RealtekAudioControl_1.47.310.0_x64__dt26b99r8h8gj\RtkUWP.exe Used GPU Memory : Not available in WDDM driver model Capabilities EGM : disabled ``` Without external power, getting ~4t/s from the model. ```terminal nvidia-smi -q ==============NVSMI LOG============== Timestamp : Fri Apr 25 00:49:04 2025 Driver Version : 572.83 CUDA Version : 12.8 Attached GPUs : 1 GPU 00000000:01:00.0 Product Name : NVIDIA GeForce RTX 4090 Laptop GPU Product Brand : GeForce Product Architecture : Ada Lovelace Display Mode : Disabled Display Active : Disabled Persistence Mode : N/A Addressing Mode : N/A MIG Mode Current : N/A Pending : N/A Accounting Mode : Disabled Accounting Mode Buffer Size : 4000 Driver Model Current : WDDM Pending : WDDM Serial Number : N/A GPU UUID : GPU-9e774f15-e659-849b-aa06-4415cca19573 Minor Number : N/A VBIOS Version : 95.03.33.00.8b MultiGPU Board : No Board ID : 0x100 Board Part Number : N/A GPU Part Number : 2717-750-A1 FRU Part Number : N/A Platform Info Chassis Serial Number : N/A Slot Number : N/A Tray Index : N/A Host ID : N/A Peer Type : N/A Module Id : 1 GPU Fabric GUID : N/A Inforom Version Image Version : G002.0000.00.03 OEM Object : 2.0 ECC Object : N/A Power Management Object : N/A Inforom BBX Object Flush Latest Timestamp : N/A Latest Duration : N/A GPU Operation Mode Current : N/A Pending : N/A GPU C2C Mode : N/A GPU Virtualization Mode Virtualization Mode : None Host VGPU Mode : N/A vGPU Heterogeneous Mode : N/A GPU Reset Status Reset Required : Requested functionality has been deprecated Drain and Reset Recommended : Requested functionality has been deprecated GPU Recovery Action : None GSP Firmware Version : N/A IBMNPU Relaxed Ordering Mode : N/A PCI Bus : 0x01 Device : 0x00 Domain : 0x0000 Device Id : 0x271710DE Bus Id : 00000000:01:00.0 Sub System Id : 0x231617AA GPU Link Info PCIe Generation Max : 4 Current : 1 Device Current : 1 Device Max : 4 Host Max : 5 Link Width Max : 16x Current : 8x Bridge Chip Type : N/A Firmware : N/A Replays Since Reset : 0 Replay Number Rollovers : 0 Tx Throughput : 2900 KB/s Rx Throughput : 100 KB/s Atomic Caps Outbound : N/A Atomic Caps Inbound : N/A Fan Speed : N/A Performance State : P8 Clocks Event Reasons Idle : Active Applications Clocks Setting : Not Active SW Power Cap : Not Active HW Slowdown : Not Active HW Thermal Slowdown : Not Active HW Power Brake Slowdown : Not Active Sync Boost : Not Active SW Thermal Slowdown : Not Active Display Clock Setting : Not Active Sparse Operation Mode : N/A FB Memory Usage Total : 16376 MiB Reserved : 326 MiB Used : 8415 MiB Free : 7636 MiB BAR1 Memory Usage Total : 16384 MiB Used : 16356 MiB Free : 28 MiB Conf Compute Protected Memory Usage Total : N/A Used : N/A Free : N/A Compute Mode : Default Utilization GPU : 0 % Memory : 0 % Encoder : 0 % Decoder : 0 % JPEG : 0 % OFA : 0 % Encoder Stats Active Sessions : 0 Average FPS : 0 Average Latency : 0 FBC Stats Active Sessions : 0 Average FPS : 0 Average Latency : 0 DRAM Encryption Mode Current : N/A Pending : N/A ECC Mode Current : N/A Pending : N/A ECC Errors Volatile SRAM Correctable : N/A SRAM Uncorrectable Parity : N/A SRAM Uncorrectable SEC-DED : N/A DRAM Correctable : N/A DRAM Uncorrectable : N/A Aggregate SRAM Correctable : N/A SRAM Uncorrectable Parity : N/A SRAM Uncorrectable SEC-DED : N/A DRAM Correctable : N/A DRAM Uncorrectable : N/A SRAM Threshold Exceeded : N/A Aggregate Uncorrectable SRAM Sources SRAM L2 : N/A SRAM SM : N/A SRAM Microcontroller : N/A SRAM PCIE : N/A SRAM Other : N/A Retired Pages Single Bit ECC : N/A Double Bit ECC : N/A Pending Page Blacklist : N/A Remapped Rows Correctable Error : 0 Uncorrectable Error : 0 Pending : No Remapping Failure Occurred : No Bank Remap Availability Histogram Max : 128 bank(s) High : 0 bank(s) Partial : 0 bank(s) Low : 0 bank(s) None : 0 bank(s) Temperature GPU Current Temp : 52 C GPU T.Limit Temp : 35 C GPU Shutdown T.Limit Temp : -5 C GPU Slowdown T.Limit Temp : -2 C GPU Max Operating T.Limit Temp : 0 C GPU Target Temperature : 87 C Memory Current Temp : N/A Memory Max Operating T.Limit Temp : N/A GPU Power Readings Average Power Draw : N/A Instantaneous Power Draw : 3.99 W Current Power Limit : 40.00 W Requested Power Limit : 40.00 W Default Power Limit : 80.00 W Min Power Limit : 5.00 W Max Power Limit : 105.00 W GPU Memory Power Readings Average Power Draw : N/A Instantaneous Power Draw : N/A Module Power Readings Average Power Draw : N/A Instantaneous Power Draw : N/A Current Power Limit : N/A Requested Power Limit : N/A Default Power Limit : N/A Min Power Limit : N/A Max Power Limit : N/A Power Smoothing : N/A Workload Power Profiles Requested Profiles : N/A Enforced Profiles : N/A Clocks Graphics : 210 MHz SM : 210 MHz Memory : 405 MHz Video : 765 MHz Applications Clocks Graphics : N/A Memory : N/A Default Applications Clocks Graphics : N/A Memory : N/A Deferred Clocks Memory : N/A Max Clocks Graphics : 3105 MHz SM : 3105 MHz Memory : 9001 MHz Video : 2415 MHz Max Customer Boost Clocks Graphics : N/A Clock Policy Auto Boost : N/A Auto Boost Default : N/A Voltage Graphics : N/A Fabric State : N/A Status : N/A CliqueId : N/A ClusterUUID : N/A Health Bandwidth : N/A Route Recovery in progress : N/A Route Unhealthy : N/A Access Timeout Recovery : N/A Processes GPU instance ID : N/A Compute instance ID : N/A Process ID : 1048 Type : C+G Name : Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 3536 Type : C+G Name : C:\Program Files\Microsoft OneDrive\OneDrive.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 7576 Type : C+G Name : C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.22.10731.0_x64__8wekyb3d8bbwe\WindowsTerminal.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 8452 Type : C+G Name : C:\Program Files\WindowsApps\Microsoft.6365217CE6EB4_102.2503.28002.0_x64__8wekyb3d8bbwe\MicrosoftSecurityApp\MicrosoftSecurityApp.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 8792 Type : C+G Name : C:\Windows\explorer.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 15300 Type : C+G Name : C:\Windows\explorer.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 15404 Type : C+G Name : C:\Program Files (x86)\Microsoft\EdgeWebView\Application\134.0.3124.93\msedgewebview2.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 19872 Type : C+G Name : C:\Program Files\Elgato\StreamDeck\StreamDeck.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 21216 Type : C+G Name : C:\Users\mirag\AppData\Local\Programs\Bitwarden\Bitwarden.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 22064 Type : C+G Name : C:\Program Files\Common Files\Adobe\Adobe Desktop Common\HEX\Creative Cloud UI Helper.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 23248 Type : C+G Name : C:\Program Files\Docker\Docker\frontend\Docker Desktop.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 24024 Type : C+G Name : C:\Windows\SystemApps\Microsoft.Windows.AppRep.ChxApp_cw5n1h2txyewy\CHXSmartScreen.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 24780 Type : C+G Name : C:\Program Files\WindowsApps\AcrobatNotificationClient_1.0.4.0_x86__e1rzdqpraam7r\AcrobatNotificationClient.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 27124 Type : C+G Name : C:\Windows\System32\ApplicationFrameHost.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 27664 Type : C+G Name : C:\Program Files (x86)\Microsoft\EdgeWebView\Application\134.0.3124.93\msedgewebview2.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 28620 Type : C+G Name : C:\Program Files\WindowsApps\MSTeams_25060.205.3499.6849_x64__8wekyb3d8bbwe\ms-teams.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 32588 Type : C+G Name : C:\Windows\SystemApps\ShellExperienceHost_cw5n1h2txyewy\ShellExperienceHost.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 35120 Type : C+G Name : C:\Windows\explorer.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 37272 Type : C Name : C:\Users\mirag\AppData\Local\Programs\Ollama\ollama.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 37400 Type : C+G Name : C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 44456 Type : C+G Name : C:\Windows\explorer.exe Used GPU Memory : Not available in WDDM driver model GPU instance ID : N/A Compute instance ID : N/A Process ID : 48620 Type : C+G Name : C:\Program Files\WindowsApps\RealtekSemiconductorCorp.RealtekAudioControl_1.47.310.0_x64__dt26b99r8h8gj\RtkUWP.exe Used GPU Memory : Not available in WDDM driver model Capabilities EGM : disabled ``` Lenovo 'Ultra Performance Mode' was supposedly active for this P1 Gen6 i9-13900H 64GB RAM RTX 4090 Laptop GPU laptop, both when external power was and was not connected. Other AI models were not as affected, a Llama-3.1-8b q4_K_M derived model (ie. Llama-augment) does 48t/s with external power and 20t/s without external model. Though, it does seem odd these are at all affected...
GiteaMirror commented 2026-04-22 14:02:29 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 25, 2025):

--- external power
+++ internal power
@@ -179,9 +179,9 @@
         Memory Max Operating T.Limit Temp : N/A
     GPU Power Readings
         Average Power Draw                : N/A
-        Instantaneous Power Draw          : 10.02 W
-        Current Power Limit               : 102.25 W
-        Requested Power Limit             : 102.25 W
+        Instantaneous Power Draw          : 3.99 W
+        Current Power Limit               : 40.00 W
+        Requested Power Limit             : 40.00 W
         Default Power Limit               : 80.00 W
         Min Power Limit                   : 5.00 W
         Max Power Limit                   : 105.00 W

Internal power caps GPU power usage at 40W. Other systems in the laptop may also run slower. On some devices, this can be adjust with nvidia-smi -pl.

<!-- gh-comment-id:2829098789 --> @rick-github commented on GitHub (Apr 25, 2025): ```diff --- external power +++ internal power @@ -179,9 +179,9 @@ Memory Max Operating T.Limit Temp : N/A GPU Power Readings Average Power Draw : N/A - Instantaneous Power Draw : 10.02 W - Current Power Limit : 102.25 W - Requested Power Limit : 102.25 W + Instantaneous Power Draw : 3.99 W + Current Power Limit : 40.00 W + Requested Power Limit : 40.00 W Default Power Limit : 80.00 W Min Power Limit : 5.00 W Max Power Limit : 105.00 W ``` Internal power caps GPU power usage at 40W. Other systems in the laptop may also run slower. On some devices, this can be adjust with [`nvidia-smi -pl`](https://docs.nvidia.com/deploy/nvidia-smi/index.html#:~:text=%2Dpl%2C%20%2D%2Dpower%2Dlimit%3DPOWER_LIMIT).
GiteaMirror commented 2026-04-22 14:02:30 -05:00
Author
Owner
Copy Link

@mirage335 commented on GitHub (Apr 25, 2025):

Hmm... looks like there is indeed a different power limit (you beat me to posting). Indeed, hot plugging external power does seem to reach the full amount of speed up / slow down that I seem to see streaming .

Definitely ok with that, I don't think I expected to do stuff like that on battery that much anyway. Just as long as this isn't some case where ollama gets 'stuck' always running the model in slow mode, that would break scripted use cases.

Very grateful these days ollama seems to make smart decisions about not trying to run both a Llama-3.1-8b (ie. Llama-augment) and this Mistral-Small_3_1-24b-q3_k_s model simultaneously, and instead runs these models one after another if needed (given the 16GB VRAM).

<!-- gh-comment-id:2829100433 --> @mirage335 commented on GitHub (Apr 25, 2025): Hmm... looks like there is indeed a different power limit (you beat me to posting). Indeed, hot plugging external power does seem to reach the full amount of speed up / slow down that I seem to see streaming . Definitely ok with that, I don't think I expected to do stuff like that on battery that much anyway. Just as long as this isn't some case where ollama gets 'stuck' always running the model in slow mode, that would break scripted use cases. Very grateful these days ollama seems to make smart decisions about not trying to run both a Llama-3.1-8b (ie. Llama-augment) and this Mistral-Small_3_1-24b-q3_k_s model simultaneously, and instead runs these models one after another if needed (given the 16GB VRAM).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label model
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#32589
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?