mirror of
https://github.com/moghtech/komodo.git
synced 2026-05-08 21:28:16 -05:00
[GH-ISSUE #125] Inaccurate RAM usage detection with ZFS #9420
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/komodo#9420
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Chloe-ko on GitHub (Oct 14, 2024).
Original GitHub issue: https://github.com/moghtech/komodo/issues/125
Hello,
first off I want to start this off with saying I absolutely love Komodo, it's by far the best Docker(-Compose) Deployment tool I've ever used! Thank you for this software!
Now to the issue, Komodo is persistently and non-stop giving me this Alert:
I knew this was wrong and that I am not using this much memory, so I went to investigate.
To my surprise
free -hactually shows the same usage!Which at first had me worried, but then following this up by checking with htop and btop, they suddenly reported different (and more within reason) usages:
Following this I went to investigate and eventually found the reason for this discrepancy: ZFS, the file system
ZFS is very memory-intensive as it reaches higher capacities (and I have a total of 82TB of storage using ZFS) as it uses what's called ZFS ARC. This is a cache that is kept in-memory.
Just like normal Linux, ZFS ARC separates it's RAM usage into "Used" and "Available" - Used RAM being Memory that it actually needs, while Available Memory is Memory that is being used but ZFS can release to the user at any given time if needed.
(I'll assume you already know this topic, but if not, here's a small tl;dr: https://www.linuxatemyram.com/)
I confirmed this using btop which actually has a setting for this:
While this is set to true, btop shows the "correct" RAM Usage of 64GB Used, 61GB Available.
Setting this to false makes btop reflect what Komodo and
free-hshow: 119GB used, ~6GB available.I double checked and with my other servers, where ZFS is not in use, Komodo correctly only considers the used RAM used, so this is confirmed to be a ZFS specific issue.
Generally speaking, Available RAM should be considered "free" RAM from a user perspective.
It'd be great if this behavior could be adjusted as otherwise I have an alert active 24/7, thank you :)
@mbecker20 commented on GitHub (Oct 15, 2024):
Thanks for the detailed report!
Currently the
used_memorycomes from this method: https://docs.rs/sysinfo/0.32.0/sysinfo/struct.System.html#method.used_memoryIt seems this just take
total - free(not available) memory. I agree that from a user perspective, available memory is usually the important metric, not free memory. So in 1.15.9 theused_memorywill come fromtotal_memory - available_memory. This should resolve the open alert issue if available mem reports more than free mem.Since some user may care about the
free_memoryas well, I've also added another fieldmem_free_gb. This gives all the data available:mem_free_gbmem_total_gb - mem_used_gbmem_used_gb(ormem_total_gb - mem_free_gb)mem_total_gb@Chloe-ko commented on GitHub (Oct 15, 2024):
Hey, thanks for the quick action!
Sadly I have to report that this didn't help in my specific case.
As you can see here, it still thinks that my Server is on 90%+ RAM usage:
And just for sanity checking, at the same time, btop reports:
This is a very ZFS-specific issue, though, and I assume not an issue for most users.
I did some more research and this issue seems to be with the fact that most filesystems just use the linux built-in file caching, which the kernel can properly differentiate between available and used.
ZFS implements its own file caching in memory (the thing called ARC), but this results in the kernel seeing all the memory that ZFS uses as "used" and not as available, regardless of whether ZFS can free up the memory or not.
The only way I've found to work around this seems to be having the application itself manually check the ZFS ARC memory and subtract it - there doesn't seem to be a way to get this info just from built-in commands that only check the kernel, as the kernel doesn't know about ZFS-internals.
I've found this which gives some valuable hints: https://superuser.com/a/1137417
I've tried checking the kernel stats from that answer and can confirm that
cat /proc/spl/kstat/zfs/arcstatsdoes output info about the ARC state even from inside the komodo/periphery container, so this is a resource that could be used whether in a bare-metal install or a containerized deployment.The most valuable part of the output seems to be the fields c, c_min, c_max and size.
The superuser answer already elaborates on most of these:
cis the target size of the ARC in bytesc_maxis the maximum size of the ARC in bytessizeis the current size of the ARC in bytesHowever, the answer doesn't explain
c_min, though as the name implies, that's the minimum size that the ARC cache will shrink to. c_min is, essentially, the "used" memory for ZFS, as this seems to be the amount of memory that ZFS will not free up for the user."available" memory for ZFS could be calculated as
size-c_min. (the amount of memory it won't free up substracted from the total amount of memory used = the amount of memory it will free up, as per my understanding)Using awk the values can be extracted in gigabytes like this (The output from arcstats is initially in bytes, the
/ 1073741824converts it to Gigabytes, of course if the initial calculation should be in bytes before the conversion to GB that can be removed):Current size of the entire ZFS ARC (zfs_size_gb):
awk '/^size/ { print $3 / 1073741824 }' < /proc/spl/kstat/zfs/arcstatsMinimum size of ARC (zfs_min_c_gb):
awk '/^c_min/ { print $3 / 1073741824 }' < /proc/spl/kstat/zfs/arcstatsGiven that, the available memory for a machine could be calculated as:
mem_total_gb - (mem_used_gb - zfs_size_gb + zfs_min_c_gb)or
mem_total_gb - mem_used_gb + zfs_size_gb - zfs_min_c_gbOf course, it would be reasonable to first check if zfs is in use (by checking if
/proc/spl/kstat/zfs/arcstatsexists at all), but I think this would make sense to implement given that ZFS is a regularly used filesystem.@NeurekaSoftware commented on GitHub (Jun 19, 2025):
Yeah, this is still an issue for me as well!
I am looking forward to this being fixed so Komodo stops complaining about memory usage.
@chiefy commented on GitHub (Feb 25, 2026):
I just installed Komodo and having this issue. I disabled the alert for the particular server but the alert persists, is that a separate issue?
@Enstrayed commented on GitHub (Mar 8, 2026):
Can replicate this issue as well. This is kinda frustrating as there is no way to suppress individual warning types.
Beszel does properly differentiate ARC using the aforementioned
arcstatsfile, parsing it using this function.ZFS does also cause weirdness with disk usage calculation, since it measures used/free for each mountpoint, it ends up inflating the total amount of disk space a system has.
@Crosis47 commented on GitHub (Mar 26, 2026):
I opened an issue for this giving an idea for a fix that hopefully can be easily implemented into Komodo.
#1273
@NeurekaSoftware commented on GitHub (Mar 26, 2026):
You should have added that info here, not created a duplicate issue.
@Crosis47 commented on GitHub (Mar 26, 2026):
I didn't know this issue existed at the time. I wasn't concerned with the RAM reporting issue so I wasn't looking for it. I didn't see anything about disk mount reporting issues so I created my issue. Shortly after that I searched on ZFS and found this issue. It's also technically not the same issue and was just mentioned in Enstrayed's comment.
@Enstrayed commented on GitHub (Mar 26, 2026):
This issue concerns how Komodo reads memory usage; #1273 is about how it reads disk usage. They both have to do with ZFS but are separate issues, combining them wouldn't make any sense. I mentioned it in my comment since #1273 didn't exist when I made it but that sentence is off-topic to this issue.
@NeurekaSoftware commented on GitHub (Mar 26, 2026):
Oh, my mistake! :)