Alerts for some persisting conditions should not be repeated #134

New Issue

GiteaMirror · 2025-10-31T15:02:57-05:00

GiteaMirror commented

2025-10-31 15:02:57 -05:00

Originally created by @gitcatpeter on GitHub (Nov 21, 2024).

Some types of alerts should be sent only once - when triggered for the first time.
Such alerts should not be repeated every several hours, even though the issue/condition still persists.

Good example would be StackImageUpdateAvailable triggered when there is a newer image available. If the stack is not updated, the same alert will be repeated ad nauseam.

We could discuss, whether other conditions, like high CPU usage should be repeated or not. But some alerts like said StackImageUpdateAvailable should be fired only once.

Thoughts?

Originally created by @gitcatpeter on GitHub (Nov 21, 2024). Some types of alerts should be sent only once - when triggered for the first time. Such alerts should not be repeated every several hours, even though the issue/condition still persists. Good example would be **StackImageUpdateAvailable** triggered when there is a newer image available. If the stack is not updated, the same alert will be repeated ad nauseam. We could discuss, whether other conditions, like high CPU usage should be repeated or not. But some alerts like said **StackImageUpdateAvailable** should be fired only once. Thoughts?

GiteaMirror added the enhancement label 2025-10-31 15:02:57 -05:00

GiteaMirror commented

2025-10-31 15:02:57 -05:00

@mbecker20 commented on GitHub (Nov 21, 2024):

This is indeed the intention, and there is logic to limit the alert to only be sent once. For StackImageUpdateAvailable, the logic is currently stored in application memory, leading to the alert being sent out again if Komodo Core is restarted before the image is updated. I believe this is the cause of the alert being sent out again.

Nonetheless the implementation for update available alert logic can definitely be refined some more. They currently have no "lifetime" like the system status alerts for example, or the Resource Sync pending updates alert.

I didn't see it before but the alert for StackImageUpdateAvailable should work exactly like the Resource Sync pending updates alert, and show unresolved status in the UI alert dropdown. It should also use the same logic Resource Sync pending updates uses to avoid redundant alert even between restarts. This should solve your issue with StackImageUpdateAvailable multiple alerts.

@mbecker20 commented on GitHub (Nov 21, 2024): This is indeed the intention, and there is logic to limit the alert to only be sent once. For StackImageUpdateAvailable, the logic is currently stored in application memory, leading to the alert being sent out again if Komodo Core is restarted before the image is updated. I believe this is the cause of the alert being sent out again. Nonetheless the implementation for update available alert logic can definitely be refined some more. They currently have no "lifetime" like the system status alerts for example, or the Resource Sync pending updates alert. I didn't see it before but the alert for StackImageUpdateAvailable should work exactly like the Resource Sync pending updates alert, and show unresolved status in the UI alert dropdown. It should also use the same logic Resource Sync pending updates uses to avoid redundant alert even between restarts. This should solve your issue with StackImageUpdateAvailable multiple alerts.

GiteaMirror commented

2025-10-31 15:02:57 -05:00

@gitcatpeter commented on GitHub (Nov 21, 2024):

Komodo core and periphery uptime is >2days, but I received two or more notifications for a few stacks.
For instance I have loki01 stack containing amongst others grafana/grafana:latest
It is possible they had published today two images with such tag, but is that the case here (same alert id...)?

morning alert:
11/20/2024, 6:57:48 AM

{
  "id": "67396ced7af2e094dcaf33c4",
  "name": "loki01",
  "server_id": "673936487af2e094dcaf24f9",
  "server_name": "docker01",
  "service": "grafana",
  "image": "grafana/grafana:latest"
}

afternoon alert:
11/20/2024, 6:07:28 PM

{
  "id": "67396ced7af2e094dcaf33c4",
  "name": "loki01",
  "server_id": "673936487af2e094dcaf24f9",
  "server_name": "docker01",
  "service": "grafana",
  "image": "grafana/grafana:latest"
}

@gitcatpeter commented on GitHub (Nov 21, 2024): Komodo core and periphery uptime is >2days, but I received two or more notifications for a few stacks. For instance I have loki01 stack containing amongst others **grafana/grafana:latest** It is possible they had published today two images with such tag, but is that the case here (same alert id...)? 1. morning alert: 11/20/2024, 6:57:48 AM ``` { "id": "67396ced7af2e094dcaf33c4", "name": "loki01", "server_id": "673936487af2e094dcaf24f9", "server_name": "docker01", "service": "grafana", "image": "grafana/grafana:latest" } ``` 2. afternoon alert: 11/20/2024, 6:07:28 PM ``` { "id": "67396ced7af2e094dcaf33c4", "name": "loki01", "server_id": "673936487af2e094dcaf24f9", "server_name": "docker01", "service": "grafana", "image": "grafana/grafana:latest" } ```

GiteaMirror commented

2025-10-31 15:02:58 -05:00

@mbecker20 commented on GitHub (Nov 21, 2024):

There may be a bug in the current implementation, its not as robust as the logic of ResourceSyncPendingUpdate alert, but its basically working or else you would see a new alert spammed every 5 mins.

In any case the solution is to refactor the logic for StackImageUpdateAvailable to be in line with ResourceSyncPendingUpdate, and use the existence of an "open" alert on DB to avoid sending redundant alert.

@mbecker20 commented on GitHub (Nov 21, 2024): There may be a bug in the current implementation, its not as robust as the logic of ResourceSyncPendingUpdate alert, but its basically working or else you would see a new alert spammed every 5 mins. In any case the solution is to refactor the logic for StackImageUpdateAvailable to be in line with ResourceSyncPendingUpdate, and use the existence of an "open" alert on DB to avoid sending redundant alert.

GiteaMirror referenced this issue

2025-10-31 15:18:39 -05:00

[PR #134] [MERGED] feat: Add docs link to topbar #691

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/komodo#134