mirror of
https://github.com/moghtech/komodo.git
synced 2026-05-07 19:35:38 -05:00
[GH-ISSUE #501] Lost four days of commits to Komodo stack repo #6118
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Dinth on GitHub (May 9, 2025).
Original GitHub issue: https://github.com/moghtech/komodo/issues/501
Hi. This is a second time this happened to me, but when it happened for the first time, i thought im just going crazy and havent had GH actions to prove otherwise.
For the last couple of weeks ive been working a lot on moving all my stacks to Komodo (and GH - im new to GH too), optimising them, making them work, etc. This morning, when i logged in to Komodo i have noticed that the recently created stacks, while running in Komodo, have GH compose file showing up as not initialised, while other stacks are failing to be redeployed, due to issues which i was sure i have fixed already. When i logged in to GH, i have noticed that the files which should be created and changes which should be commited to GH in the last few days are really not showing up in the History section of the repo, so i am indeed going crazy:
But, i also have a GH action which is being run on each file changed in each commit, and that one shows something else, im not going crazy after all:
Then i noticed GH activity history, which shows this:
That force push made 11h ago on the screenshot is the first push within 5 days which shows on the repository history (first screenshot), so its somewhere around that when the contents of the GH repository were wiped without a trace. But besides that force push being tagged as a "Force push" (not exactly sure what does it mean, but other pushes by Komodo are not tagged in this way) the details of that force push dont indicate wiping anything:
So my questions i guess are:
@Dinth commented on GitHub (May 9, 2025):
Looking at the log file in the repo cache this is it:
7898291c92e77f21b2953f9210b9703a3944cd8d is the last commit before loosing data
@beckerinj commented on GitHub (May 9, 2025):
The way that stack commit works, either initialize or update file, is by issuing a
WriteCommitComposeContentscall to periphery:7a3b2b542d/bin/periphery/src/api/compose.rs (L247)The first part of this function is call to
pull_or_clone_stack. This should ensure to contents on the server are up to date, and if it fails it will abort the rest (ie the push).In the case it is successful, the next step is to write the update, and commit the file. The git stuff starts with the
git add:7a3b2b542d/lib/git/src/commit.rs (L75)It only
addthe single file that was changed to the commit. Then it commits with message, and finally will force push:7a3b2b542d/lib/git/src/commit.rs (L120)I'm not sure what part of this process would lead to the behavior you describe. It will force push, but only after successfully pulling latest, which would include all the commits up to that point. Also, in the commit there is output for each of these steps, any answers will be in there. Can you find the Log in Komodo for the commit you suspect cause this issue?
@Dinth commented on GitHub (May 9, 2025):
Here is Komodo log from that day:
message (2).txt
and two consecutive commits to GH:
757890d606/r720-omv(contains folders like fgc, searxng, pricebuddy)and the next one:
578a0dd6d2/r720-omv(new folders are gone, commits lost in existing folders)also interesting thing is that the 757890d6065d44808aeea77eb186dfbf27d221c0 and previous commits now show "This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository."
@mbecker20 commented on GitHub (May 9, 2025):
It won't be in the core logs, it is stored in the database. We need to see the output of the commands I linked.
@Dinth commented on GitHub (May 10, 2025):
I think thats the one. But the only difference i can see between this and logs for other Write Stack Contents is that "(forced update)" bit
@jschneekloth commented on GitHub (May 19, 2025):
I just hit this as well, blew away entire git repository including history. I have daily backups so only lost a day of work, but that force push is very heavy handed if there is a misconfiguration with the state in komodo. My use case roughly went like this:
stack_folder/new_stack_nameand missed adding proper repo directory as well@rsocko commented on GitHub (May 31, 2025):
I just ran into this issue myself for the second time. Like the others it happened to me once before early in using Komodo a month ago and I assumed I must have done something wrong and simply started over.
Yesterday it happened a second time and now I lost quite a bit of config and all my stacks are disconnected from GH save for the one I just added.
Would love some help on two things:
I love the ability to have my config managed and synchronized to GitHub. But this feels very fragile with whatever I'm doing wrong. Would love to have the system warn or stop me from this outcome.
@Dinth commented on GitHub (May 31, 2025):
Another small issue im experiencing - im not sure if its related, but perhaps - is that sometimes Komodo doesnt refresh the edited files. Ie. I edit the compose file, click Save, confirm the changes, file gets saved successfully, but doesnt refresh in the UI. I need to click the refresh button manually, to get the changes i just submitted. If i wont notice that and carry on editing, i can overwrite the previous change i pushed.
Also it seems that setting a branch protection:
Prevents this from happening, without any negative side effects i noticed. So, if blocking force push doesnt break anything, is there really a reason for Komodo doing force pushes?
@Dinth commented on GitHub (May 31, 2025):
Oh actually, i just found a "side effect" of branch protection. I have created a new stack, but when i try to initialise it, I got:
@rsocko commented on GitHub (May 31, 2025):
@Dinth - I am relatively new to GH and GH Actions. Would you mind sharing any more info about what GH Actions you have configured to run on any commit so that I might replicate it as a bit of protection or recovery info?
@Dinth commented on GitHub (May 31, 2025):
Me too, my action doesnt even work and fails every time, still saved my %%% here:) Check this: https://github.com/Dinth/komodo_library/blob/main/.github/workflows/docker-compose-check.yml
The log looks like this:
and you get link to each commit, even if its gone from repo history
@jschneekloth commented on GitHub (Jun 1, 2025):
IMO, I think force push should be dropped. There shouldn't be a need for it, and if anything goes wrong, it really goes wrong. If for some reason one cannot programtically work out how to submit w/o a force push, probably best to leave it up to the user to unwind and fix things up as they have most likely done something out of band already...
@rsocko commented on GitHub (Jun 2, 2025):
I just recreated my environment and repo from scratch. I had basically nothing left in the GitHub repo, so I deleted each stack in Komodo and recreated the resource sync. Then I methodically created each stack once again and sync'd to GitHub. For now I have a clean and functioning repo and Komodo sync.
In an effort to make any such recovery much faster and to (hopefully) make it a little easier to track down which step or steps caused the force push / rebase, I added 2 GitHub actions to my repo that I thought I would share. They are somewhat redundant, but I thought it would be useful for the time being to have extra backup copies.
I used Copilot to help me create these two actions and tweak them. It was quite easy! I've stored the setup I'm using in Gists in case they help others.
Notes:
I still agree that eliminating the force push would be best, but will be using this as backup in the interim.
@mbecker20 commented on GitHub (Jun 7, 2025):
The force push is removed in 1.18.1. In cases where it would remove / reorder commits (which you wouldn't want anyways, leads to issues like above) the commit will fail, and may require resolution like pulling repo via refresh action, or maybe recloning repo before commit works.