Ability to automatically push to remote mirror repository #1512

@mxmehl commented on GitHub (Feb 12, 2018):

Good idea, I have been looking for such a option already.

IMHO this would increase adoption of Gitea. People would be more inclined to using Gitea if they knew that they could easily mirror stuff to GitLab/Github without having to fiddle with hooks.

@mxmehl commented on GitHub (Feb 12, 2018): Good idea, I have been looking for such a option already. IMHO this would increase adoption of Gitea. People would be more inclined to using Gitea if they knew that they could easily mirror stuff to GitLab/Github without having to fiddle with hooks.

GiteaMirror commented

@ghost commented on GitHub (Feb 22, 2018):

I'm also pretty interested in this, and wouldn't mind having a go at implementing it if someone could point me in the right direction!

@ghost commented on GitHub (Feb 22, 2018): I'm also pretty interested in this, and wouldn't mind having a go at implementing it if someone could point me in the right direction!

GiteaMirror commented

@thehowl commented on GitHub (Mar 3, 2018):

Right now I am doing this on a project of mine. Since I am the admin of the instance, I have access to the git hooks of the repos, and I can add this as the post-receive hook:

#!/bin/bash
git push --mirror --quiet https://user:token@github.com/osuripple/hanayo.git &> /dev/null
echo 'osuripple/hanayo updated'

Of course, this is just a hack, and a proper mechanism would be gladly appreciated.

@thehowl commented on GitHub (Mar 3, 2018): Right now I am doing this on a project of mine. Since I am the admin of the instance, I have access to the git hooks of the repos, and I can add this as the post-receive hook: ```bash #!/bin/bash git push --mirror --quiet https://user:token@github.com/osuripple/hanayo.git &> /dev/null echo 'osuripple/hanayo updated' ``` Of course, this is just a hack, and a proper mechanism would be gladly appreciated.

GiteaMirror commented

@ulm0 commented on GitHub (Aug 6, 2018):

this would be a really nice feature to have in Gitea, please!

@ulm0 commented on GitHub (Aug 6, 2018): this would be a really nice feature to have in Gitea, please!

GiteaMirror commented

@ghost commented on GitHub (Aug 12, 2018):

Yeah this would be an amazingly useful feature!

@ghost commented on GitHub (Aug 12, 2018): Yeah this would be an amazingly useful feature!

GiteaMirror commented

@ghost commented on GitHub (Aug 13, 2018):

Is this really just a limitation of other systems disguised to look like a feature in Gitea? If Webhooks can do it why run the risk and adding complexity trying to make Gitea manage features absent from other products?

Now, if we could have an SVN push mirror that would be extremely useful use case for the ad hoc nature of the umptysquillion poorly managed WordPress SVN repos in existence today who would gladly being porting over go Gitea to simplify their workflow and reduce premature aging.

@ghost commented on GitHub (Aug 13, 2018): Is this really just a limitation of other systems disguised to look like a feature in Gitea? If Webhooks can do it why run the risk and adding complexity trying to make Gitea manage features absent from other products? Now, if we could have an SVN push mirror that would be extremely useful use case for the ad hoc nature of the umptysquillion poorly managed WordPress SVN repos in existence today who would gladly being porting over go Gitea to simplify their workflow and reduce premature aging.

GiteaMirror commented

@snth commented on GitHub (Nov 9, 2018):

@thehowl Thanks. I made a slight modification so I don't have to wait for the push to the mirror to complete (which can be slow).

#!/bin/bash
git push --mirror --quiet https://user:token@github.com/osuripple/hanayo.git &> /dev/null &
echo 'osuripple/hanayo update initiated.'

Of course now you no longer have a confirmation that the push succeeded but a main motivation for my use case of gitea was to avoid the long lags of updates to slow remote hosts.

@snth commented on GitHub (Nov 9, 2018): @thehowl Thanks. I made a slight modification so I don't have to wait for the push to the mirror to complete (which can be slow). ``` #!/bin/bash git push --mirror --quiet https://user:token@github.com/osuripple/hanayo.git &> /dev/null & echo 'osuripple/hanayo update initiated.' ``` Of course now you no longer have a confirmation that the push succeeded but a main motivation for my use case of gitea was to avoid the long lags of updates to slow remote hosts.

GiteaMirror commented

@pklapperich commented on GitHub (Mar 15, 2019):

The git hook trick won't work for mirror repos. And a lot of services don't respond to inbound webhooks, for example gitea itself doesn't, a simple ssh server acting as a git remote, etc. Having this built in would be useful.

@pklapperich commented on GitHub (Mar 15, 2019): The git hook trick won't work for mirror repos. And a lot of services don't respond to inbound webhooks, for example gitea itself doesn't, a simple ssh server acting as a git remote, etc. Having this built in would be useful.

GiteaMirror commented

@lunny commented on GitHub (Mar 16, 2019):

You have to store the token or private key on gitea if we implement that.

@lunny commented on GitHub (Mar 16, 2019): You have to store the token or private key on gitea if we implement that.

GiteaMirror commented

@chotaire commented on GitHub (Apr 7, 2019):

This is a very interesting feature!

@chotaire commented on GitHub (Apr 7, 2019): This is a very interesting feature!

GiteaMirror commented

@belliash commented on GitHub (May 3, 2019):

+1

@belliash commented on GitHub (May 3, 2019): +1

GiteaMirror commented

@HarvsG commented on GitHub (Jul 18, 2019):

This would be a very useful feature. It would be great to have a system where users could develop a project on our hosted Gitea instance and then 'publish' it to a public git service by setting it up as a mirrored repo.

@HarvsG commented on GitHub (Jul 18, 2019): This would be a very useful feature. It would be great to have a system where users could develop a project on our hosted Gitea instance and then 'publish' it to a public git service by setting it up as a mirrored repo.

GiteaMirror commented

@millergarym commented on GitHub (Sep 22, 2019):

You have to store the token or private key on gitea if we implement that.

No, ssh auth forwarding could be used.

@millergarym commented on GitHub (Sep 22, 2019): > You have to store the token or private key on gitea if we implement that. No, ssh auth forwarding could be used.

GiteaMirror commented

@petrac-daniel commented on GitHub (Oct 1, 2019):

After I used the hack from above, I've written an post receive script to update the remote repo with all changes from gitea. All changes made in gitea will be automatically pushed to the "mirror repo" but not the other way around. Updating the gitea repo is still manually.

It is unfortunately still necessary to update the config file of the repo and the git authentication on the maschine running gitea:

[remote "origin"]
	url = <remote_url_here>
	fetch = +refs/heads/*:refs/remotes/origin/*


#!/usr/bin/env bash

z40=0000000000000000000000000000000000000000

while read local_ref local_sha remote_ref remote_sha
do
    if [[ $remote_ref =~ refs\/heads\/(.+) ]]; then
        branch_name=${BASH_REMATCH[1]}
        
        if [ "$local_sha" = $z40 ]
        then
            # Handle delete
            echo HANDLE DELETE BRANCH ON <remote_url_here>
            echo ----------
            git push --delete origin $branch_name
            echo ----------
        else
            echo HANDLE PUSH BRANCH ON ON <remote_url_here>
            echo ----------
            git push --set-upstream origin $branch_name
            echo ----------
        fi
    else
        echo "unable to parse remote branch $branch_name"
    fi
done

@petrac-daniel commented on GitHub (Oct 1, 2019): After I used the hack from above, I've written an post receive script to update the remote repo with all changes from gitea. All changes made in gitea will be automatically pushed to the "mirror repo" but not the other way around. Updating the gitea repo is still manually. It is unfortunately still necessary to update the config file of the repo and the git authentication on the maschine running gitea: ``` [remote "origin"] url = <remote_url_here> fetch = +refs/heads/*:refs/remotes/origin/* ``` ``` #!/usr/bin/env bash z40=0000000000000000000000000000000000000000 while read local_ref local_sha remote_ref remote_sha do if [[ $remote_ref =~ refs\/heads\/(.+) ]]; then branch_name=${BASH_REMATCH[1]} if [ "$local_sha" = $z40 ] then # Handle delete echo HANDLE DELETE BRANCH ON <remote_url_here> echo ---------- git push --delete origin $branch_name echo ---------- else echo HANDLE PUSH BRANCH ON ON <remote_url_here> echo ---------- git push --set-upstream origin $branch_name echo ---------- fi else echo "unable to parse remote branch $branch_name" fi done ```

GiteaMirror commented

@PeterHindes commented on GitHub (Nov 21, 2019):

Any progress on adding this to the gitea source? It seems like all we have are workarounds, Fairly clever ones at that, but some actual progress would be a good sign.

Gitlab has this feature built in already Might be some useable code there.

@PeterHindes commented on GitHub (Nov 21, 2019): Any progress on adding this to the gitea source? It seems like all we have are workarounds, Fairly clever ones at that, but some actual progress would be a good sign. [Gitlab has this feature built in already](https://docs.gitlab.com/ee/user/project/repository/repository_mirroring.html) Might be some useable code there.

GiteaMirror commented

@WattsC-90 commented on GitHub (Mar 26, 2020):

I am interested in this feature primarily because our current infrastructure uses mirrored servers globally for developers to use to improve speeds, there is a node next to (virtually) the build agent/cluster to speed up build checkouts etc. too. This includes failover too, if the primary node goes down, then we can turn one of the mirrors into primary. (I must note, we are using SVN, so the use of master/slave nodes may be redundant, but I want to pitch in with why I was looking and my use case to help, if possible, improve the end result.) I am loving the feel of this project and shall be looking to pull our SVN system over to GIT if corporate decision makers allow... just needs the right feature set!

Another use case we have discussed previously is to commit to one server for code reviews, and then once the code review is done, push it to the master. Whenever you checkout, you get it from the master. Similar to the diagram here (only the last two stages with "push to gerrit" and "submit". Mirroring functionality would allow such a feature to be offered.

[referenced from: https://www.bsi-software.com/fileadmin/daten/processed/5/7/csm_Gerrit_workflow_33e6332529.png]

@WattsC-90 commented on GitHub (Mar 26, 2020): I am interested in this feature primarily because our current infrastructure uses mirrored servers globally for developers to use to improve speeds, there is a node next to (virtually) the build agent/cluster to speed up build checkouts etc. too. This includes failover too, if the primary node goes down, then we can turn one of the mirrors into primary. (I must note, we are using SVN, so the use of master/slave nodes may be redundant, but I want to pitch in with why I was looking and my use case to help, if possible, improve the end result.) I am loving the feel of this project and shall be looking to pull our SVN system over to GIT if corporate decision makers allow... just needs the right feature set! Another use case we have discussed previously is to commit to one server for code reviews, and then once the code review is done, push it to the master. Whenever you checkout, you get it from the master. Similar to the diagram here (only the last two stages with `"push to gerrit"` and `"submit"`. Mirroring functionality would allow such a feature to be offered. ![image](https://user-images.githubusercontent.com/10712792/77649909-865de500-6f62-11ea-9148-0e85e743e7ab.png) [referenced from: https://www.bsi-software.com/fileadmin/daten/_processed_/5/7/csm_Gerrit_workflow_33e6332529.png]

GiteaMirror commented

@lafriks commented on GitHub (Mar 28, 2020):

It's not ideal but drone ci could be used to push changes further to other git server

@lafriks commented on GitHub (Mar 28, 2020): It's not ideal but drone ci could be used to push changes further to other git server

GiteaMirror commented

@WattsC-90 commented on GitHub (Mar 29, 2020):

@lafriks I have TeamCity already in place and working well for the business, so dont really want to change CI product, just keen to have a commit gating process like the one shown.

@WattsC-90 commented on GitHub (Mar 29, 2020): @lafriks I have TeamCity already in place and working well for the business, so dont really want to change CI product, just keen to have a commit gating process like the one shown.

GiteaMirror commented

@StrangeWill commented on GitHub (Mar 29, 2020):

@WattsC-90 I mean it could be done in TeamCity too, it's an interesting suggestion and I'm considering it -- we have an internal CI server that we build everything on, then sometimes our customers have their own repos/servers/etc that right now we manually push to, would be nice to stop doing that..

I could just make part of TeamCity's build process to push out to the external repo on successful build (a good-ish idea), and even if TeamCity was on the other "side" of the push, I could just make a build that does nothing but sync on change.

Not the best workaround, but even with Gitlab we're worried that repo mirroring and security won't be the best, so it is kind of a "if nothing else works, make CI do it".

@StrangeWill commented on GitHub (Mar 29, 2020): @WattsC-90 I mean it could be done in TeamCity too, it's an interesting suggestion and I'm considering it -- we have an internal CI server that we build _everything_ on, then sometimes our customers have their own repos/servers/etc that right now we _manually_ push to, would be nice to stop doing that.. I could just make part of TeamCity's build process to push out to the external repo on successful build (a good-ish idea), and even if TeamCity was on the other "side" of the push, I could just make a build that does nothing but sync on change. Not the best workaround, but even with Gitlab we're worried that repo mirroring and security won't be the best, so it is kind of a "if nothing else works, make CI do it".

GiteaMirror commented

@WattsC-90 commented on GitHub (Apr 3, 2020):

@StrangeWill I found this in my downtime, might be an interesting reference to how it could be done... https://github.com/takezoe/gitmesh just wonder if by having a topology similar to that shown in the github repo linked, the logic could be put in the "controller" level (see link), to push to "golden" repo (or potentially any other task..)

@WattsC-90 commented on GitHub (Apr 3, 2020): @StrangeWill I found this in my downtime, might be an interesting reference to how it *could* be done... https://github.com/takezoe/gitmesh just wonder if by having a topology similar to that shown in the github repo linked, the logic could be put in the "controller" level (see link), to push to "golden" repo (or potentially any other task..)

GiteaMirror commented

@poperigby commented on GitHub (May 1, 2020):

I wonder if it would also be possible to mirror the issue tracker with a bot.

@poperigby commented on GitHub (May 1, 2020): I wonder if it would also be possible to mirror the issue tracker with a bot.

GiteaMirror commented

@keks24 commented on GitHub (Aug 5, 2020):

@thehowl Thanks. I made a slight modification so I don't have to wait for the push to the mirror to complete (which can be slow).
#!/bin/bash
git push --mirror --quiet https://user:token@github.com/osuripple/hanayo.git &> /dev/null &
echo 'osuripple/hanayo update initiated.'
Of course now you no longer have a confirmation that the push succeeded but a main motivation for my use case of gitea was to avoid the long lags of updates to slow remote hosts.

I took your adaptation a bit further:
https://github.com/keks24/git-hooks/blob/master/home/gitea/.local/share/gitea/hooks/post-receive.d/post-receive

Be aware, that git push --force will fail on the mirrors, if branches are protected.

@keks24 commented on GitHub (Aug 5, 2020): > @thehowl Thanks. I made a slight modification so I don't have to wait for the push to the mirror to complete (which can be slow). > > ``` > #!/bin/bash > git push --mirror --quiet https://user:token@github.com/osuripple/hanayo.git &> /dev/null & > echo 'osuripple/hanayo update initiated.' > ``` > > Of course now you no longer have a confirmation that the push succeeded but a main motivation for my use case of gitea was to avoid the long lags of updates to slow remote hosts. I took your adaptation a bit further: https://github.com/keks24/git-hooks/blob/master/home/gitea/.local/share/gitea/hooks/post-receive.d/post-receive Be aware, that `git push --force` will fail on the mirrors, if branches are protected.

GiteaMirror commented

@lunny commented on GitHub (Aug 5, 2020):

If we want gitea to do that, it depends on #12065

@lunny commented on GitHub (Aug 5, 2020): If we want gitea to do that, it depends on #12065

GiteaMirror commented

@neumantm commented on GitHub (Sep 5, 2020):

For anyone wanting to update a github mirror with a deploy key rather than access token:

#!/bin/bash

# Setup ssh config
grep -qxF 'Include config.d/*' ~/.ssh/config || echo 'Include config.d/*' >> ~/.ssh/config
mkdir --parents ~/.ssh/config.d/
cat << EOF > ~/.ssh/config.d/github-mirror-setup
Host github.com
CheckHostIP no
EOF
# Need to add github to known hosts, because cannot press yes on first connect.
grep -qF 'github.com' ~/.ssh/known_hosts || echo 'github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==' >> ~/.ssh/known_hosts

# Setup ssh-agent
eval `ssh-agent`
cat << EOF | ssh-add -
-----BEGIN OPENSSH PRIVATE KEY-----
<your private key here>
-----END OPENSSH PRIVATE KEY-----
EOF

ssh -T git@github.com
git push --mirror git@github.com:<username>/<reponame>.git
echo 'github mirror updated.'

@neumantm commented on GitHub (Sep 5, 2020): For anyone wanting to update a github mirror with a deploy key rather than access token: ``` #!/bin/bash # Setup ssh config grep -qxF 'Include config.d/*' ~/.ssh/config || echo 'Include config.d/*' >> ~/.ssh/config mkdir --parents ~/.ssh/config.d/ cat << EOF > ~/.ssh/config.d/github-mirror-setup Host github.com CheckHostIP no EOF # Need to add github to known hosts, because cannot press yes on first connect. grep -qF 'github.com' ~/.ssh/known_hosts || echo 'github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==' >> ~/.ssh/known_hosts # Setup ssh-agent eval `ssh-agent` cat << EOF | ssh-add - -----BEGIN OPENSSH PRIVATE KEY----- <your private key here> -----END OPENSSH PRIVATE KEY----- EOF ssh -T git@github.com git push --mirror git@github.com:<username>/<reponame>.git echo 'github mirror updated.' ```

GiteaMirror commented

@rexzhang commented on GitHub (Sep 23, 2020):

This is a very nice feature!

@rexzhang commented on GitHub (Sep 23, 2020): This is a very nice feature!

GiteaMirror commented

@kdumontnu commented on GitHub (Feb 9, 2021):

Bounty added:

If you'd like to promote this, please add

[![Bountysource](https://api.bountysource.com/badge/issue?issue_id=54827645)](https://www.bountysource.com/issues/54827645-ability-to-automatically-push-to-remote-mirror-repository)

To the Issue summary

Edit: @techknowlogick can this get the bounty badge?

@kdumontnu commented on GitHub (Feb 9, 2021): Bounty added: [![Bountysource](https://api.bountysource.com/badge/issue?issue_id=54827645)](https://www.bountysource.com/issues/54827645-ability-to-automatically-push-to-remote-mirror-repository) If you'd like to promote this, please add ``` [![Bountysource](https://api.bountysource.com/badge/issue?issue_id=54827645)](https://www.bountysource.com/issues/54827645-ability-to-automatically-push-to-remote-mirror-repository) ``` To the Issue summary Edit: @techknowlogick can this get the `bounty` badge?

GiteaMirror commented

@KN4CK3R commented on GitHub (Mar 25, 2021):

I've started some days ago working on this.

@KN4CK3R commented on GitHub (Mar 25, 2021): I've started some days ago working on this.

GiteaMirror commented