Repo mirroring doesn&#39;t mirror LFS data. #324

@lunny commented on GitHub (Feb 15, 2017):

This is the same reason with #732. Let's close this and discuss there.

@lunny commented on GitHub (Feb 15, 2017): This is the same reason with #732. Let's close this and discuss there.

GiteaMirror commented

@fabian-z commented on GitHub (Feb 24, 2017):

I'm afraid I have to disagree: #732 is not related. Gitea cannot currently retrieve LFS objects from another server. This would basically require adding a complete LFS client on top of the current implementation.

@damomurf: Please see GitHub documentation regarding repository mirroring with LFS

@fabian-z commented on GitHub (Feb 24, 2017): I'm afraid I have to disagree: #732 is not related. Gitea cannot currently retrieve LFS objects from another server. This would basically require adding a complete LFS client on top of the current implementation. @damomurf: Please see GitHub documentation regarding [repository mirroring with LFS](https://help.github.com/articles/duplicating-a-repository/#mirroring-a-repository-that-contains-git-large-file-storage-objects)

GiteaMirror commented

@lunny commented on GitHub (Feb 24, 2017):

Seems you are right.

@lunny commented on GitHub (Feb 24, 2017): Seems you are right.

GiteaMirror commented

@fabian-z commented on GitHub (Feb 24, 2017):

I'd suggest to tag this as a feature request. Current behaviour is basically expected imho and supporting LFS mirrors will be challenging in some regards:

Need LFS client implementation
- CLI client not developed for interactive use
- Reusing source would require rewriting to some extent
Using CLI client
- Would use at least double the disk space for the LFS objects during mirror
- Continuous mirror pulling would likely need double the disk space all the time
The operation will need to scan all files in every commit for LFS meta files, potentially taking very long for large repositories

Just out of interest, does somebody know whether GitHub support automated LFS object mirroring?

For future reference:
GitLab issue, git-lfs issue

@fabian-z commented on GitHub (Feb 24, 2017): I'd suggest to tag this as a feature request. Current behaviour is basically expected imho and supporting LFS mirrors will be challenging in some regards: * Need LFS client implementation * CLI client not developed for interactive use * Reusing source would require rewriting to some extent * Using CLI client * Would use at least double the disk space for the LFS objects during mirror * Continuous mirror pulling would likely need double the disk space all the time * The operation will need to scan all files in every commit for LFS meta files, potentially taking very long for large repositories Just out of interest, does somebody know whether GitHub support automated LFS object mirroring? For future reference: [GitLab issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/1252), [git-lfs issue](https://github.com/git-lfs/git-lfs/issues/1762)

GiteaMirror commented

@lunny commented on GitHub (Feb 24, 2017):

Maybe put a notice on migrating UI at first.

@lunny commented on GitHub (Feb 24, 2017): Maybe put a notice on migrating UI at first.

GiteaMirror commented

@fabian-z commented on GitHub (Feb 24, 2017):

I'll look into adding a note to the UI when LFS support is enabled.

@fabian-z commented on GitHub (Feb 24, 2017): I'll look into adding a note to the UI when LFS support is enabled.

GiteaMirror commented

@boskiv commented on GitHub (Mar 27, 2018):

What is steps for mirroring LFS now? Is there some hook to make LFS pull with cmd ?
Or may be I need to setup some cron task ?

@boskiv commented on GitHub (Mar 27, 2018): What is steps for mirroring LFS now? Is there some hook to make LFS pull with cmd ? Or may be I need to setup some cron task ?

GiteaMirror commented

@stale[bot] commented on GitHub (Feb 4, 2019):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

@stale[bot] commented on GitHub (Feb 4, 2019): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

GiteaMirror commented

@fabian-z commented on GitHub (Feb 27, 2019):

@boskiv
You will currently have to manually mirror the repository if you wish to transfer the LFS objects, too.
It may or may not be (easily) possible to setup a custom mirror using cronjobs and shell scripts, but you would need to plan for double the disk space usage (permanently) and possibly use a separate local user account.

Please also see the GitHub docs for manual steps.

@fabian-z commented on GitHub (Feb 27, 2019): @boskiv You will currently have to manually mirror the repository if you wish to transfer the LFS objects, too. It may or may not be (easily) possible to setup a custom mirror using cronjobs and shell scripts, but you would need to plan for double the disk space usage (permanently) and possibly use a separate local user account. Please also see the [GitHub docs](https://help.github.com/articles/duplicating-a-repository/#mirroring-a-repository-that-contains-git-large-file-storage-objects) for manual steps.

GiteaMirror commented

@mloskot commented on GitHub (Apr 26, 2019):

BTW, the GitHub docs is what currently Git LFS recommends as per https://github.com/git-lfs/git-lfs/issues/2342#issuecomment-310323647

@fabian-z Forgive me if I'm hijacking the issue, but could you explain your last comment in more details, in the context explained below?

I have a GitHub repo with files in Git LFS.
I'm looking for solution to have Gitea serve mirror that repo (just Git repo).

TL;TR: Is it possible?

Would something along these lines work?

Mirror GitHub repo with LFS

git clone --bare <GITHUB-URL> <GITHUB-DIR>
git lfs fetch origin --all
git push --mirror <GITEA-URL>
git lfs push --all <GITEA-URL>

Attach Gitea to serve the files pushed to <GITEA-URL>

Frequently run

cd <GITHUB-DIR>
git fetch origin --all
git lfs fetch origin --all
git push --all <GITEA-URL>
git lfs push --all <GITEA-URL>

@mloskot commented on GitHub (Apr 26, 2019): BTW, the GitHub docs is what currently Git LFS recommends as per https://github.com/git-lfs/git-lfs/issues/2342#issuecomment-310323647 ------ @fabian-z Forgive me if I'm hijacking the issue, but could you explain your last comment in more details, in the context explained below? I have a GitHub repo with files in Git LFS. I'm looking for solution to have Gitea serve mirror that repo (just Git repo). TL;TR: Is it possible? Would something along these lines work? 1. Mirror GitHub repo with LFS ``` git clone --bare <GITHUB-URL> <GITHUB-DIR> git lfs fetch origin --all git push --mirror <GITEA-URL> git lfs push --all <GITEA-URL> ``` 2. Attach Gitea to serve the files pushed to `<GITEA-URL>` 3. Frequently run ``` cd <GITHUB-DIR> git fetch origin --all git lfs fetch origin --all git push --all <GITEA-URL> git lfs push --all <GITEA-URL> ```

GiteaMirror commented

@GlassedSilver commented on GitHub (Feb 29, 2020):

Is there anyone kinda sorta working on it? Or is there anyone who figured out a way how to do this manually/scripted externally for the time being?

I'm using Gitea to keep backups of software I use or just find interesting, just in case... Obviously having the LFS objects as well is often very handy, many times necessary.

@GlassedSilver commented on GitHub (Feb 29, 2020): Is there anyone kinda sorta working on it? Or is there anyone who figured out a way how to do this manually/scripted externally for the time being? I'm using Gitea to keep backups of software I use or just find interesting, just in case... Obviously having the LFS objects as well is often very handy, many times necessary.

GiteaMirror commented

@zeripath commented on GitHub (Feb 29, 2020):

No one is working on this. A bounty on bountysource might spur someone to look at it.

From a quick glance we would need to implement an LFS client - possibly through vendoring git-lfs/git-lfs. There would need to be significant changes to the mirroring UI too.

@zeripath commented on GitHub (Feb 29, 2020): No one is working on this. A bounty on bountysource might spur someone to look at it. From a quick glance we would need to implement an LFS client - possibly through vendoring git-lfs/git-lfs. There would need to be significant changes to the mirroring UI too.

GiteaMirror commented

@GlassedSilver commented on GitHub (May 28, 2020):

Chipped in 70 bucks into the bounty: https://www.bountysource.com/issues/41700547-repo-mirroring-doesn-t-mirror-lfs-data

@GlassedSilver commented on GitHub (May 28, 2020): Chipped in 70 bucks into the bounty: https://www.bountysource.com/issues/41700547-repo-mirroring-doesn-t-mirror-lfs-data

GiteaMirror commented

@GlassedSilver commented on GitHub (Jun 2, 2020):

Is it rude for me to ask for a maintainer of the repo to add the bounty tag and maybe declare the bounty value in the title?

Cheers and overall many many thanks for this lovely project to everyone who is working on it. It's of great use to me. :)

@GlassedSilver commented on GitHub (Jun 2, 2020): Is it rude for me to ask for a maintainer of the repo to add the bounty tag and maybe declare the bounty value in the title? Cheers and overall many many thanks for this lovely project to everyone who is working on it. It's of great use to me. :)

GiteaMirror commented

@techknowlogick commented on GitHub (Jun 2, 2020):

@GlassedSilver thanks for your support :) I've just applied the bounty label to make this easier to find.

@techknowlogick commented on GitHub (Jun 2, 2020): @GlassedSilver thanks for your support :) I've just applied the bounty label to make this easier to find.

GiteaMirror commented