Extend gitea to add custom lfs upload/download adaptors #4358

New Issue

GiteaMirror · 2025-11-02T05:48:02-06:00

GiteaMirror commented

2025-11-02 05:48:02 -06:00

Originally created by @praveentiru on GitHub (Nov 20, 2019).

Gitea version (or commit ref): 9619ccf
Git version: v1.10.0
Operating system: Windows
Database (use [x]):
- PostgreSQL
- MySQL
- MSSQL
- SQLite
Can you reproduce the bug at https://try.gitea.io:
- Yes (provide example URL)
- No
- Not relevant
Log gist:

Description

This is an enhancement request that I can work on once everyone agrees on spec. This request is inspired from Bitbucket's LFS Media Adapter. Adapter chunks large files and uploads them in parallel.

Proposal:

Add a config which would take a list as input.
Each item in list will hold an object with two key-value pairs
1. adapter=mymagicsauce
2. url=https://www.xxx.com/lfssauce
Object link and verify link for upload will be built using url key value as base
Gitea will expose an url for adapter service hit to verify upload of files
Person extending with custom adapter will also provide client adapter component.

Looking forward to feedback from the team.

Originally created by @praveentiru on GitHub (Nov 20, 2019). - Gitea version (or commit ref): 9619ccf - Git version: v1.10.0 - Operating system: Windows - Database (use `[x]`): - [ ] PostgreSQL - [x] MySQL - [ ] MSSQL - [ ] SQLite - Can you reproduce the bug at https://try.gitea.io: - [ ] Yes (provide example URL) - [ ] No - [x] Not relevant - Log gist: ## Description This is an enhancement request that I can work on once everyone agrees on spec. This request is inspired from [Bitbucket's LFS Media Adapter](https://confluence.atlassian.com/bitbucket/bitbucket-lfs-media-adapter-856699998.html). Adapter chunks large files and uploads them in parallel. Proposal: 1. Add a config which would take a list as input. 2. Each item in list will hold an object with two key-value pairs 1. adapter=mymagicsauce 2. url=https://www.xxx.com/lfssauce 3. Object link and verify link for upload will be built using url key value as base 4. Gitea will expose an url for adapter service hit to verify upload of files 5. Person extending with custom adapter will also provide client adapter component. Looking forward to feedback from the team.

GiteaMirror added the type/proposal label 2025-11-02 05:48:02 -06:00

GiteaMirror closed this issue

2025-11-02 05:48:02 -06:00

GiteaMirror commented

2025-11-02 05:48:03 -06:00

@zeripath commented on GitHub (Nov 21, 2019):

This sounds interesting. How does it compare to tus.io (#1723)?

Gitea recently got some LFS repository management settings pages - would this mean those stopped working?

@zeripath commented on GitHub (Nov 21, 2019): This sounds interesting. How does it compare to tus.io (#1723)? Gitea recently got some LFS repository management settings pages - would this mean those stopped working?

GiteaMirror commented

2025-11-02 05:48:03 -06:00

@praveentiru commented on GitHub (Nov 21, 2019):

@zeripath tus.io is only one approach. But, there can be lot of different approaches possible. Few options:

Upload to a cloud bucket in multiple parallel connections
Manage LFS objects in a local storage while Gitea is hosted on cloud (Sensitive binary data)
Chunk binary files and upload only the chuncks needed and re-assemble them after upload

So, an extensible approach will allow people to build solutions with different approach. People who use Gitea can select which options to use and this code need not sit in gitea code base.

@praveentiru commented on GitHub (Nov 21, 2019): @zeripath tus.io is only one approach. But, there can be lot of different approaches possible. Few options: - Upload to a cloud bucket in multiple parallel connections - Manage LFS objects in a local storage while Gitea is hosted on cloud (Sensitive binary data) - Chunk binary files and upload only the chuncks needed and re-assemble them after upload So, an extensible approach will allow people to build solutions with different approach. People who use Gitea can select which options to use and this code need not sit in gitea code base.

GiteaMirror commented

2025-11-02 05:48:03 -06:00

@zeripath commented on GitHub (Nov 21, 2019):

Ah, so you're suggesting that instead of offering the Gitea LFS server we can set up the repo to proffer a different LFS server?

@zeripath commented on GitHub (Nov 21, 2019): Ah, so you're suggesting that instead of offering the Gitea LFS server we can set up the repo to proffer a different LFS server?

GiteaMirror commented

2025-11-02 05:48:03 -06:00

@praveentiru commented on GitHub (Nov 21, 2019):

I am proposing Gitea LFS server would still be primary controller. It will manage everything with exception of Upload/Download of LFS files. That piece will be managed by a separate server-client combination.

@praveentiru commented on GitHub (Nov 21, 2019): I am proposing Gitea LFS server would still be primary controller. It will manage everything with **exception of Upload/Download of LFS files**. That piece will be managed by a separate server-client combination.

GiteaMirror commented

2025-11-02 05:48:04 -06:00

@praveentiru commented on GitHub (Nov 25, 2019):

Here is execution flow that I see:

LFS server receives Batch Upload or, Download request
LFS server authenticates the request
If not authenticated, exit
LFS server reads from settings list of transfer adapters supported with preference
LFS server identifies most preferred adapter that is supported by current server
LFS server sends the Batch request to upload/download adapter server
Adapter server responds with fully formed response
LFS forwards the response to client
Client transfer client and adapter manage upload/download
On successful completion, adapter server calls on verify link of Gitea server
Transfer cycle closes

Complications with this approach:

Adapter control where LFS files are stored. When there is no parity adapters supported in clients using a repo, LFS files would end up spread around.
1. Restricting exposure to single adapter is brutish way to force that all clients have necessary client side adapter in place.

Looking for feedback on approach before working on the same.

@praveentiru commented on GitHub (Nov 25, 2019): Here is execution flow that I see: 1. LFS server receives Batch Upload or, Download request 2. LFS server authenticates the request 3. If not authenticated, exit 4. LFS server reads from settings list of transfer adapters supported with preference 5. LFS server identifies most preferred adapter that is supported by current server 6. LFS server sends the Batch request to upload/download adapter server 7. Adapter server responds with fully formed response 8. LFS forwards the response to client 9. Client transfer client and adapter manage upload/download 10. On successful completion, adapter server calls on verify link of Gitea server 11. Transfer cycle closes Complications with this approach: 1. Adapter control where LFS files are stored. When there is no parity adapters supported in clients using a repo, LFS files would end up spread around. 1. Restricting exposure to single adapter is brutish way to force that all clients have necessary client side adapter in place. Looking for feedback on approach before working on the same.

GiteaMirror commented

2025-11-02 05:48:04 -06:00

@lunny commented on GitHub (Sep 3, 2020):

I think this will be easier after #12518 merged.

@lunny commented on GitHub (Sep 3, 2020): I think this will be easier after #12518 merged.

GiteaMirror commented

2025-11-02 05:48:04 -06:00

@lunny commented on GitHub (Oct 14, 2020):

Let's close this since we have the modules/storage package and the interfaces and just send PRs to implement new adaptors.

@lunny commented on GitHub (Oct 14, 2020): Let's close this since we have the `modules/storage` package and the interfaces and just send PRs to implement new adaptors.

GiteaMirror referenced this issue

2025-11-02 12:33:03 -06:00

[PR #4358] [MERGED] Fix docker build #17343

GiteaMirror referenced this issue

2025-11-02 14:02:05 -06:00

[PR #4932] [CLOSED] fix bug forget to remove Stopwatch when remove repository #17555

Sign in to join this conversation.