github-starred/gitea
2
0
Fork 0
mirror of https://github.com/go-gitea/gitea.git synced 2026-03-16 13:13:23 -05:00
Code Issues 2.6k Packages Projects Releases 100 Wiki Activity

Minio S3 storage not working for LFS or attachments #6208

New Issue
Closed
opened 2025-11-02 06:48:23 -06:00 by GiteaMirror · 12 comments
GiteaMirror commented 2025-11-02 06:48:23 -06:00
Owner
Copy Link

Originally created by @adamb70 on GitHub (Oct 23, 2020).

  • Gitea version (or commit ref): 1.14.0+dev-69-gd2ad4dec6
  • Git version: 2.26.2
  • Operating system: Ubuntu 20

Using latest Docker image.

Description

I am trying to use S3 to host files in attachments and LFS using the minio storage type, but the files are instead being stored locally. I think this may be a problem of improper configuration, as there are no errors logged so it seems that Gitea doesn't even try to save the files to S3. The relevant parts of the ini file are below, any help is appreciated.

[server]
APP_DATA_PATH    = /data/gitea
DOMAIN           = git.mydomain.com
SSH_DOMAIN       = git.mydomain.com
HTTP_PORT        = 3000
ROOT_URL         = https://git.mydomain.com/
DISABLE_SSH      = false
SSH_PORT         = 22
SSH_LISTEN_PORT  = 22
LFS_START_SERVER = true
LFS_CONTENT_PATH = /data/git/lfs
LFS_JWT_SECRET   = {secret}
OFFLINE_MODE     = false

[lfs]
STORAGE_TYPE = minio
SERVE_DIRECT = false
MINIO_ENDPOINT = s3.eu-west-1.amazonaws.com
MINIO_ACCESS_KEY_ID = {secret}
MINIO_SECRET_ACCESS_KEY = {secret}
MINIO_BUCKET = my-gitea-bucket
MINIO_LOCATION = eu-west-1
MINIO_BASE_PATH = /lfs
MINIO_USE_SSL = true

[attachment]
STORAGE_TYPE = minio
SERVE_DIRECT = false
MINIO_ENDPOINT = s3.eu-west-1.amazonaws.com
MINIO_ACCESS_KEY_ID = {secret}
MINIO_SECRET_ACCESS_KEY = {secret}
MINIO_BUCKET = my-gitea-bucket
MINIO_LOCATION = eu-west-1
MINIO_BASE_PATH = /attachment
MINIO_USE_SSL = true
Originally created by @adamb70 on GitHub (Oct 23, 2020). - Gitea version (or commit ref): 1.14.0+dev-69-gd2ad4dec6 - Git version: 2.26.2 - Operating system: Ubuntu 20 Using latest Docker image. - Database (use `[x]`): - [x] PostgreSQL - [ ] MySQL - [ ] MSSQL - [ ] SQLite - Log gist: https://gist.github.com/adamb70/0f58c68486d826cd4810792e1dad9b66 ## Description I am trying to use S3 to host files in attachments and LFS using the minio storage type, but the files are instead being stored locally. I think this may be a problem of improper configuration, as there are no errors logged so it seems that Gitea doesn't even try to save the files to S3. The relevant parts of the ini file are below, any help is appreciated. [server] APP_DATA_PATH = /data/gitea DOMAIN = git.mydomain.com SSH_DOMAIN = git.mydomain.com HTTP_PORT = 3000 ROOT_URL = https://git.mydomain.com/ DISABLE_SSH = false SSH_PORT = 22 SSH_LISTEN_PORT = 22 LFS_START_SERVER = true LFS_CONTENT_PATH = /data/git/lfs LFS_JWT_SECRET = {secret} OFFLINE_MODE = false [lfs] STORAGE_TYPE = minio SERVE_DIRECT = false MINIO_ENDPOINT = s3.eu-west-1.amazonaws.com MINIO_ACCESS_KEY_ID = {secret} MINIO_SECRET_ACCESS_KEY = {secret} MINIO_BUCKET = my-gitea-bucket MINIO_LOCATION = eu-west-1 MINIO_BASE_PATH = /lfs MINIO_USE_SSL = true [attachment] STORAGE_TYPE = minio SERVE_DIRECT = false MINIO_ENDPOINT = s3.eu-west-1.amazonaws.com MINIO_ACCESS_KEY_ID = {secret} MINIO_SECRET_ACCESS_KEY = {secret} MINIO_BUCKET = my-gitea-bucket MINIO_LOCATION = eu-west-1 MINIO_BASE_PATH = /attachment MINIO_USE_SSL = true
GiteaMirror added the type/bug label 2025-11-02 06:48:23 -06:00
GiteaMirror commented 2025-11-02 06:48:24 -06:00
Author
Owner
Copy Link

@lunny commented on GitHub (Oct 24, 2020):

Could you find storages logs when gitea start, like below

2020/10/24 09:16:20 ...s/storage/storage.go:151:initAttachments() [I] Initialising Attachment storage with type:
2020/10/24 09:16:20 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/data/attachments
2020/10/24 09:16:20 ...s/storage/storage.go:145:initAvatars() [I] Initialising Avatar storage with type:
2020/10/24 09:16:20 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/data/avatars
2020/10/24 09:16:20 ...s/storage/storage.go:163:initRepoAvatars() [I] Initialising Repository Avatar storage with type:
2020/10/24 09:16:20 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/data/repo-avatars
2020/10/24 09:16:20 ...s/storage/storage.go:157:initLFS() [I] Initialising LFS storage with type:
2020/10/24 09:16:20 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/data/lfs
@lunny commented on GitHub (Oct 24, 2020): Could you find storages logs when gitea start, like below ```log 2020/10/24 09:16:20 ...s/storage/storage.go:151:initAttachments() [I] Initialising Attachment storage with type: 2020/10/24 09:16:20 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/data/attachments 2020/10/24 09:16:20 ...s/storage/storage.go:145:initAvatars() [I] Initialising Avatar storage with type: 2020/10/24 09:16:20 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/data/avatars 2020/10/24 09:16:20 ...s/storage/storage.go:163:initRepoAvatars() [I] Initialising Repository Avatar storage with type: 2020/10/24 09:16:20 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/data/repo-avatars 2020/10/24 09:16:20 ...s/storage/storage.go:157:initLFS() [I] Initialising LFS storage with type: 2020/10/24 09:16:20 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/data/lfs ```
GiteaMirror commented 2025-11-02 06:48:24 -06:00
Author
Owner
Copy Link

@adamb70 commented on GitHub (Oct 24, 2020):

I don't see anything about storage in the logs. Here is the startup log: https://gist.github.com/adamb70/5415b31be481db6fc325773c0c956c12

And here is the logging config in my ini file:

[log]
MODE                 = console
LEVEL                = trace
REDIRECT_MACARON_LOG = true
MACARON              = console
ROUTER               = console
ROOT_PATH            = /data/gitea/log

If it helps at all, here is my full gitea-app.ini file: https://gist.github.com/adamb70/963accf8c1e8b28f43c5a336f6262259

@adamb70 commented on GitHub (Oct 24, 2020): I don't see anything about storage in the logs. Here is the startup log: https://gist.github.com/adamb70/5415b31be481db6fc325773c0c956c12 And here is the logging config in my ini file: [log] MODE = console LEVEL = trace REDIRECT_MACARON_LOG = true MACARON = console ROUTER = console ROOT_PATH = /data/gitea/log If it helps at all, here is my full gitea-app.ini file: https://gist.github.com/adamb70/963accf8c1e8b28f43c5a336f6262259
GiteaMirror commented 2025-11-02 06:48:24 -06:00
Author
Owner
Copy Link

@lunny commented on GitHub (Oct 24, 2020):

It seems there are something lost in your log?

@lunny commented on GitHub (Oct 24, 2020): It seems there are something lost in your log?
GiteaMirror commented 2025-11-02 06:48:24 -06:00
Author
Owner
Copy Link

@adamb70 commented on GitHub (Oct 24, 2020):

I am starting Gitea up through docker-compose, so maybe some log info doesn't get sent through to the console?

I have tried again with setting log MODE=file, and the content in gitea.log is still the same as the console output so it doesn't seem like it's Docker related.

@adamb70 commented on GitHub (Oct 24, 2020): I am starting Gitea up through docker-compose, so maybe some log info doesn't get sent through to the console? I have tried again with setting log `MODE=file`, and the content in gitea.log is still the same as the console output so it doesn't seem like it's Docker related.
GiteaMirror commented 2025-11-02 06:48:25 -06:00
Author
Owner
Copy Link

@zeripath commented on GitHub (Oct 24, 2020):

@adamb70 please update to at least g25b7766673. We added logging in from that point to help us diagnose what was going on with storage.

(Assuming you're not running gitea with --port - which you shouldn't anyway - just jump to the latest)

@zeripath commented on GitHub (Oct 24, 2020): @adamb70 please update to at least g25b7766673. We added logging in from that point to help us diagnose what was going on with storage. (Assuming you're not running gitea with --port - which you shouldn't anyway - just jump to the latest)
GiteaMirror commented 2025-11-02 06:48:25 -06:00
Author
Owner
Copy Link

@adamb70 commented on GitHub (Oct 24, 2020):

Thanks for the new release @zeripath. The full log is as follows: https://gist.github.com/adamb70/5e315aa4bc8afde849e64033703e7ddb

The relevant storage part is here:

gitea      | 2020/10/24 21:37:42 ...s/storage/storage.go:151:initAttachments() [I] Initialising Attachment storage with type:
gitea      | 2020/10/24 21:37:42 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/attachments
gitea      | 2020/10/24 21:37:42 ...s/storage/storage.go:145:initAvatars() [I] Initialising Avatar storage with type:
gitea      | 2020/10/24 21:37:42 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/avatars
gitea      | 2020/10/24 21:37:42 ...s/storage/storage.go:163:initRepoAvatars() [I] Initialising Repository Avatar storage with type:
gitea      | 2020/10/24 21:37:42 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/repo-avatars
gitea      | 2020/10/24 21:37:42 ...s/storage/storage.go:157:initLFS() [I] Initialising LFS storage with type:
gitea      | 2020/10/24 21:37:42 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/git/lfs

It seems like it's still trying to use local storage, despite the app.ini file.

I am binding the ini file in the Docker container to a local file outside so it can be managed from outside the container. This seems to work fine to change any other settings in the ini file, it's just the storage I'm seeing the issue with.

docker-compose.yml

services:
  gitea:
    image: gitea/gitea:latest
    container_name: gitea
	...
    volumes:
      - gitea:/data
      - ./gitea-app.ini:/data/gitea/conf/app.ini  # <-----
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
@adamb70 commented on GitHub (Oct 24, 2020): Thanks for the new release @zeripath. The full log is as follows: https://gist.github.com/adamb70/5e315aa4bc8afde849e64033703e7ddb The relevant storage part is here: ``` gitea | 2020/10/24 21:37:42 ...s/storage/storage.go:151:initAttachments() [I] Initialising Attachment storage with type: gitea | 2020/10/24 21:37:42 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/attachments gitea | 2020/10/24 21:37:42 ...s/storage/storage.go:145:initAvatars() [I] Initialising Avatar storage with type: gitea | 2020/10/24 21:37:42 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/avatars gitea | 2020/10/24 21:37:42 ...s/storage/storage.go:163:initRepoAvatars() [I] Initialising Repository Avatar storage with type: gitea | 2020/10/24 21:37:42 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/repo-avatars gitea | 2020/10/24 21:37:42 ...s/storage/storage.go:157:initLFS() [I] Initialising LFS storage with type: gitea | 2020/10/24 21:37:42 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/git/lfs ``` It seems like it's still trying to use local storage, despite the app.ini file. I am binding the ini file in the Docker container to a local file outside so it can be managed from outside the container. This seems to work fine to change any other settings in the ini file, it's just the storage I'm seeing the issue with. ``` docker-compose.yml services: gitea: image: gitea/gitea:latest container_name: gitea ... volumes: - gitea:/data - ./gitea-app.ini:/data/gitea/conf/app.ini # <----- - /etc/timezone:/etc/timezone:ro - /etc/localtime:/etc/localtime:ro ```
GiteaMirror commented 2025-11-02 06:48:25 -06:00
Author
Owner
Copy Link

@zeripath commented on GitHub (Oct 24, 2020):

Looks like the storage type isn't being set...

@zeripath commented on GitHub (Oct 24, 2020): Looks like the storage type isn't being set...
GiteaMirror commented 2025-11-02 06:48:25 -06:00
Author
Owner
Copy Link

@zeripath commented on GitHub (Oct 24, 2020):

OK it looks like there's a bug here:

c5020cff3d/modules/setting/storage.go (L49)

I bet that this should actually default to typ instead of "".

in the meantime I think if you create a new ini section:

[storage.minio]
STORAGE_TYPE=minio

That would make the minio work

@zeripath commented on GitHub (Oct 24, 2020): OK it looks like there's a bug here: https://github.com/go-gitea/gitea/blob/c5020cff3df0a7d77ff7a453a2007edc7e5c3667/modules/setting/storage.go#L49 I bet that this should actually default to `typ` instead of `""`. in the meantime I think if you create a new ini section: ```ini [storage.minio] STORAGE_TYPE=minio ``` That would make the minio work
GiteaMirror commented 2025-11-02 06:48:26 -06:00
Author
Owner
Copy Link

@adamb70 commented on GitHub (Oct 24, 2020):

Strangely I'm now getting the following error:

gitea      | 2020/10/24 22:34:07 ...s/storage/storage.go:151:initAttachments() [I] Initialising Attachment storage with type: minio
gitea      | 2020/10/24 22:34:07 ...les/storage/minio.go:91:NewMinioStorage() [I] Creating Minio storage at : with base path
gitea      | 2020/10/24 22:34:07 routers/init.go:60:NewServices() [F] storage init failed: Endpoint:  does not follow ip address or domain name standards.

It seems like it is interpreting the endpoint URL as an empty string for some reason. Here's the line that I think is throwing the error https://github.com/minio/minio-go/blob/master/utils.go#L91 .

app.ini

[lfs]
STORAGE_TYPE = minio
MINIO_BASE_PATH = /lfs

[attachment]
STORAGE_TYPE = minio
MINIO_BASE_PATH = /attachment

[storage.minio]
STORAGE_TYPE=minio
SERVE_DIRECT = false
MINIO_ENDPOINT = s3.eu-west-1.amazonaws.com
MINIO_ACCESS_KEY_ID = {secret}
MINIO_SECRET_ACCESS_KEY = {secret}
MINIO_BUCKET = my-gitea-bucket
MINIO_LOCATION = eu-west-1
MINIO_BASE_PATH = /attachment
MINIO_USE_SSL = true
@adamb70 commented on GitHub (Oct 24, 2020): Strangely I'm now getting the following error: ``` gitea | 2020/10/24 22:34:07 ...s/storage/storage.go:151:initAttachments() [I] Initialising Attachment storage with type: minio gitea | 2020/10/24 22:34:07 ...les/storage/minio.go:91:NewMinioStorage() [I] Creating Minio storage at : with base path gitea | 2020/10/24 22:34:07 routers/init.go:60:NewServices() [F] storage init failed: Endpoint: does not follow ip address or domain name standards. ``` It seems like it is interpreting the endpoint URL as an empty string for some reason. Here's the line that I think is throwing the error https://github.com/minio/minio-go/blob/master/utils.go#L91 . ``` app.ini [lfs] STORAGE_TYPE = minio MINIO_BASE_PATH = /lfs [attachment] STORAGE_TYPE = minio MINIO_BASE_PATH = /attachment [storage.minio] STORAGE_TYPE=minio SERVE_DIRECT = false MINIO_ENDPOINT = s3.eu-west-1.amazonaws.com MINIO_ACCESS_KEY_ID = {secret} MINIO_SECRET_ACCESS_KEY = {secret} MINIO_BUCKET = my-gitea-bucket MINIO_LOCATION = eu-west-1 MINIO_BASE_PATH = /attachment MINIO_USE_SSL = true ```
GiteaMirror commented 2025-11-02 06:48:26 -06:00
Author
Owner
Copy Link

@adamb70 commented on GitHub (Oct 25, 2020):

It seems like the config in NewMinioStorage() is set to null values. I've never used golang before, but is this line correct?

25b7766673/modules/storage/minio.go (L89)

Some Googling suggests it should be config, ok := configInterface.(MinioStorageConfig).

@adamb70 commented on GitHub (Oct 25, 2020): It seems like the `config` in `NewMinioStorage()` is set to null values. I've never used golang before, but is this line correct? https://github.com/go-gitea/gitea/blob/25b7766673867d2a9ac32fde6cb0d719f04b2953/modules/storage/minio.go#L89 Some Googling suggests it should be `config, ok := configInterface.(MinioStorageConfig)`.
GiteaMirror commented 2025-11-02 06:48:26 -06:00
Author
Owner
Copy Link

@zeripath commented on GitHub (Oct 25, 2020):

nope that's not it.

The problem will be in modules/setting/storage.go

@zeripath commented on GitHub (Oct 25, 2020): nope that's not it. The problem will be in modules/setting/storage.go
GiteaMirror commented 2025-11-02 06:48:27 -06:00
Author
Owner
Copy Link

@zeripath commented on GitHub (Oct 25, 2020):

Actually it wasn't!

The issue is that the toConfig string wasn't matching the Mappable interface because the config wasn't being passed as a pointer.

There was a further bug in the MapTo.

Sorry about this - I think I have it finally fixed now.

@zeripath commented on GitHub (Oct 25, 2020): Actually it wasn't! The issue is that the toConfig string wasn't matching the Mappable interface because the config wasn't being passed as a pointer. There was a further bug in the MapTo. Sorry about this - I think I have it finally fixed now.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
Mirrored from GitHub Pull Request
 reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
No Label type/bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#6208
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?