Some S3 compatible services does not have chunked upload properly implemented #9826

New Issue

2025-11-02T08:50:37-06:00

GiteaMirror commented

2025-11-02 08:50:37 -06:00

Originally created by @stevefan1999-personal on GitHub (Nov 11, 2022).

Oops seems like the template is totally disregarded, so I will just put in some ad-hoc info here.

Description

The title speaks itself. This is prominent in the case of OpenStack Swift, prior to 2.11.0, does not implement chunked upload properly and just let the data passthrough, causing a corruption in the object store side.

This issue can be remediated by testing for the compatibility of this specific feature by uploading a big chunk of zeroed data (whatever it is) and retrieving it and see if it is still all zeroes. Also remember it should be tested per defined storage type because not everyone might use the same S3 endpoint. This is just a one-off test from startup so I wouldn't say the API call would be very high.

Well..although the more obvious answer is that QNAP should move on from an ancient version of OpenShift Swift (they are using 1.11.0)

Environment

Gitea Version: 1.18.0-rc0

From your first screenshot it looks like QuObjects saved the chunked upload into the file without removing the chunk parts. Maybe you can configure something to allow chunked uploads? Otherwise I don't know if we can help you.

edit: Looks like you found the problem.

If you can compile Gitea you may try this (minio.go):
func (m *MinioStorage) Save(path string, r io.Reader, size int64) (int64, error) {
	uploadInfo, err := m.client.PutObject(
		m.ctx,
		m.bucket,
		m.buildMinioPath(path),
		r,
		size,
- 		minio.PutObjectOptions{ContentType: "application/octet-stream"},
+ 		minio.PutObjectOptions{ContentType: "application/octet-stream", DisableContentSha256: true},
	)
	if err != nil {
		return 0, convertMinioErr(err)
	}
	return uploadInfo.Size, nil
}
That may disable the signature for uploads.

https://github.com/minio/minio-go/blob/36e5116001d9e05dc27e8ecc790d261bd1632906/api.go#L841

Originally posted by @KN4CK3R in https://github.com/go-gitea/gitea/issues/21320#issuecomment-1311781013

Originally created by @stevefan1999-personal on GitHub (Nov 11, 2022). Oops seems like the template is totally disregarded, so I will just put in some ad-hoc info here. ## Description The title speaks itself. This is prominent in the case of OpenStack Swift, prior to 2.11.0, does not implement chunked upload properly and just let the data passthrough, causing a corruption in the object store side. This issue can be remediated by testing for the compatibility of this specific feature by uploading a big chunk of zeroed data (whatever it is) and retrieving it and see if it is still all zeroes. Also remember it should be tested per defined storage type because not everyone might use the same S3 endpoint. This is just a one-off test from startup so I wouldn't say the API call would be very high. Well..although the more obvious answer is that QNAP should move on from an ancient version of OpenShift Swift (they are using 1.11.0) ## Environment - Gitea Version: 1.18.0-rc0 --- > From your first screenshot it looks like QuObjects saved the chunked upload into the file without removing the chunk parts. Maybe you can configure something to allow chunked uploads? Otherwise I don't know if we can help you. > > edit: Looks like you found the problem. > > If you can compile Gitea you may try this (`minio.go`): > ```diff > func (m *MinioStorage) Save(path string, r io.Reader, size int64) (int64, error) { > uploadInfo, err := m.client.PutObject( > m.ctx, > m.bucket, > m.buildMinioPath(path), > r, > size, > - minio.PutObjectOptions{ContentType: "application/octet-stream"}, > + minio.PutObjectOptions{ContentType: "application/octet-stream", DisableContentSha256: true}, > ) > if err != nil { > return 0, convertMinioErr(err) > } > return uploadInfo.Size, nil > } > ``` > That may disable the signature for uploads. > > https://github.com/minio/minio-go/blob/36e5116001d9e05dc27e8ecc790d261bd1632906/api.go#L841 _Originally posted by @KN4CK3R in https://github.com/go-gitea/gitea/issues/21320#issuecomment-1311781013_

GiteaMirror added the type/enhancement label 2025-11-02 08:50:37 -06:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/gitea#9826