ObjectStorage without chunk-encoding support corrupts files (the action_runner log cannot be viewed) #14152

@wxiaoguang commented on GitHub (Feb 19, 2025):

It seems to be a plain-text file? The foot bytes (from log) are 4b73e\r\n\r\n

@wxiaoguang commented on GitHub (Feb 19, 2025): It seems to be a plain-text file? The foot bytes (from log) are `4b73e\r\n\r\n`

GiteaMirror commented

@OAMchronicle commented on GitHub (Feb 19, 2025):

// file gitea/modules/actions/log_test.go
This is the test case I wrote

package actions

import (
	"code.gitea.io/gitea/modules/zstd"
	"fmt"
	"io"
	"os"
	"testing"
)

func TestTransferLogs(t *testing.T) {
	name := "1131.log"
	f, err := os.Open(name)
	if err != nil {
		return
	}
	defer f.Close()
	var reader io.Reader = f

	r, w := io.Pipe()
	reader = r
	zstdWriter, err := zstd.NewSeekableWriter(w, logZstdBlockSize)
	if err != nil {
		fmt.Errorf("zstd NewSeekableWriter: %w", err)
	}
	go func() {
		defer func() {
			_ = w.CloseWithError(zstdWriter.Close())
		}()
		if _, err := io.Copy(zstdWriter, f); err != nil {
			_ = w.CloseWithError(err)
			return
		}
	}()
	file, err := os.OpenFile("1131.log.zst", os.O_CREATE|os.O_RDWR, 0666)
	if err != nil {
		return
	}
	defer file.Close()
	_, err = io.Copy(file, reader)
	if err != nil {
		return
	}
	/*
		minioStorage, err := storage.NewMinioStorage(context.Background(),
			&setting.Storage{
				MinioConfig: setting.MinioStorageConfig{
					Endpoint:           "x.x.x.x:8333",
					AccessKeyID:        "xxx",
					SecretAccessKey:    "xxx",
					Bucket:             "gitea",
					UseSSL:             false,
					InsecureSkipVerify: true,
				},
			})
		if err != nil {
			return
		}
		_, err = minioStorage.Save("logs/1131.log.zst", reader, -1)
		if err != nil {
			return
		}
	*/
}

There is no problem when using local files. This issue will be triggered when uploading to S3

@OAMchronicle commented on GitHub (Feb 19, 2025): // file gitea/modules/actions/log_test.go This is the test case I wrote <details> ```go package actions import ( "code.gitea.io/gitea/modules/zstd" "fmt" "io" "os" "testing" ) func TestTransferLogs(t *testing.T) { name := "1131.log" f, err := os.Open(name) if err != nil { return } defer f.Close() var reader io.Reader = f r, w := io.Pipe() reader = r zstdWriter, err := zstd.NewSeekableWriter(w, logZstdBlockSize) if err != nil { fmt.Errorf("zstd NewSeekableWriter: %w", err) } go func() { defer func() { _ = w.CloseWithError(zstdWriter.Close()) }() if _, err := io.Copy(zstdWriter, f); err != nil { _ = w.CloseWithError(err) return } }() file, err := os.OpenFile("1131.log.zst", os.O_CREATE|os.O_RDWR, 0666) if err != nil { return } defer file.Close() _, err = io.Copy(file, reader) if err != nil { return } /* minioStorage, err := storage.NewMinioStorage(context.Background(), &setting.Storage{ MinioConfig: setting.MinioStorageConfig{ Endpoint: "x.x.x.x:8333", AccessKeyID: "xxx", SecretAccessKey: "xxx", Bucket: "gitea", UseSSL: false, InsecureSkipVerify: true, }, }) if err != nil { return } _, err = minioStorage.Save("logs/1131.log.zst", reader, -1) if err != nil { return } */ } ``` </details> There is no problem when using local files. This issue will be triggered when uploading to S3

GiteaMirror commented

@OAMchronicle commented on GitHub (Feb 19, 2025):

Does it happen for any logs of action runs, or only this log?

It seems that the file is broken for some reasons.

Currently, I know it's a log

@OAMchronicle commented on GitHub (Feb 19, 2025): > Does it happen for any logs of action runs, or only this log? > > It seems that the file is broken for some reasons. Currently, I know it's a log

GiteaMirror commented

@wxiaoguang commented on GitHub (Feb 19, 2025):

Will the S3 storage auto decompress the compressed zstd files and respond plain-text?

@wxiaoguang commented on GitHub (Feb 19, 2025): Will the S3 storage **auto** decompress the compressed zstd files and respond plain-text?

GiteaMirror commented

@OAMchronicle commented on GitHub (Feb 19, 2025):

I don't know, I don't think it will happen. The S3 I am using is seaweed

@OAMchronicle commented on GitHub (Feb 19, 2025): I don't know, I don't think it will happen. The S3 I am using is seaweed

GiteaMirror commented

@wxiaoguang commented on GitHub (Feb 19, 2025):

It seems to be a plain-text file? The first bytes (from log) are 4b73e\r\n\r\n

According to your log, the response is plain-text.

Could you help to confirm the seaweed's behavior? For example: use your test code , and read the logs/1131.log.zst raw content and dump it to see whether it is compressed or decompressed? (or: maybe the response is jut incorrect? just a guess)

@wxiaoguang commented on GitHub (Feb 19, 2025): > It seems to be a plain-text file? The first bytes (from log) are `4b73e\r\n\r\n` According to your log, the response is plain-text. Could you help to confirm the seaweed's behavior? For example: use your test code , and read the `logs/1131.log.zst` raw content and dump it to see whether it is compressed or decompressed? (or: maybe the response is jut incorrect? just a guess)

GiteaMirror commented

@OAMchronicle commented on GitHub (Feb 19, 2025):

1131.log.obs.zst is directly uploaded to S3 based on the test code。
1131. log.zst is manually uploaded to S3 based on the test code saved locally

@OAMchronicle commented on GitHub (Feb 19, 2025): <img width="650" alt="Image" src="https://github.com/user-attachments/assets/0aa915d8-172d-4045-b2ef-63783e0a3828" /> <img width="1204" alt="Image" src="https://github.com/user-attachments/assets/a580b668-b2dc-4b63-b56f-b2bca63b2365" /> 1131.log.obs.zst is directly uploaded to S3 based on the test code。 1131. log.zst is manually uploaded to S3 based on the test code saved locally

GiteaMirror commented

2025-11-02 11:04:33 -06:00

@OAMchronicle commented on GitHub (Feb 19, 2025):

func TestTransferLogs1(t *testing.T) {
	open, err := os.Open("1131.log.zst")
	if err != nil {
		return
	}
	minioStorage, err := storage.NewMinioStorage(context.Background(),
		&setting.Storage{
			MinioConfig: setting.MinioStorageConfig{
				Endpoint:           "x.x.x.x:8333",
				AccessKeyID:        "xx",
				SecretAccessKey:    "xx",
				Bucket:             "gitea",
				UseSSL:             false,
				InsecureSkipVerify: true,
			},
		})
	if err != nil {
		return
	}
	_, err = minioStorage.Save("logs/1131.log.local.zst", open, -1)
	if err != nil {
		return
	}
}

After testing, it seems that Seaweed encountered an exception during upload

@OAMchronicle commented on GitHub (Feb 19, 2025): <details> <img width="1229" alt="Image" src="https://github.com/user-attachments/assets/d4729dfa-8fe6-405f-9fdf-fbb940e9a0b6" /> ```go func TestTransferLogs1(t *testing.T) { open, err := os.Open("1131.log.zst") if err != nil { return } minioStorage, err := storage.NewMinioStorage(context.Background(), &setting.Storage{ MinioConfig: setting.MinioStorageConfig{ Endpoint: "x.x.x.x:8333", AccessKeyID: "xx", SecretAccessKey: "xx", Bucket: "gitea", UseSSL: false, InsecureSkipVerify: true, }, }) if err != nil { return } _, err = minioStorage.Save("logs/1131.log.local.zst", open, -1) if err != nil { return } } ``` </details> After testing, it seems that Seaweed encountered an exception during upload

GiteaMirror commented

@wxiaoguang commented on GitHub (Feb 19, 2025):

Thank you for the feedback! I didn't quite use Actions or Seaweed, so at the moment, I have no more idea.

If you could help to figure out "what's wrong" (the key point, for example: is it Gitea's bug? which call/code is wrong; or Seaweed's problem? what's the difference), then maybe there could be some solutions to "fix".

@wxiaoguang commented on GitHub (Feb 19, 2025): Thank you for the feedback! I didn't quite use Actions or Seaweed, so at the moment, I have no more idea. If you could help to figure out "what's wrong" (the key point, for example: is it Gitea's bug? which call/code is wrong; or Seaweed's problem? what's the difference), then maybe there could be some solutions to "fix".

GiteaMirror commented

2025-11-02 11:04:33 -06:00

@OAMchronicle commented on GitHub (Feb 19, 2025):

Thank you for the feedback! I didn't quite use Actions or Seaweed, so at the moment, I have no more idea.

If you could help to figure out "what's wrong" (the key point, for example: is it Gitea's bug? which call/code is wrong; or Seaweed's problem? what's the difference), then maybe there could be some solutions to "fix".

When I use AWS/S3 SDK to upload, there is no such problem

@OAMchronicle commented on GitHub (Feb 19, 2025): > Thank you for the feedback! I didn't quite use Actions or Seaweed, so at the moment, I have no more idea. > > If you could help to figure out "what's wrong" (the key point, for example: is it Gitea's bug? which call/code is wrong; or Seaweed's problem? what's the difference), then maybe there could be some solutions to "fix". When I use AWS/S3 SDK to upload, there is no such problem

GiteaMirror commented

2025-11-02 11:04:33 -06:00

@OAMchronicle commented on GitHub (Feb 19, 2025):

//test code

package S3

import (
	"context"
	"crypto/tls"
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/credentials"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"net/http"
	"os"
	"testing"
	"upfile/src/conf"
)

var Cli *s3.Client

func init() {
	staticResolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...any) (aws.Endpoint, error) {
		return aws.Endpoint{
			PartitionID:       "aws",
			URL:               "http://x.x.x.x:8333",
			SigningRegion:     region,
			HostnameImmutable: true,
		}, nil
	})

	tr := &http.Transport{
		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
	}
	httpClient := &http.Client{Transport: tr}

	cfg, _ := config.LoadDefaultConfig(context.Background(),
		config.WithRegion("xx"),
		config.WithEndpointResolverWithOptions(staticResolver),
		config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("xx", "xx", "")),
		config.WithHTTPClient(httpClient),
	)

	Cli = s3.NewFromConfig(cfg)
}

func puttest(key string, obsClient *s3.Client) bool {
	file, err := os.Open(key)
	if err != nil {
		conf.Logger.Error(err)
		return false
	}
	defer file.Close()
	_, err = obsClient.PutObject(context.TODO(), &s3.PutObjectInput{
		Bucket: aws.String("gitea"),
		Key:    &key,
		Body:   file,
	})

	if err != nil {
		return false
	}

	return true
}
func TestUp(t *testing.T) {

	puttest("1131.log.zst", Cli)
}

@OAMchronicle commented on GitHub (Feb 19, 2025): //test code <details> ```go package S3 import ( "context" "crypto/tls" "github.com/aws/aws-sdk-go-v2/aws" "github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/credentials" "github.com/aws/aws-sdk-go-v2/service/s3" "net/http" "os" "testing" "upfile/src/conf" ) var Cli *s3.Client func init() { staticResolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...any) (aws.Endpoint, error) { return aws.Endpoint{ PartitionID: "aws", URL: "http://x.x.x.x:8333", SigningRegion: region, HostnameImmutable: true, }, nil }) tr := &http.Transport{ TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, } httpClient := &http.Client{Transport: tr} cfg, _ := config.LoadDefaultConfig(context.Background(), config.WithRegion("xx"), config.WithEndpointResolverWithOptions(staticResolver), config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("xx", "xx", "")), config.WithHTTPClient(httpClient), ) Cli = s3.NewFromConfig(cfg) } func puttest(key string, obsClient *s3.Client) bool { file, err := os.Open(key) if err != nil { conf.Logger.Error(err) return false } defer file.Close() _, err = obsClient.PutObject(context.TODO(), &s3.PutObjectInput{ Bucket: aws.String("gitea"), Key: &key, Body: file, }) if err != nil { return false } return true } func TestUp(t *testing.T) { puttest("1131.log.zst", Cli) } ``` <img width="1205" alt="Image" src="https://github.com/user-attachments/assets/3d0bf3ed-68ff-46b9-a5d4-4812c018b959" /> </details>

GiteaMirror commented

@OAMchronicle commented on GitHub (Feb 19, 2025):

So could this be an issue between Minio and AWS/S3 SDK

@OAMchronicle commented on GitHub (Feb 19, 2025): So could this be an issue between Minio and AWS/S3 SDK

GiteaMirror commented

@wxiaoguang commented on GitHub (Feb 19, 2025):

When using Minio SDK, what's the content of uploaded file? Decompressed? Truncated? Or randomly corrupted? Could you provide the file samples?

@wxiaoguang commented on GitHub (Feb 19, 2025): When using Minio SDK, what's the content of uploaded file? Decompressed? Truncated? Or randomly corrupted? Could you provide the file samples?

GiteaMirror commented

1131.log
1131.log.zst.txt

@OAMchronicle commented on GitHub (Feb 19, 2025):

You need to rename 1131.log.zst.txt to 1131.log.zst (GitHub cannot upload. zst files)

1131.log.zst is compressed from 1131.log

@OAMchronicle commented on GitHub (Feb 19, 2025): [1131.log](https://github.com/user-attachments/files/18858557/1131.log) [1131.log.zst.txt](https://github.com/user-attachments/files/18858556/1131.log.zst.txt) You need to rename 1131.log.zst.txt to 1131.log.zst (GitHub cannot upload. zst files) 1131.log.zst is compressed from 1131.log

GiteaMirror commented

@wxiaoguang commented on GitHub (Feb 19, 2025):

I have checked your samples: 1131.log.zst.txt (1131.log.zst) is right, after decompression, it matches 1131.log.

And I would reiterate my previous comment:

It seems to be a plain-text file? The foot bytes (from log) are 4b73e\r\n\r\n
Will the S3 storage auto decompress the compressed zstd files and respond plain-text?

We can see the the foot bytes of 1131.log is also something like 9c8ca\r\n\r\n.

So the best guess is: when Gitea tries to read the compressed "zstd" file from your S3, your S3 responds decompressed plain-text file. Then Gitea's zstd reader can't read it.

@wxiaoguang commented on GitHub (Feb 19, 2025): I have checked your samples: `1131.log.zst.txt (1131.log.zst)` is right, after decompression, it matches `1131.log`. And I would reiterate my previous comment: > It seems to be a plain-text file? The foot bytes (from log) are `4b73e\r\n\r\n` > Will the S3 storage auto decompress the compressed zstd files and respond plain-text? We can see the the foot bytes of `1131.log` is also something like `9c8ca\r\n\r\n`. ---- So the best guess is: when Gitea tries to read the compressed "zstd" file from your S3, your S3 responds decompressed plain-text file. Then Gitea's zstd reader can't read it.

GiteaMirror commented

@OAMchronicle commented on GitHub (Feb 19, 2025):

I have checked your samples: 1131.log.zst.txt (1131.log.zst) is right, after decompression, it matches 1131.log.

And I would reiterate my previous comment:

It seems to be a plain-text file? The foot bytes (from log) are 4b73e\r\n\r\n
Will the S3 storage auto decompress the compressed zstd files and respond plain-text?

We can see the the foot bytes of 1131.log is also something like 9c8ca\r\n\r\n.

So the best guess is: when Gitea tries to read the compressed "zstd" file from your S3, your S3 responds decompressed plain-text file. Then Gitea's zstd reader can't read it.

No, the file I sent was manually compressed. It must be the correct file. This issue is that when using Minio's SDK to upload, the file will be damaged

@OAMchronicle commented on GitHub (Feb 19, 2025): > I have checked your samples: `1131.log.zst.txt (1131.log.zst)` is right, after decompression, it matches `1131.log`. > > And I would reiterate my previous comment: > > > It seems to be a plain-text file? The foot bytes (from log) are `4b73e\r\n\r\n` > > Will the S3 storage auto decompress the compressed zstd files and respond plain-text? > > We can see the the foot bytes of `1131.log` is also something like `9c8ca\r\n\r\n`. > > So the best guess is: when Gitea tries to read the compressed "zstd" file from your S3, your S3 responds decompressed plain-text file. Then Gitea's zstd reader can't read it. No, the file I sent was manually compressed. It must be the correct file. This issue is that when using Minio's SDK to upload, the file will be damaged

GiteaMirror commented

2025-11-02 11:04:35 -06:00

@wxiaoguang commented on GitHub (Feb 19, 2025):

No, the file I sent was manually compressed. It must be the correct file. This issue is that when using Minio's SDK to upload, the file will be damaged

Then how would you explain these "footer bytes" problem? That file is a plain-text file.

And I have said many times: you need to figure out the Seaweed's behavior (with Minio SDK):

Could you help to confirm the seaweed's behavior? For example: use your test code , and read the logs/1131.log.zst raw content and dump it to see whether it is compressed or decompressed? (or: maybe the response is jut incorrect? just a guess)

If:

the uploaded content is right (in zstd compression)
the downloaded content is right (in zstd compression)

Then nothing wrong would happen, right?

So, which step is wrong? "the uploaded content"? or "the downloaded content"?

@wxiaoguang commented on GitHub (Feb 19, 2025): > No, the file I sent was manually compressed. It must be the correct file. This issue is that when using Minio's SDK to upload, the file will be damaged Then how would you explain these "footer bytes" problem? That file is a plain-text file. And I have said many times: you need to figure out the Seaweed's behavior (with Minio SDK): > > Could you help to confirm the seaweed's behavior? For example: use your test code , and read the `logs/1131.log.zst` raw content and dump it to see whether it is compressed or decompressed? (or: maybe the response is jut incorrect? just a guess) If: * the uploaded content is right (in zstd compression) * the downloaded content is right (in zstd compression) Then nothing wrong would happen, right? So, which step is wrong? "the uploaded content"? or "the downloaded content"?

GiteaMirror commented

@wxiaoguang commented on GitHub (Feb 19, 2025):

This issue is that when using Minio's SDK to upload, the file will be damaged

How the file is "damaged"? What's the content of uploaded file? Decompressed? Truncated? Or randomly corrupted? Could you provide the file samples?

Your samples above are ALL right.

@wxiaoguang commented on GitHub (Feb 19, 2025): > This issue is that when using Minio's SDK to upload, the file will be damaged How the file is "damaged"? What's the content of uploaded file? Decompressed? Truncated? Or randomly corrupted? Could you provide the file samples? Your samples above are ALL right.

GiteaMirror commented

2025-11-02 11:04:35 -06:00

@OAMchronicle commented on GitHub (Feb 19, 2025):

the uploaded content

@OAMchronicle commented on GitHub (Feb 19, 2025): the uploaded content

GiteaMirror commented

2025-11-02 11:04:35 -06:00

@wxiaoguang commented on GitHub (Feb 19, 2025):

the uploaded content

But your samples above are ALL right.

@wxiaoguang commented on GitHub (Feb 19, 2025): > the uploaded content But your samples above are ALL right.

GiteaMirror commented

@wxiaoguang commented on GitHub (Feb 19, 2025):

What's the content of uploaded file?

I mean, the "UPLOADED" file.

Since you suppose that the uploaded file is damaged, could you provide the uploaded and damaged one? But not the one you used to upload.

@wxiaoguang commented on GitHub (Feb 19, 2025): > What's the content of uploaded file? I mean, the "UPLOADED" file. Since you suppose that the uploaded file is damaged, could you provide the uploaded and damaged one? But not the one you used to upload.

GiteaMirror commented

1131.log.minio.zst.txt
1131.log.s3.zst.txt

@OAMchronicle commented on GitHub (Feb 19, 2025):

//test code

package actions

import (
	"code.gitea.io/gitea/modules/setting"
	"code.gitea.io/gitea/modules/storage"
	"code.gitea.io/gitea/modules/zstd"
	"context"
	"crypto/tls"
	"fmt"
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/credentials"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"io"
	"net/http"
	"os"
	"testing"
)

func TestTransferLogs1(t *testing.T) {
	open, err := os.Open("1131.log.zst")
	if err != nil {
		return
	}
	minioStorage, err := storage.NewMinioStorage(context.Background(),
		&setting.Storage{
			MinioConfig: setting.MinioStorageConfig{
				Endpoint:           "x.x.x.x:8333",
				AccessKeyID:        "xx",
				SecretAccessKey:    "xx",
				Bucket:             "gitea",
				UseSSL:             false,
				InsecureSkipVerify: true,
			},
		})
	if err != nil {
		return
	}
	_, err = minioStorage.Save("1131.log.minio.zst", open, -1)
	if err != nil {
		return
	}
}

var Cli *s3.Client

func init() {
	staticResolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...any) (aws.Endpoint, error) {
		return aws.Endpoint{
			PartitionID:       "aws",
			URL:               "http://x.x.x.x:8333",
			SigningRegion:     region,
			HostnameImmutable: true,
		}, nil
	})

	tr := &http.Transport{
		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
	}
	httpClient := &http.Client{Transport: tr}

	cfg, _ := config.LoadDefaultConfig(context.Background(),
		config.WithRegion("us-east-1"),
		config.WithEndpointResolverWithOptions(staticResolver),
		config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("xx", "xx", "")),
		config.WithHTTPClient(httpClient),
	)

	Cli = s3.NewFromConfig(cfg)
}

func puttest(key string, obsClient *s3.Client) bool {
	file, err := os.Open(key)
	if err != nil {
		return false
	}
	defer file.Close()
	_, err = obsClient.PutObject(context.TODO(), &s3.PutObjectInput{
		Bucket: aws.String("gitea"),
		Key:    aws.String("1131.log.s3.zst"),
		Body:   file,
	})

	if err != nil {
		return false
	}

	return true
}

func TestUp(t *testing.T) {
	puttest("1131.log.zst", Cli)
}

Execute TestUp and TestTransferLogs1 separately, and the results are as follows

@OAMchronicle commented on GitHub (Feb 19, 2025): //test code <details> ```go package actions import ( "code.gitea.io/gitea/modules/setting" "code.gitea.io/gitea/modules/storage" "code.gitea.io/gitea/modules/zstd" "context" "crypto/tls" "fmt" "github.com/aws/aws-sdk-go-v2/aws" "github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/credentials" "github.com/aws/aws-sdk-go-v2/service/s3" "io" "net/http" "os" "testing" ) func TestTransferLogs1(t *testing.T) { open, err := os.Open("1131.log.zst") if err != nil { return } minioStorage, err := storage.NewMinioStorage(context.Background(), &setting.Storage{ MinioConfig: setting.MinioStorageConfig{ Endpoint: "x.x.x.x:8333", AccessKeyID: "xx", SecretAccessKey: "xx", Bucket: "gitea", UseSSL: false, InsecureSkipVerify: true, }, }) if err != nil { return } _, err = minioStorage.Save("1131.log.minio.zst", open, -1) if err != nil { return } } var Cli *s3.Client func init() { staticResolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...any) (aws.Endpoint, error) { return aws.Endpoint{ PartitionID: "aws", URL: "http://x.x.x.x:8333", SigningRegion: region, HostnameImmutable: true, }, nil }) tr := &http.Transport{ TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, } httpClient := &http.Client{Transport: tr} cfg, _ := config.LoadDefaultConfig(context.Background(), config.WithRegion("us-east-1"), config.WithEndpointResolverWithOptions(staticResolver), config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("xx", "xx", "")), config.WithHTTPClient(httpClient), ) Cli = s3.NewFromConfig(cfg) } func puttest(key string, obsClient *s3.Client) bool { file, err := os.Open(key) if err != nil { return false } defer file.Close() _, err = obsClient.PutObject(context.TODO(), &s3.PutObjectInput{ Bucket: aws.String("gitea"), Key: aws.String("1131.log.s3.zst"), Body: file, }) if err != nil { return false } return true } func TestUp(t *testing.T) { puttest("1131.log.zst", Cli) } ``` Execute TestUp and TestTransferLogs1 separately, and the results are as follows <img width="1291" alt="Image" src="https://github.com/user-attachments/assets/3546d22b-3385-412d-8a67-46aa1f89413a" /> </details> [1131.log.minio.zst.txt](https://github.com/user-attachments/files/18858747/1131.log.minio.zst.txt) [1131.log.s3.zst.txt](https://github.com/user-attachments/files/18858746/1131.log.s3.zst.txt)

GiteaMirror commented

https://stackoverflow.com/questions/55816844/aws-s3-get-after-put-includes-chunk-signature-data

@wxiaoguang commented on GitHub (Feb 19, 2025):

OK, we can simply cat the damaged file, then the problem is clear:

@wxiaoguang commented on GitHub (Feb 19, 2025): OK, we can simply `cat` the damaged file, then the problem is clear: https://stackoverflow.com/questions/55816844/aws-s3-get-after-put-includes-chunk-signature-data ![Image](https://github.com/user-attachments/assets/be156684-6ce7-4af4-a86c-e16b66308940)

GiteaMirror commented