Setup Mirror failure: Migration failed. Stream 7 was not closed cleanly #13381

Closed
opened 2025-11-02 10:40:35 -06:00 by GiteaMirror · 7 comments
Owner

Originally created by @Slyke on GitHub (Aug 10, 2024).

Description

When setting up a mirror for large repositories I get the following error:

migration failed: clone error: exit status 128 - error: RPC failed; curl 92 HTTP/2 stream 7 was not closed cleanly: CANCEL (err 8) error: 2200 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output

Pod log:

anly: CANCEL (err 8)
error: 2916 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
2024/08/10 11:56:04 ...ervices/task/task.go:53:handler() [E] Run task failed: migration failed: clone error: exit status 128 - error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8)
error: 2916 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
2024/08/10 11:56:06 ...eb/routing/logger.go:102:func1() [I] router: completed GET /user/task/33 for 10.244.0.167:0, 200 OK in 11.9ms @ user/task.go:16(user.TaskStatus)

I can mirror smaller repos with no problem. Seems to be an issue with larger ones. I have only tested this with Github repos.

I've seen examples on how to fix this with git. Apparently:

git config http.postBuffer 52428800

Will fix it. Unsure how to set this inside Gitea though.

I tried to get this to happen on the demo site, but the migration just never finishes: https://gitea.com/ny2Jd942gxMtypbx/test

Gitea Version

1.22.0

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

image

Git Version

No response

Operating System

Docker

How are you running Gitea?

Running on Kubernetes.

Database

PostgreSQL

Originally created by @Slyke on GitHub (Aug 10, 2024). ### Description When setting up a mirror for large repositories I get the following error: ``` migration failed: clone error: exit status 128 - error: RPC failed; curl 92 HTTP/2 stream 7 was not closed cleanly: CANCEL (err 8) error: 2200 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output ``` Pod log: ``` anly: CANCEL (err 8) error: 2916 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output 2024/08/10 11:56:04 ...ervices/task/task.go:53:handler() [E] Run task failed: migration failed: clone error: exit status 128 - error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8) error: 2916 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output 2024/08/10 11:56:06 ...eb/routing/logger.go:102:func1() [I] router: completed GET /user/task/33 for 10.244.0.167:0, 200 OK in 11.9ms @ user/task.go:16(user.TaskStatus) ``` I can mirror smaller repos with no problem. Seems to be an issue with larger ones. I have only tested this with Github repos. I've seen examples on how to fix this with git. Apparently: ``` git config http.postBuffer 52428800 ``` Will fix it. Unsure how to set this inside Gitea though. I tried to get this to happen on the demo site, but the migration just never finishes: https://gitea.com/ny2Jd942gxMtypbx/test ### Gitea Version 1.22.0 ### Can you reproduce the bug on the Gitea demo site? No ### Log Gist _No response_ ### Screenshots ![image](https://github.com/user-attachments/assets/2d31e253-a1b1-4220-85a5-f28ebe73e00d) ### Git Version _No response_ ### Operating System Docker ### How are you running Gitea? Running on Kubernetes. ### Database PostgreSQL
GiteaMirror added the issue/needs-feedbacktype/bug labels 2025-11-02 10:40:36 -06:00
Author
Owner

@Slyke commented on GitHub (Aug 11, 2024):

On the Gitea demo site, it timed out after about 24 hours with the following error:

Migrating from https://github.com/qmk/qmk_firmware failed.

error while listing repos: GET https://api.github.com/repos/qmk/qmk_firmware/pulls/9116/reviews/413400876/comments?per_page=100: 500 []
@Slyke commented on GitHub (Aug 11, 2024): On the Gitea demo site, it timed out after about 24 hours with the following error: ``` Migrating from https://github.com/qmk/qmk_firmware failed. error while listing repos: GET https://api.github.com/repos/qmk/qmk_firmware/pulls/9116/reviews/413400876/comments?per_page=100: 500 [] ```
Author
Owner

@yp05327 commented on GitHub (Oct 8, 2024):

If you are using nogogit version, you can execute git config, as Gitea uses git cmd directly.
About the Gitea demo site, Github API has access rate limit, maybe it is caused by it. And it is a known issue.

@yp05327 commented on GitHub (Oct 8, 2024): If you are using nogogit version, you can execute git config, as Gitea uses git cmd directly. About the Gitea demo site, Github API has access rate limit, maybe it is caused by it. And it is a known issue.
Author
Owner

@nealey commented on GitHub (Dec 2, 2024):

I am running into this consistently, trying to mirror https://github.com/adactio/TheSession-data, which has a few 15MB files checked in.

@nealey commented on GitHub (Dec 2, 2024): I am running into this consistently, trying to mirror https://github.com/adactio/TheSession-data, which has a few 15MB files checked in.
Author
Owner

@RavenNil commented on GitHub (Dec 11, 2024):

Gitea 1.22.4 docker version, mirror https://github.com/adactio/TheSession-data, with following error:

migration failed: clone error: exit status 128 
- error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8) error: 6685 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output
@RavenNil commented on GitHub (Dec 11, 2024): Gitea 1.22.4 docker version, mirror https://github.com/adactio/TheSession-data, with following error: ```txt migration failed: clone error: exit status 128 - error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8) error: 6685 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output ```
Author
Owner

@lunny commented on GitHub (Dec 11, 2024):

Both the two repositories can be mirrored successfully in demo site.
https://demo.gitea.com/lunny/TheSession-data
https://demo.gitea.com/lunny/qmk_firmware-mirror

@lunny commented on GitHub (Dec 11, 2024): Both the two repositories can be mirrored successfully in demo site. https://demo.gitea.com/lunny/TheSession-data https://demo.gitea.com/lunny/qmk_firmware-mirror
Author
Owner

@RavenNil commented on GitHub (Dec 12, 2024):

Gitea version: 1.22.5 docker
Database: postgres:16 docker

After adding the following to /data/gitea/home/.gitconfig,

[http]
        postBuffer = 524288000
        version = HTTP/1.1
        sslVerify = false

The error reason is changed.

migration failed: clone error: exit status 128 - error: RPC failed; curl 56 OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 0 error: 2880 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output

BTW, when i run git clone https://github.com/adactio/TheSession-data inside the Gitea Docker container, everything works fine...

30d6a2d52923:/# GIT_CONFIG=/data/gitea/home/.gitconfig git clone -v https://github.com/adactio/TheSession-data 
Cloning into 'TheSession-data'...
POST git-upload-pack (175 bytes)
POST git-upload-pack (202 bytes)
remote: Enumerating objects: 8116, done.
remote: Counting objects: 100% (499/499), done.
remote: Compressing objects: 100% (122/122), done.
remote: Total 8116 (delta 377), reused 499 (delta 377), pack-reused 7617 (from 1)
Receiving objects: 100% (8116/8116), 1.09 GiB | 2.22 MiB/s, done.
Resolving deltas: 100% (6174/6174), done.
Updating files: 100% (22/22), done.
@RavenNil commented on GitHub (Dec 12, 2024): Gitea version: 1.22.5 docker Database: postgres:16 docker After adding the following to /data/gitea/home/.gitconfig, ```txt [http] postBuffer = 524288000 version = HTTP/1.1 sslVerify = false ``` The error reason is changed. ```txt migration failed: clone error: exit status 128 - error: RPC failed; curl 56 OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 0 error: 2880 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output ``` BTW, when i run `git clone https://github.com/adactio/TheSession-data` inside the Gitea Docker container, everything works fine... ```txt 30d6a2d52923:/# GIT_CONFIG=/data/gitea/home/.gitconfig git clone -v https://github.com/adactio/TheSession-data Cloning into 'TheSession-data'... POST git-upload-pack (175 bytes) POST git-upload-pack (202 bytes) remote: Enumerating objects: 8116, done. remote: Counting objects: 100% (499/499), done. remote: Compressing objects: 100% (122/122), done. remote: Total 8116 (delta 377), reused 499 (delta 377), pack-reused 7617 (from 1) Receiving objects: 100% (8116/8116), 1.09 GiB | 2.22 MiB/s, done. Resolving deltas: 100% (6174/6174), done. Updating files: 100% (22/22), done. ```
Author
Owner

@GiteaBot commented on GitHub (Jan 11, 2025):

We close issues that need feedback from the author if there were no new comments for a month. 🍵

@GiteaBot commented on GitHub (Jan 11, 2025): We close issues that need feedback from the author if there were no new comments for a month. :tea:
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#13381