Webhooks randomly fail with 408 timeout #2618

@sbrl commented on GitHub (Jan 13, 2019):

Anyone at all? @lunny? Someone?

@sbrl commented on GitHub (Jan 13, 2019): Anyone at all? @lunny? Someone?

GiteaMirror commented

@barryp commented on GitHub (Jan 21, 2019):

I'm getting a similar thing sort of randomly

Delivery: Post http://www.home:9100/hooks/deploy-site: read tcp 10.0.0.107:12078->10.0.0.101:9100: i/o timeout

If I try a Test Delivery, it usually works fine, although that could be because the first invocation already did some time-consuming downloads - and the second invocation can complete much quicker. I'm using the same webhook @sbrl is using, although not behind nginx. The script it invokes can take maybe 5-15 seconds, perhaps even longer. It'd be nice to know what the timeout was, and even better if it could be configurable.

@barryp commented on GitHub (Jan 21, 2019): I'm getting a similar thing sort of randomly Delivery: Post http://www.home:9100/hooks/deploy-site: read tcp 10.0.0.107:12078->10.0.0.101:9100: i/o timeout If I try a Test Delivery, it usually works fine, although that could be because the first invocation already did some time-consuming downloads - and the second invocation can complete much quicker. I'm using the same [webhook](https://github.com/adnanh/webhook) @sbrl is using, although not behind nginx. The script it invokes can take maybe 5-15 seconds, perhaps even longer. It'd be nice to know what the timeout was, and even better if it could be configurable.

GiteaMirror commented

@sbrl commented on GitHub (Jan 23, 2019):

I think the timeout is already configurable, @barryp. It's this in custom/app.ini:

[webhook]
DELIVER_TIMEOUT = 60

....60 is the value I have set. If I try the "test" delivery, it does work fine some of the time. If I try it manually with curl from the same host, it always works. I think it's definitely a bug in Gitea.

@sbrl commented on GitHub (Jan 23, 2019): I think the timeout is already configurable, @barryp. It's this in `custom/app.ini`: ```ini [webhook] DELIVER_TIMEOUT = 60 ``` ....60 is the value I have set. If I try the "test" delivery, it does work fine some of the time. If I try it manually with `curl` _from the same host_, it _always_ works. I think it's definitely a bug in Gitea.

GiteaMirror commented

@sbrl commented on GitHub (Jan 23, 2019):

I can confirm that this is still an issue with the latest v1.7.0 release of Gitea.

@sbrl commented on GitHub (Jan 23, 2019): I can confirm that this is still an issue with the latest v1.7.0 release of Gitea.

GiteaMirror commented

@sbrl commented on GitHub (Feb 4, 2019):

Furthermore, I have a tcpdump capture that demonstrates the issue. It's a HTTPS connection, but hopefully still proves useful. Contact me (my email address is bugs at starbeamrainbowlabs dot com) to obtain said dump.

(Note to self: It's in my downloads)

@sbrl commented on GitHub (Feb 4, 2019): Furthermore, I have a `tcpdump` capture that demonstrates the issue. It's a HTTPS connection, but hopefully still proves useful. Contact me (my email address is `bugs at starbeamrainbowlabs dot com`) to obtain said dump. _(Note to self: It's in my downloads)_

GiteaMirror commented

@lunny commented on GitHub (Feb 5, 2019):

Can not reproduce on try.gitea.io and my instances.

@lunny commented on GitHub (Feb 5, 2019): Can not reproduce on try.gitea.io and my instances.

GiteaMirror commented

@sbrl commented on GitHub (Feb 6, 2019):

@lunny Oh dear. That's awkward. Do you have an email address I can send this packet capture to, if you think it'd help?

If further details are needed, I could try doing it over regular http. If you've got any other suggestions as to what could potentially be the cause, I'd love to hear it and I'll investigate.

I'm currently investigating using the git post-receive hook instead and writing a bash script as a workaround.

@sbrl commented on GitHub (Feb 6, 2019): @lunny Oh dear. That's awkward. Do you have an email address I can send this packet capture to, if you think it'd help? If further details are needed, I could try doing it over regular http. If you've got any other suggestions as to what could potentially be the cause, I'd love to hear it and I'll investigate. I'm currently investigating using the git post-receive hook instead and writing a bash script as a workaround.

GiteaMirror commented

@joyqat commented on GitHub (Mar 1, 2019):

Have the same problem

It might be a network problem. Does Gitea have a "retry when failed" config?

@joyqat commented on GitHub (Mar 1, 2019): Have the same problem ![image](https://user-images.githubusercontent.com/10184445/53627628-e9882580-3c43-11e9-97d4-7f31decdca7d.png) It might be a network problem. Does Gitea have a "retry when failed" config?

GiteaMirror commented

@lunny commented on GitHub (Mar 1, 2019):

@sbrl You can send message to me on discord.

@lunny commented on GitHub (Mar 1, 2019): @sbrl You can send message to me on discord.

GiteaMirror commented

@stale[bot] commented on GitHub (Apr 30, 2019):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

@stale[bot] commented on GitHub (Apr 30, 2019): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

GiteaMirror commented

@sbrl commented on GitHub (May 1, 2019):

Looks like I have issues with GitHub sending Webhooks to the same server, so I think it's probably a network issue - or something else on my end. Thanks for investigating though!

@sbrl commented on GitHub (May 1, 2019): Looks like I have issues with GitHub sending Webhooks to the same server, so I think it's probably a network issue - or something else on my end. Thanks for investigating though!

GiteaMirror commented

@c0h1b4 commented on GitHub (Aug 5, 2019):

I am having this same issues on a kubernetes deployment of Gitea with Drone.io.

Randomly Delivery: Post https://xxxxx/hook?...: dial tcp: i/o timeout

Sometimes it goes smooth, sometimes it stalls. Then if I press "Test Delivery" it goes thru. An auto retry with a X number of tries would probably fix it.

@c0h1b4 commented on GitHub (Aug 5, 2019): I am having this same issues on a kubernetes deployment of Gitea with Drone.io. Randomly Delivery: Post https://xxxxx/hook?...: dial tcp: i/o timeout Sometimes it goes smooth, sometimes it stalls. Then if I press "Test Delivery" it goes thru. An auto retry with a X number of tries would probably fix it.

GiteaMirror commented

@lafriks commented on GitHub (Aug 5, 2019):

That could also be network connectivity issues or dns problems

@lafriks commented on GitHub (Aug 5, 2019): That could also be network connectivity issues or dns problems

GiteaMirror commented

@c0h1b4 commented on GitHub (Aug 6, 2019):

No. The issue here is that I'm using a small arm cluster with kubernetes. Sometimes the timeout is > 5 secs. I fixed increasing the webhook timeout to 30 secs. But I believe that an automatic retry for a custom number of tries would be very welcome.

@c0h1b4 commented on GitHub (Aug 6, 2019): No. The issue here is that I'm using a small arm cluster with kubernetes. Sometimes the timeout is > 5 secs. I fixed increasing the webhook timeout to 30 secs. But I believe that an automatic retry for a custom number of tries would be very welcome.

GiteaMirror commented

@sbrl commented on GitHub (Aug 7, 2019):

What a co-incidence, @c0h1b4! My situation was something like this:

Gitea server ----> [ Internet ] -----> Home router ------> Raspberry Pi

If you're having issues with arm, maybe it's a bug in the linux networking stack for arm? I've no clue how to start debugging that though.

I'm pretty sure it's a network issue and nothing to do with Gitea. I know this because I setup a GitHub webhook that pointed tot he same place, and it also failed randomly in the exact same fashion.

I ended up writing a PHP proxy script that does the automatic retry that I run on a more stable box in the cloud (incidentally, this is the same box that has the Gitea server running). Said script looks a bit like this: https://pastebin.com/Un9B01s1

Very curious indeed.

@sbrl commented on GitHub (Aug 7, 2019): What a co-incidence, @c0h1b4! My situation was something like this: Gitea server ----> [ Internet ] -----> Home router ------> Raspberry Pi If you're having issues with arm, maybe it's a bug in the linux networking stack for arm? I've no clue how to start debugging that though. I'm pretty sure it's a network issue and nothing to do with Gitea. I know this because I setup a GitHub webhook that pointed tot he same place, and it also failed randomly in the exact same fashion. I ended up writing a PHP proxy script that does the automatic retry that I run on a more stable box in the cloud (incidentally, this is the same box that has the Gitea server running). Said script looks a bit like this: https://pastebin.com/Un9B01s1 Very curious indeed.

GiteaMirror commented

@src386 commented on GitHub (Oct 14, 2019):

I think I have the same issue, in a amd64 Docker environment.
Test delivery never worked (i/o timeout) but using curl with the same url works.
Tested both https (via traefik) and http (direct container to container).

Adding the DELIVER_TIMEOUT = 60 option slightly improves the situation, now I get HTTP 500, at least it's something.

@src386 commented on GitHub (Oct 14, 2019): I think I have the same issue, in a amd64 Docker environment. Test delivery never worked (i/o timeout) but using curl with the same url works. Tested both https (via traefik) and http (direct container to container). Adding the ```DELIVER_TIMEOUT = 60``` option slightly improves the situation, now I get HTTP 500, at least it's something.

GiteaMirror commented

@sbrl commented on GitHub (Oct 15, 2019):

Have you tried it with GitHub too, @src386?

@sbrl commented on GitHub (Oct 15, 2019): Have you tried it with GitHub too, @src386?

GiteaMirror commented

@deanpcmad commented on GitHub (Oct 27, 2019):

I'm also having this issue and I've tried multiple ways of trying to fix it...

Gitea in a docker container at home
Gitea running normally at home
Gitea on Docker on a VPS
Gitea running normally on a VPS
Both a clean database and imported database

I get the same i/o timeout issue no matter what I do. Of course webhooks are quite important especially with CI/CD. I was hoping to move to Gitea this weekend but after spending hours trying to get this work, I'm not sure now lol

Edit: using the latest stable version of Gitea btw which is 1.9.4

Another edit an hour later: I've sorted it. Turns out Drone checks if there's a .drone.yml file and if there isn't one, it won't allow a webhook? Very odd but glad I've sorted it!

@deanpcmad commented on GitHub (Oct 27, 2019): I'm also having this issue and I've tried multiple ways of trying to fix it... - Gitea in a docker container at home - Gitea running normally at home - Gitea on Docker on a VPS - Gitea running normally on a VPS - Both a clean database and imported database I get the same i/o timeout issue no matter what I do. Of course webhooks are quite important especially with CI/CD. I was hoping to move to Gitea this weekend but after spending hours trying to get this work, I'm not sure now lol Edit: using the latest stable version of Gitea btw which is 1.9.4 Another edit an hour later: I've sorted it. Turns out Drone checks if there's a `.drone.yml` file and if there isn't one, it won't allow a webhook? Very odd but glad I've sorted it!

GiteaMirror commented

@c0h1b4 commented on GitHub (Oct 28, 2019):

I'm also having this issue and I've tried multiple ways of trying to fix it...

Gitea in a docker container at home

Gitea running normally at home

Gitea on Docker on a VPS

Gitea running normally on a VPS

Both a clean database and imported database

I get the same i/o timeout issue no matter what I do. Of course webhooks are quite important especially with CI/CD. I was hoping to move to Gitea this weekend but after spending hours trying to get this work, I'm not sure now lol

Edit: using the latest stable version of Gitea btw which is 1.9.4

Another edit an hour later: I've sorted it. Turns out Drone checks if there's a .drone.yml file and if there isn't one, it won't allow a webhook? Very odd but glad I've sorted it!

Did you tried playing with the DELIVER_TIMEOUT parameter in the config? It helped me to fix this issue.

@c0h1b4 commented on GitHub (Oct 28, 2019): > I'm also having this issue and I've tried multiple ways of trying to fix it... > > * Gitea in a docker container at home > * Gitea running normally at home > * Gitea on Docker on a VPS > * Gitea running normally on a VPS > * Both a clean database and imported database > > I get the same i/o timeout issue no matter what I do. Of course webhooks are quite important especially with CI/CD. I was hoping to move to Gitea this weekend but after spending hours trying to get this work, I'm not sure now lol > > Edit: using the latest stable version of Gitea btw which is 1.9.4 > > Another edit an hour later: I've sorted it. Turns out Drone checks if there's a `.drone.yml` file and if there isn't one, it won't allow a webhook? Very odd but glad I've sorted it! Did you tried playing with the `DELIVER_TIMEOUT` parameter in the config? It helped me to fix this issue.

GiteaMirror commented

@sbrl commented on GitHub (Oct 29, 2019):

Many root causes, 1 symptom.

@sbrl commented on GitHub (Oct 29, 2019): Many root causes, 1 symptom.

GiteaMirror commented

@Veitor commented on GitHub (Mar 3, 2020):

same problem..

I have set DELIVER_TIMEOUT=30 and still timeout.
I execute curl <webhook URL> command inside my gitea container, it's very fast and no problem.
I test webhook within another repository, it's worked. This repository's webhook URL is same with previous repository's webhook URL, all they are my Drone CI adress.
How can it time out?

@Veitor commented on GitHub (Mar 3, 2020): same problem.. - I have set `DELIVER_TIMEOUT=30` and still timeout. - I execute `curl <webhook URL>` command inside my gitea container, it's very fast and no problem. - I test webhook within another repository, it's worked. This repository's webhook URL is same with previous repository's webhook URL, all they are my Drone CI adress. How can it time out? ![image](https://user-images.githubusercontent.com/7483890/75736641-c4d3fb80-5d38-11ea-8168-69af34e72903.png) ![image](https://user-images.githubusercontent.com/7483890/75736695-dcab7f80-5d38-11ea-90b0-a1d19981a8ff.png) ![image](https://user-images.githubusercontent.com/7483890/75736767-0e244b00-5d39-11ea-9541-03a5ec64135f.png)

GiteaMirror commented

@Veitor commented on GitHub (Mar 3, 2020):

same problem..

I have set DELIVER_TIMEOUT=30 and still timeout.

I execute curl <webhook URL> command inside my gitea container, it's very fast and no problem.

I test webhook within another repository, it's worked. This repository's webhook URL is same with previous repository's webhook URL, all they are my Drone CI adress.
How can it time out?

ok. I have solved this problem.
my gitea repository's webhook is Drone URL. and the point is that CI repository - setting - Main - Configuration specified yaml file named .drone.yml, and my git repository's file named drone.yml!!
that's worker after modified file name... :)

However, this problem should not caused timeout error, network has no problem!

@Veitor commented on GitHub (Mar 3, 2020): > same problem.. > > * I have set `DELIVER_TIMEOUT=30` and still timeout. > * I execute `curl <webhook URL>` command inside my gitea container, it's very fast and no problem. > * I test webhook within another repository, it's worked. This repository's webhook URL is same with previous repository's webhook URL, all they are my Drone CI adress. > How can it time out? > ![image](https://user-images.githubusercontent.com/7483890/75736641-c4d3fb80-5d38-11ea-8168-69af34e72903.png) > ![image](https://user-images.githubusercontent.com/7483890/75736695-dcab7f80-5d38-11ea-90b0-a1d19981a8ff.png) > ![image](https://user-images.githubusercontent.com/7483890/75736767-0e244b00-5d39-11ea-9541-03a5ec64135f.png) ok. I have solved this problem. my gitea repository's webhook is Drone URL. and the point is that CI `repository - setting - Main - Configuration` specified yaml file named `.drone.yml`, and my git repository's file named `drone.yml`!! that's worker after modified file name... :) However, this problem should not caused timeout error, network has no problem!

GiteaMirror commented

@deanpcmad commented on GitHub (Mar 3, 2020):

^ Yeah my thoughts too

@deanpcmad commented on GitHub (Mar 3, 2020): ^ Yeah my thoughts too

GiteaMirror commented

@sbrl commented on GitHub (Mar 6, 2020):

Glad you fixed it! :D

Although for future reference if you see these symptoms, it's almost certainly the particular issue I experienced here.

@sbrl commented on GitHub (Mar 6, 2020): Glad you fixed it! :D Although for future reference if you see these symptoms, it's almost certainly the particular issue I experienced here.

GiteaMirror commented