Gitea is slow when external avatar services are down #2898

@gaah666 commented on GitHub (Apr 25, 2019):

@lunny Yes I know that. That is not the issue.

The issue is that when you want to use them, but the service is down the gitea pages stop loading, waiting for services that are down for a full minute.
Better behavior is that the pages should load anyway (since avatars, although nice to have isn't crucial for gitea usage).

@gaah666 commented on GitHub (Apr 25, 2019): @lunny Yes I know that. That is not the issue. The issue is that when you want to use them, but the service is down the gitea pages stop loading, waiting for services that are down for a full minute. Better behavior is that the pages should load anyway (since avatars, although nice to have isn't crucial for gitea usage).

GiteaMirror commented

@zeripath commented on GitHub (Apr 25, 2019):

Agreed. Why don't we provide a per user avatar link which on request redirects to the looked up avatar instead of providing the avatar link directly.

@zeripath commented on GitHub (Apr 25, 2019): Agreed. Why don't we provide a per user avatar link which on request redirects to the looked up avatar instead of providing the avatar link directly.

GiteaMirror commented

https://gitlab.com/strk/go-libravatar/blob/master/libravatar.go#L191

@mrsdizzie commented on GitHub (Apr 26, 2019):

@zeripath I think that would have the same problem, that each page would request those links trying to display an avatar, which would then need the real link.

I think there are probably multiple problems related to avatars from what I've heard, but after going down a hole I think this is something particular (sorry for long comments):

The error logs above are specific to using federated avatars and looking up the SRV record for the domain name in the users email address, which comes from the go-libavatar library we use here:

strk.kbt.io/projects/go/libravatar

This exmple seems to have ENABLE_FEDERATED_AVATAR = true and the user email is set to x@localhost

So the library above is doing an SRV query to localhost based on that email address.

The go dns resolver will look in /etc/resolv.conf on a unix machine by default usually has something likeoptions timeout:2 attempts:5

So timeout after 2 seconds and try 5 times. That is per nameserver. So if somebody were using two name servers (say 8.8.8.8, 8.8.4.4) it would do that SRV query 5 times for each nameserver, timing out after 2 seconds. That would be 20 seconds total for each avatar -- as seen in the logs above. If the person above had say, 3 avatars to look up it would time out after exactly 1 minute (20 * 3).

Obviously this problem can be exponentially worse if you are trying to look up 5, 10, 20 of these. Or hard to diagnose if you use federated avatars and of 10 of them, one is a bad email address that adds an extra 20 seconds load.

So there are a couple of things to think about:

1: Don't try to load more than one of the same avatar if the first one has failed. Perhaps this can happen by looking up what needs to be loaded and doing it all at once, failing if the first one times out.

2: Don't rely on even one DNS lookup that can take 20 seconds to fail. This would probably mean changes to the referenced library at strk.kbt.io/projects/go/libravatar

3: Just store a copy of federated/gravatar avatar locally and refresh that copy every x amount of time as a background task. Do the original lookup when an account is set up so it doesn't block normal page loads and then refresh at an interval.

4: Maybe as a temporary fix for some examples like this one, don't lookup an avatar for localhost or other probably not real domain names

I think a good future solution is to do lookup of the avatar and save that image locally, refreshing either on a time basis, or perhaps if somebody visits their profile page as well. Even if the timeouts are made better, I don't think each page load should rely on links and/or connections to outside sites that may or may not be working at the moment. This would fix the "gitea is slow because of another server" problem and also avoid the potential "sometimes my pages loose all avatars" problem too. I believe this is something that popular services that make use of Gravatar do (Github, bigger WordPress sites, etc...)

Alternatively, if the above change isn't likely to happen perhaps dropping support for federated avatars (not gravatar or similar services). It can't ever be reasonable to run a dozen SRV queries when trying to load a page, even with a better timeouts.

@mrsdizzie commented on GitHub (Apr 26, 2019): @zeripath I think that would have the same problem, that each page would request those links trying to display an avatar, which would then need the real link. I think there are probably multiple problems related to avatars from what I've heard, but after going down a hole I think this is something particular (sorry for long comments): The error logs above are specific to using federated avatars and looking up the SRV record for the domain name in the users email address, which comes from the go-libavatar library we use here: strk.kbt.io/projects/go/libravatar https://gitlab.com/strk/go-libravatar/blob/master/libravatar.go#L191 This exmple seems to have ```ENABLE_FEDERATED_AVATAR = true``` and the user email is set to ```x@localhost``` So the library above is doing an SRV query to localhost based on that email address. The go dns resolver will look in ```/etc/resolv.conf``` on a unix machine by default usually has something like```options timeout:2 attempts:5``` So timeout after 2 seconds and try 5 times. That is per nameserver. So if somebody were using two name servers (say 8.8.8.8, 8.8.4.4) it would do that SRV query 5 times for each nameserver, timing out after 2 seconds. That would be 20 seconds total for each avatar -- as seen in the logs above. If the person above had say, 3 avatars to look up it would time out after exactly 1 minute (20 * 3). Obviously this problem can be exponentially worse if you are trying to look up 5, 10, 20 of these. Or hard to diagnose if you use federated avatars and of 10 of them, one is a bad email address that adds an extra 20 seconds load. So there are a couple of things to think about: 1: Don't try to load more than one of the same avatar if the first one has failed. Perhaps this can happen by looking up what needs to be loaded and doing it all at once, failing if the first one times out. 2: Don't rely on even one DNS lookup that can take 20 seconds to fail. This would probably mean changes to the referenced library at strk.kbt.io/projects/go/libravatar 3: Just store a copy of federated/gravatar avatar locally and refresh that copy every x amount of time as a background task. Do the original lookup when an account is set up so it doesn't block normal page loads and then refresh at an interval. 4: Maybe as a temporary fix for some examples like this one, don't lookup an avatar for localhost or other probably not real domain names I think a good future solution is to do lookup of the avatar and save that image locally, refreshing either on a time basis, or perhaps if somebody visits their profile page as well. Even if the timeouts are made better, I don't think each page load should rely on links and/or connections to outside sites that may or may not be working at the moment. This would fix the "gitea is slow because of another server" problem and also avoid the potential "sometimes my pages loose all avatars" problem too. I believe this is something that popular services that make use of Gravatar do (Github, bigger WordPress sites, etc...) Alternatively, if the above change isn't likely to happen perhaps dropping support for federated avatars (not gravatar or similar services). It can't ever be reasonable to run a dozen SRV queries when trying to load a page, even with a better timeouts.

GiteaMirror commented

@techknowlogick commented on GitHub (Apr 26, 2019):

As @strk is a member of Gitea maintainer team, sending patches upstream would likely be possible.

We could use github.com/miekg/dns which can leverage DNS timeouts, and also possible cache timeouts (maybe for 1 hour, or something smaller than caching the positive responses).

@techknowlogick commented on GitHub (Apr 26, 2019): As @strk is a member of Gitea maintainer team, sending patches upstream would likely be possible. We could use github.com/miekg/dns which can leverage DNS timeouts, and also possible cache timeouts (maybe for 1 hour, or something smaller than caching the positive responses).

GiteaMirror commented

@lunny commented on GitHub (Apr 26, 2019):

As @mrsdizzie 's advice, we could cache all external avatars and refresh them every day.

@lunny commented on GitHub (Apr 26, 2019): As @mrsdizzie 's advice, we could cache all external avatars and refresh them every day.

GiteaMirror commented

@mrsdizzie commented on GitHub (Apr 26, 2019):

I think we could/should do both of those: Make the fetching of the avatars better but also cache them on disk for displaying; Ideally Gitea shouldn't rely on connections to any third party servers for content to display on page views if possible.

@mrsdizzie commented on GitHub (Apr 26, 2019): I think we could/should do both of those: Make the fetching of the avatars better but also cache them on disk for displaying; Ideally Gitea shouldn't rely on connections to any third party servers for content to display on page views if possible.

GiteaMirror commented

@gaah666 commented on GitHub (Apr 26, 2019):

That was really what I wanted to say, as pointed out above. I wrote about avatars because that's how I noticed it, but my problem is really that gitea should function as it should, even if all/any external links/sites are down.

And, IMHO, external services add value (I love avatars) so it would really be unfortunate if it was removed... Just treat it as a best effort, if it works it works but don't let core functionality depend on them.

@gaah666 commented on GitHub (Apr 26, 2019): That was really what I wanted to say, as pointed out above. I wrote about avatars because that's how I noticed it, but my problem is really that gitea should function as it should, even if all/any external links/sites are down. And, IMHO, external services add value (I love avatars) so it would really be unfortunate if it was removed... Just treat it as a best effort, if it works it works but don't let core functionality depend on them.

GiteaMirror commented

@strk commented on GitHub (Jun 25, 2019):

I'm very happy to accept a patch for go-libravatar to tweak timeouts/TTLs.

Lookup could be maybe done by some background process to reduce the pain.

As per local cache, that one could be implemented in Gitea itself, no need to be in the library (for example we might want Gitea to include in that cache avatars that were obtained in a different way)

@strk commented on GitHub (Jun 25, 2019): I'm very happy to accept a patch for `go-libravatar` to tweak `timeouts/TTLs.` Lookup could be maybe done by some background process to reduce the pain. As per local cache, that one could be implemented in Gitea itself, no need to be in the library (for example we might want Gitea to include in that cache avatars that were obtained in a different way)

GiteaMirror commented

@stale[bot] commented on GitHub (Aug 24, 2019):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

@stale[bot] commented on GitHub (Aug 24, 2019): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

GiteaMirror commented

@lunny commented on GitHub (Aug 25, 2019):

@strk any progress? I think maybe you can move go-libravatar from gitlab to gitea instance if you want some help on that.

@lunny commented on GitHub (Aug 25, 2019): @strk any progress? I think maybe you can move `go-libravatar` from gitlab to gitea instance if you want some help on that.

GiteaMirror commented

@strk commented on GitHub (Aug 26, 2019):

There's already a mirror on gitlab: https://gitlab.com/strk/go-libravatar/

@strk commented on GitHub (Aug 26, 2019): There's already a mirror on gitlab: https://gitlab.com/strk/go-libravatar/

GiteaMirror commented

@strk commented on GitHub (Aug 26, 2019):

And yes, help is welcome (although I still think cache should be done in Gitea itself)

@strk commented on GitHub (Aug 26, 2019): And yes, help is welcome (although I still think cache should be done in Gitea itself)

GiteaMirror commented

@strk commented on GitHub (Sep 19, 2019):

I've a plan: gitea always passes to the browser a url to self (/user/x/avatar/size) and the server takes care of the redirect. How does that sound ?

@strk commented on GitHub (Sep 19, 2019): I've a plan: gitea always passes to the browser a url to self (`/user/x/avatar/size`) and the server takes care of the redirect. How does that sound ?

GiteaMirror commented