Performance issues with conversation tab in PR with comments/threads and images/rst files changes #13172

@Persimmonboy commented on GitHub (Jun 19, 2024):

I think the problem is related to the frontend part on the client side that handles the review sections with comments, as the backend seems to load in ~15 seconds, after which about 3 minutes or more comments are loaded:

@Persimmonboy commented on GitHub (Jun 19, 2024): I think the problem is related to the frontend part on the client side that handles the review sections with comments, as the backend seems to load in ~15 seconds, after which about 3 minutes or more comments are loaded: ![image](https://github.com/go-gitea/gitea/assets/8835079/4fe2fe38-1fac-4349-a622-65525703907b) ![image](https://github.com/go-gitea/gitea/assets/8835079/62fa41e5-d4b7-468b-9859-809b93c4fb8a) ![image](https://github.com/go-gitea/gitea/assets/8835079/7467fe44-3e76-49f2-b360-57a8d3e3884d) ![image](https://github.com/go-gitea/gitea/assets/8835079/4edf1d16-9613-45b4-9af1-ddaee95ad371)

GiteaMirror commented

@wxiaoguang commented on GitHub (Jun 19, 2024):

Since 1.21 is outdated and won't get regular fixes, could you upgrade to 1.22-nighty (it will be 1.22.1 soon and contains a lot of bug fixes)?

Then could you provide the browser's profile?

Update: if it's impossible to upgrade, continue debugging with 1.21 might also be OK .....

@wxiaoguang commented on GitHub (Jun 19, 2024): Since 1.21 is outdated and won't get regular fixes, could you upgrade to 1.22-nighty (it will be 1.22.1 soon and contains a lot of bug fixes)? Then could you provide the browser's profile? Update: if it's impossible to upgrade, continue debugging with 1.21 might also be OK ..... <details> ![image](https://github.com/go-gitea/gitea/assets/2114189/a6f2ea86-ae52-49c8-9040-d13f9f5c58b8) <details>

GiteaMirror commented

@wxiaoguang commented on GitHub (Jun 19, 2024):

Well, indeed ..... slow $.fn.tab

@wxiaoguang commented on GitHub (Jun 19, 2024): Well, indeed ..... slow `$.fn.tab`

GiteaMirror commented

Link to bucket with performance profle json

@Persimmonboy commented on GitHub (Jun 19, 2024):

@wxiaoguang I updated to version 1.22-nightly, but nothing changed. PR conversation tab loading remained slow.

Current version:

gitea-69c59cbf5f-d6nd2:/var/lib/gitea$ gitea --version
Gitea version 1.22.0+56-ge8e43a7ee4 built with GNU Make 4.4.1, go1.22.4 : bindata, timetzdata, sqlite, sqlite_unlock_notify
gitea-69c59cbf5f-d6nd2:/var/lib/gitea$

@Persimmonboy commented on GitHub (Jun 19, 2024): @wxiaoguang I updated to version 1.22-nightly, but nothing changed. PR conversation tab loading remained slow. Current version: ``` gitea-69c59cbf5f-d6nd2:/var/lib/gitea$ gitea --version Gitea version 1.22.0+56-ge8e43a7ee4 built with GNU Make 4.4.1, go1.22.4 : bindata, timetzdata, sqlite, sqlite_unlock_notify gitea-69c59cbf5f-d6nd2:/var/lib/gitea$ ``` [Link to bucket with performance profle json](https://gitea-issue.obs.eu-de.otc.t-systems.com/Trace-20240619T171021.json)

GiteaMirror commented

@wxiaoguang commented on GitHub (Jun 19, 2024):

Thank you very much. I will try to reproduce it and try to fix

@wxiaoguang commented on GitHub (Jun 19, 2024): Thank you very much. I will try to reproduce it and try to fix

GiteaMirror commented

@wxiaoguang commented on GitHub (Jun 20, 2024):

I can reproduce it now. WIP: Fix tab performance #31437 .

Need to figure out how to fix ....

@wxiaoguang commented on GitHub (Jun 20, 2024): I can reproduce it now. WIP: Fix tab performance #31437 . Need to figure out how to fix ....

GiteaMirror commented

https://demo.gitea.com/ashakhba/application-perfomance-managment/pulls/1

@Persimmonboy commented on GitHub (Jun 20, 2024):

@wxiaoguang

Thanks for the quick reply!

Do you think it's not a combination of comments with closed reviews that can also have comments and just comments?

I'm trying to reproduce the problem, and perfomance profile looks similar, but it loads faster, because in my problematic PR resolved reviews with comments and the comments themselves are more, which I mentioned in the first post, than in the test one.

Here is my test repository:

@Persimmonboy commented on GitHub (Jun 20, 2024): @wxiaoguang Thanks for the quick reply! Do you think it's not a combination of comments with closed reviews that can also have comments and just comments? I'm trying to reproduce the problem, and perfomance profile looks similar, but it loads faster, because in my problematic PR resolved reviews with comments and the comments themselves are more, which I mentioned in the first post, than in the test one. Here is my test repository: https://demo.gitea.com/ashakhba/application-perfomance-managment/pulls/1 ![image](https://github.com/go-gitea/gitea/assets/8835079/fd39c137-098b-4be8-9221-398b11553ad6)

GiteaMirror commented

@wxiaoguang commented on GitHub (Jun 20, 2024):

Do you think it's not a combination of comments with closed reviews that can also have comments and just comments?

That's another point, so maybe also related.

@wxiaoguang commented on GitHub (Jun 20, 2024): > Do you think it's not a combination of comments with closed reviews that can also have comments and just comments? That's another point, so maybe also related.

GiteaMirror commented

@wxiaoguang commented on GitHub (Jun 20, 2024):

Link to bucket with performance profle json

According to this profile report, your instance's slowdown is 100% related to the $.fn.tab

Do you think it's not a combination of comments with closed reviews that can also have comments and just comments?

However, it could still be a problem (another problem ......)

Are you able to run main-nightly (aka unreleased 1.23-nightly) in your production? If yes, I could provide more changes in the main branch, because some refactoring changes might not be suitable to backport to the stable 1.22 release.

If you could only use 1.22.x at the moment .... I could try to backport some changes if they are proven to work in 1.23, but I can't 100% guarantee at the moment.

@wxiaoguang commented on GitHub (Jun 20, 2024): > [Link to bucket with performance profle json](...) According to this profile report, your instance's slowdown is 100% related to the `$.fn.tab` > Do you think it's not a combination of comments with closed reviews that can also have comments and just comments? However, it could still be a problem (another problem ......) ---- Are you able to run main-nightly (aka unreleased 1.23-nightly) in your production? If yes, I could provide more changes in the main branch, because some refactoring changes might not be suitable to backport to the stable 1.22 release. If you could only use 1.22.x at the moment .... I could try to backport some changes if they are proven to work in 1.23, but I can't 100% guarantee at the moment.

GiteaMirror commented

@Persimmonboy commented on GitHub (Jun 20, 2024):

@wxiaoguang

This is the dev environment I reproduced using helm chart gitea from the prod gitea data and database, so I can update.

The gitea/gitea:nightly-rootless from dockerhub is required version 1.23-nightly right?

@Persimmonboy commented on GitHub (Jun 20, 2024): @wxiaoguang This is the dev environment I reproduced using helm chart gitea from the prod gitea data and database, so I can update. The gitea/gitea:nightly-rootless from dockerhub is required version 1.23-nightly right?

GiteaMirror commented

@wxiaoguang commented on GitHub (Jun 20, 2024):

The gitea/gitea:nightly-rootless from dockerhub is required version 1.23-nightly right?

Yes, nightly-rootless is for unreleased 1.23-nightly, but I haven't made any attempt yet 🤣 , I will notify here when I would have made some progress.

@wxiaoguang commented on GitHub (Jun 20, 2024): > The gitea/gitea:nightly-rootless from dockerhub is required version 1.23-nightly right? Yes, `nightly-rootless` is for unreleased 1.23-nightly, but I haven't made any attempt yet 🤣 , I will notify here when I would have made some progress.

GiteaMirror commented

@wxiaoguang commented on GitHub (Jun 22, 2024):

A new gitea/gitea:nightly-rootless is ready. Could you try it? I guess:

The frontend slow-down problem should be almost resolved (it might be still a little slow not that slow)
If it is still very slow, you could try to append _ui_performance_trace=1 as query parameter to that page, then the browser's console could print some trace logs when the page is fully loaded (eg: access https://localhost/?_ui_performance_trace=1 , https://localhost/?key=value&_ui_performance_trace=1: ref #31459
And the browser's performance profile is still helpful if you could collect that data when a page is slow

@wxiaoguang commented on GitHub (Jun 22, 2024): A new `gitea/gitea:nightly-rootless` is ready. Could you try it? I guess: * The frontend slow-down problem should be almost resolved (it might be still a little slow not that slow) * If it is still very slow, you could try to append `_ui_performance_trace=1` as query parameter to that page, then the browser's console could print some trace logs when the page is fully loaded (eg: access `https://localhost/?_ui_performance_trace=1` , `https://localhost/?key=value&_ui_performance_trace=1`: ref #31459 * And the browser's performance profile is still helpful if you could collect that data when a page is slow

GiteaMirror commented

Link to bucket with performance profle json

@Persimmonboy commented on GitHub (Jun 22, 2024):

@wxiaoguang

Thanky you for quick update.

I installed the version, and yes, it is indeed a bit slow, but not as slow as it was before, the difference is very strong, the page loads much faster (180 seconds -> ~30 seconds):

Perfomance trace with option /docs/application-performance-management/pulls/6?_ui_performance_trace=1:

UPD: Changed to private.

@Persimmonboy commented on GitHub (Jun 22, 2024): @wxiaoguang Thanky you for quick update. I installed the version, and yes, it is indeed a bit slow, but not as slow as it was before, the difference is very strong, the page loads much faster (180 seconds -> ~30 seconds): ![image](https://github.com/go-gitea/gitea/assets/8835079/5b5d6037-6619-445b-b281-98fc8e60afda) Perfomance trace with option `/docs/application-performance-management/pulls/6?_ui_performance_trace=1`: [Link to bucket with performance profle json](https://gitea-issue.obs.eu-de.otc.t-systems.com/Trace-20240622T185002.json) UPD: Changed to private.

GiteaMirror commented

@wxiaoguang commented on GitHub (Jun 22, 2024):

According to the latest profile:

network takes about 22s (server backend & transport)
frontend JS takes about 7s
- dropdown takes about 3s
- tab takes about 3s

I guess I have no new idea to do quick optimization at the moment, since that PR is really large.

@wxiaoguang commented on GitHub (Jun 22, 2024): According to the latest profile: 1. network takes about 22s (server backend & transport) 2. frontend JS takes about 7s * dropdown takes about 3s * tab takes about 3s I guess I have no new idea to do quick optimization at the moment, since that PR is really large.

GiteaMirror commented

@wxiaoguang commented on GitHub (Jun 22, 2024):

There are some config options in the [git] section of app.ini (eg: MAX_GIT_DIFF_LINES, MAX_GIT_DIFF_FILES), I am not sure whether reducing these numbers could make backend respond more quickly.

@wxiaoguang commented on GitHub (Jun 22, 2024): There are some config options in the `[git]` section of app.ini (eg: `MAX_GIT_DIFF_LINES`, `MAX_GIT_DIFF_FILES`), I am not sure whether reducing these numbers could make backend respond more quickly.

GiteaMirror commented