mirror of
https://github.com/go-gitea/gitea.git
synced 2026-03-12 02:24:21 -05:00
Linking issue dependencies fails (API returns 500 error) #8810
Closed
opened 2025-11-02 08:19:26 -06:00 by GiteaMirror
·
34 comments
No Branch/Tag Specified
main
release/v1.25
release/v1.24
release/v1.23
release/v1.22
release/v1.21
release/v1.20
release/v1.19
release/v1.18
release/v1.17
release/v1.16
release/v1.15
release/v1.14
release/v1.13
release/v1.12
release/v1.11
release/v1.10
release/v1.9
release/v1.8
v1.25.3
v1.25.2
v1.25.1
v1.25.0
v1.24.7
v1.25.0-rc0
v1.26.0-dev
v1.24.6
v1.24.5
v1.24.4
v1.24.3
v1.24.2
v1.24.1
v1.24.0
v1.23.8
v1.24.0-rc0
v1.25.0-dev
v1.23.7
v1.23.6
v1.23.5
v1.23.4
v1.23.3
v1.23.2
v1.23.1
v1.23.0
v1.23.0-rc0
v1.24.0-dev
v1.22.6
v1.22.5
v1.22.4
v1.22.3
v1.22.2
v1.22.1
v1.22.0
v1.23.0-dev
v1.22.0-rc1
v1.21.11
v1.22.0-rc0
v1.21.10
v1.21.9
v1.21.8
v1.21.7
v1.21.6
v1.21.5
v1.21.4
v1.21.3
v1.21.2
v1.20.6
v1.21.1
v1.21.0
v1.21.0-rc2
v1.21.0-rc1
v1.20.5
v1.22.0-dev
v1.21.0-rc0
v1.20.4
v1.20.3
v1.20.2
v1.20.1
v1.20.0
v1.19.4
v1.21.0-dev
v1.20.0-rc2
v1.20.0-rc1
v1.20.0-rc0
v1.19.3
v1.19.2
v1.19.1
v1.19.0
v1.19.0-rc1
v1.20.0-dev
v1.19.0-rc0
v1.18.5
v1.18.4
v1.18.3
v1.18.2
v1.18.1
v1.18.0
v1.17.4
v1.18.0-rc1
v1.19.0-dev
v1.18.0-rc0
v1.17.3
v1.17.2
v1.17.1
v1.17.0
v1.17.0-rc2
v1.16.9
v1.17.0-rc1
v1.18.0-dev
v1.16.8
v1.16.7
v1.16.6
v1.16.5
v1.16.4
v1.16.3
v1.16.2
v1.16.1
v1.16.0
v1.15.11
v1.17.0-dev
v1.16.0-rc1
v1.15.10
v1.15.9
v1.15.8
v1.15.7
v1.15.6
v1.15.5
v1.15.4
v1.15.3
v1.15.2
v1.15.1
v1.14.7
v1.15.0
v1.15.0-rc3
v1.14.6
v1.15.0-rc2
v1.14.5
v1.16.0-dev
v1.15.0-rc1
v1.14.4
v1.14.3
v1.14.2
v1.14.1
v1.14.0
v1.13.7
v1.14.0-rc2
v1.13.6
v1.13.5
v1.14.0-rc1
v1.15.0-dev
v1.13.4
v1.13.3
v1.13.2
v1.13.1
v1.13.0
v1.12.6
v1.13.0-rc2
v1.14.0-dev
v1.13.0-rc1
v1.12.5
v1.12.4
v1.12.3
v1.12.2
v1.12.1
v1.11.8
v1.12.0
v1.11.7
v1.12.0-rc2
v1.11.6
v1.12.0-rc1
v1.13.0-dev
v1.11.5
v1.11.4
v1.11.3
v1.10.6
v1.12.0-dev
v1.11.2
v1.10.5
v1.11.1
v1.10.4
v1.11.0
v1.11.0-rc2
v1.10.3
v1.11.0-rc1
v1.10.2
v1.10.1
v1.10.0
v1.9.6
v1.9.5
v1.10.0-rc2
v1.11.0-dev
v1.10.0-rc1
v1.9.4
v1.9.3
v1.9.2
v1.9.1
v1.9.0
v1.9.0-rc2
v1.10.0-dev
v1.9.0-rc1
v1.8.3
v1.8.2
v1.8.1
v1.8.0
v1.8.0-rc3
v1.7.6
v1.8.0-rc2
v1.7.5
v1.8.0-rc1
v1.9.0-dev
v1.7.4
v1.7.3
v1.7.2
v1.7.1
v1.7.0
v1.7.0-rc3
v1.6.4
v1.7.0-rc2
v1.6.3
v1.7.0-rc1
v1.7.0-dev
v1.6.2
v1.6.1
v1.6.0
v1.6.0-rc2
v1.5.3
v1.6.0-rc1
v1.6.0-dev
v1.5.2
v1.5.1
v1.5.0
v1.5.0-rc2
v1.5.0-rc1
v1.5.0-dev
v1.4.3
v1.4.2
v1.4.1
v1.4.0
v1.4.0-rc3
v1.4.0-rc2
v1.3.3
v1.4.0-rc1
v1.3.2
v1.3.1
v1.3.0
v1.3.0-rc2
v1.3.0-rc1
v1.2.3
v1.2.2
v1.2.1
v1.2.0
v1.2.0-rc3
v1.2.0-rc2
v1.1.4
v1.2.0-rc1
v1.1.3
v1.1.2
v1.1.1
v1.1.0
v1.0.2
v1.0.1
v1.0.0
v0.9.99
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
Mirrored from GitHub Pull Request
Milestone
No items
No Milestone
Projects
Clear projects
No project
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/gitea#8810
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @fnetX on GitHub (Apr 10, 2022).
Originally assigned to: @6543 on GitHub.
Description
Okay, we're really lost in debugging this for hours now on Codeberg. A live meeting with 5 people cannot figure out what's exactly going on. Let's start from the beginning:
On Codeberg.org, you can no longer link issue dependencies. The API call returns an error 500 with the error message in the gitea.log:
Now of course we thought of a recent regression and reverted to pre-1.16.5: still happening. We tried to reproduce on try.gitea.io and on codeberg-test: not happening.
We tried moving a whole database snapshot into our air gapped staging instance: Not happening, not reproducible, everything fine it seems (debugging there is a bit complicated, but we tried
and got many results, whereas on Codeberg we still get 500.
Accepting that we can probably only approach the issue on our real production server, @Gusted worked out some patches for us to get more information.
Printing the generated SQL query for the function by XORM, and executing this ourselves in MySQL, worked fine and returned the correct information (about 9000 in our case).
We then checked the length of the return value of models.Issues here to see if it's an issue with countIssues or if it's an underlying issue of the options query. And the return value here was zero (0), so it indicates an issue with the query options that was being passed down.
This indicates that the query is correct, but when XORM is executing it, it will not return anything.
So this seems to be a scaling issue to us, maybe because this passes the id's of all the public repos + the further issues a user has access to, which is, 17000+ on our instance. (Maybe a point for further optimization, e.g. extend the query to allow all repos with is_private = false + explicitly private repos a user has access to?)
Also very relevant according to blame: #19244, and especially this comment, as we can't explain why this would ever return something else than 1 row from the `SELECT COUNT`` call.
Please note that our tested version includes that pull request, because @6543 pushed a cherry-pick to our branch. Of course we also tried reverting it, but - still broken.
That's where we currently are. The SQL log contains many question marks, about 17k, for each repository. That's about how many question marks we see when thinking about how to further approach this. We appreciate any help. Thank you!
Gitea Version
99e133b593Can you reproduce the bug on the Gitea demo site?
No
Log Gist
No response
Screenshots
No response
Git Version
git version 2.30.2
Operating System
Debian GNU/Linux 11
How are you running Gitea?
custom fork / deployment, see commit ref above
Database
MySQL
@wxiaoguang commented on GitHub (Apr 11, 2022):
I think it's still a bug, maybe caused byGROUP BY, or some aggr SQL else.SELECT COUNT FROM ...=> always 1 rowsSELECT COUNT FROM ... GROUP BY ...=> maybe 0 or many rows.The key problem is that if it is the case, then the result of the oldCountIssues(row[0]) is still incorrect and need to be fiexed ......updated by comment below: the SQL only returns one row, indeed.
@wxiaoguang commented on GitHub (Apr 11, 2022):
Can you share the SQL for
CountIssues(with the patch from https://github.com/go-gitea/gitea/pull/19244) ?ps: I think there might be something incorrect for the comment and the COUNT SQL. The SQL error says
row count=0, then it's impossible for the real SQL from log to get 9000 rows .... can you double check that the SQL you tried is aCOUNTSQL?@fnetX commented on GitHub (Apr 11, 2022):
Please see http://paste.debian.net/hidden/73165403/ for the SQL query we ran (manually interpolating the values).
Please ignore the line wraps, I just added them for easier readability. The
use gitea;was also added by me.Running it results in
so I'm sorry if I was unclear before: Of course it returned one row with the value of about 9k (or 10k actually), and not that many rows.
@wxiaoguang commented on GitHub (Apr 11, 2022):
The SQL looks correct. And the result is correct too.
Since everything seems correct, I can not understand how the
row count=0failure occurs, either ........I suspect there are something missing or inconsistent. And according to the comment "We tried moving a whole database snapshot into our air gapped staging instance: Not happening, not reproducible, everything fine it seems", there must be something different.
Another question is that is the code (CountIssues related) exactly the same between Codeberg Gitea and Official Gitea?
@fnetX commented on GitHub (Apr 11, 2022):
I can't see any difference. Codeberg commits are always prefixed with
CB/. The blame shows that nothing is touching these functions:https://codeberg.org/Codeberg/gitea/blame/branch/codeberg-1.16/models/issue.go
Since we can't reproduce in our staging instance with the Codeberg binary yet, we don't really have a way to hunt this down with the upstream code. We can't just deploy the plain Gitea on production I fear ...
@wxiaoguang commented on GitHub (Apr 11, 2022):
I have an idea about how to debug the problem.
Maybe we can build Codeberg-Gitea with a special flag (eg:
./gitea-codeberg my-test-issue), and use the special flag to runSearchIssues/CountIssuesdirectly with the specialized options to trigger the bug and see output. Then we just put this test binary to production server, then test with the production database without any downtime.If the special flag can trigger the bug, then it gives us the chance to find the root case.
If the special flag can not trigger the bug either ..... it must be impossible, no reason to explain it .....
@fnetX commented on GitHub (Apr 11, 2022):
I like this idea, in fact I had a similar idea about trying to produce a minimal reproducible thing (e.g. a fresh binary with only the XORM call).
I think we can do this, provided that the binary does not modify any data or execute stuff (we already considered spawning a second Gitea instance, but we'd have to make sure it doesn't e.g. trigger Matrix hooks, run queues etc).
@wxiaoguang commented on GitHub (Apr 11, 2022):
https://stackoverflow.com/questions/32973835/mysql-sometimes-erroneously-returns-0-for-count
And someone said that it could also be a mysql bug 😂 (although I don't want to suspect that it's mysql's problem, and that post is quite old, the described bug is not the same)
@wxiaoguang commented on GitHub (Apr 11, 2022):
IIRC there are some Gitea sub commands like
adminordoctor, it only do some database work.@luwol03 commented on GitHub (Apr 11, 2022):
You may can also add a db user with read-only permissions to be sure nothing is changed?
@fnetX commented on GitHub (Apr 11, 2022):
Notable in the stackoverflow thread is that count() can return 0 when nothing matches. I wasn't aware of that, but we should keep that in mind for further debugging.
Yeah, but I also feared that it would read some queue and see it would need to fire some webhooks etc. Using a subcommand that only runs the affected query appears to be safe, though (not starting the normal Gitea app itself).
@lunny commented on GitHub (Apr 12, 2022):
The SQL is too long. What's the 500 error log?
@fnetX commented on GitHub (Apr 12, 2022):
The error log line that appears on the 500 error is the one from the initial comment:
@lunny commented on GitHub (Apr 12, 2022):
So a
Select count(*)SQL with no error but return 0 records. It's wired. https://stackoverflow.com/questions/2552086/does-count-always-return-a-resultI send a PR #19380 to simplify the code to avoid other problem. This PR will guarantee to return 0 even if it's wrong.
@fnetX commented on GitHub (Apr 14, 2022):
Continuing with debugging, we found out (in chronological order)
(1:10.5.15-0+deb11u1)&(1:10.5.12-0+deb11u1), the update wasn't yet applied in the airgapped staging instanceEdit 2: The error we see from MySQL is
Commands out of sync; you can't run this command nowEdit 3:
@wxiaoguang commented on GitHub (Apr 14, 2022):
Awesome! @lunny that's why I disagree to hide the error.
@fnetX Do you know which version of MariaDB has such bug while which doesn't? Is there any bug report on MariaDB side?
@fnetX commented on GitHub (Apr 14, 2022):
see Edit above for the error. We tried with 10.5.13, still working. 10.6.7 is broken, too. We'll continue investigating.
@wxiaoguang commented on GitHub (Apr 14, 2022):
OMG, I have a MariaDB 10.6.7 with 2T data in production 😭 I think I will take a look at this problem too .....
@fnetX commented on GitHub (Apr 14, 2022):
It's probably a very edge case, though. And it's not breaking any data at least, only the read query. :)
@ashimokawa commented on GitHub (Apr 14, 2022):
We did more test, even simple queries fail with more that 1000 placeholders since mariadb 10.5.15 or 14 (which we did not test)
@lunny commented on GitHub (Apr 14, 2022):
For a keyword search, repository id placeholder cannot be avoid. Maybe we need to limit it and give a warning in search result page that only display max less 1000 results.
@wxiaoguang commented on GitHub (Apr 14, 2022):
Unable to reproduce from my side(PHP PDO has emulated prepared statements ....)@ashimokawa commented on GitHub (Apr 14, 2022):
@wxiaoguang
Weird, we can.
It gets even more crazy. Preparing the statement fails with > 65535 parameters with a proper error "too many placeholders", but between 1000-65535 even executing works but fetching results fail.
@wxiaoguang commented on GitHub (Apr 14, 2022):
OK, I can reproduce it now with pure SQL. It's really a serious bug .........
I think it's worth to report it to MariaDB issue tracker .....
MariaDB: Server version: 10.6.7-MariaDB-log MariaDB Server
100
1001
@ashimokawa commented on GitHub (Apr 14, 2022):
@wxiaoguang
Not when using mysqli instead of PDO which I did for my tests. They fail as your pure SQL test does.
Also it is strange that it is a recent regression in mariadb since 10.5.13 is still okay and only the recently released 10.5.15 is broken. I do consider this a mariadb bug since there is a proper error when really hitting the placeholder limit (>65535).
@wxiaoguang commented on GitHub (Apr 14, 2022):
I opened an issue for MairaDB: https://jira.mariadb.org/browse/MDEV-28316
@lunny commented on GitHub (Apr 14, 2022):
Amazing.
@ashimokawa commented on GitHub (Apr 14, 2022):
Already fixed upstream, but not released yet.
https://jira.mariadb.org/browse/MDEV-27937
@fnetX commented on GitHub (Apr 14, 2022):
Still, as the next (default) limit will be 65535 parameters, I think this query should be optimized before we'll be hitting that with more growth.
Doesn't this mean that large instance just won't have working issue dependency linking?
Where do the repo id's come from? All public repos, or all repos where a matching issue was found? Because when searching for some title, it's kinda weird to see soo many results for a simple title query anyway. I'd want to mainly see repos I'm associated with, and only then more results, probably.
I get right that the query does not simply provide all public repos, right?
@Gusted commented on GitHub (Apr 14, 2022):
What if we modify the code a bit to avoid passing all the repoIDs:
@wxiaoguang commented on GitHub (Apr 15, 2022):
Maybe codeberg is one of the largest one, and usually there shouldn't be so many repo IDs
It has Pros and Cons.
I believe knowing the logic fully is the precondition to make optimizations (as the question "where do the repo id's come from"). Maybe it could be optimized from the logic side fundamentally.
And, to handle all similar problems gracefully in future, some improvements:
Countreturn an error if the result is 0 row.@ashimokawa commented on GitHub (Apr 16, 2022):
When having more than 65535 parameter there will be a proper error while preparing the statement.
The current behaviour between 1000 and 65535 where 0 rows are returned silently is a bug with unexpected behaviour.
It is different from having just max placeholders down to 999.
@6543 commented on GitHub (Apr 20, 2022):
we have to use xorm builder so we create join querys and not
In 1....100kqueryswhere go code insert the constrains
@6543 commented on GitHub (Apr 24, 2022):
-> #19475