Commenting on a diff of a file with spaces separating short tokens makes the PR page inaccessible #6931

Closed
opened 2025-11-02 07:11:23 -06:00 by GiteaMirror · 23 comments
Owner

Originally created by @abread on GitHub (Feb 26, 2021).

Rendering the PR page errors with template: repo/issue/view_content/comments:470:20: executing "repo/issue/view_content/comments" at <CommentMustAsDiff (index $comms 0)>: error calling CommentMustAsDiff: runtime error: slice bounds out of range [2:1]

I was unable to get a proper stacktrace (or logs), but I manually traced the offending slice indexing to this line.

The git-diff output filename extractor (readFileName) cannot properly split the old name from the new name (and panics because instead of finding b/<filename> to remove b/ it finds the single letter bit of the old filename).

git-diff outputs unquoted paths because Gitea sets core.quotePath (git config) to false. Interestingly, readFileName has code handling quoted paths already. It did work too when I changed the offending patch in the comments table to have quoted paths.

Steps to reproduce:

  1. Open a PR creating a file whose name is something like sdafdsaf z 1 - the first document.txt (anything in the format<any amount of characters that are not a space> <1 non-space character>[ <anything>] works as far as I can tell)
  2. Comment on any line of this new file in the PR
  3. The conversation tab of this PR now gives a 500 error when accessed

Fix suggestion

Maybe there's a really good reason to keep it, but imho leaving all paths returned by git quoted (core.quotePath=true) is the best option for sane path handling everywhere.

It may already be ok to do so: setting it to true didn't break any tests on my machine at least.

  • Gitea version (or commit ref): 1.13.2
  • Git version: 2.26.2
  • Operating system: Gentoo
    • using official ebuild with USE="acct build-client filecaps pam -sqlite"
    • started by OpenRC
  • Database (use [x]):
    • PostgreSQL
    • MySQL
    • MSSQL
    • SQLite (also tested)
  • Can you reproduce the bug at https://try.gitea.io:
  • Log gist: no logs appear (not even with Trace everything)
Originally created by @abread on GitHub (Feb 26, 2021). Rendering the PR page errors with `template: repo/issue/view_content/comments:470:20: executing "repo/issue/view_content/comments" at <CommentMustAsDiff (index $comms 0)>: error calling CommentMustAsDiff: runtime error: slice bounds out of range [2:1]` I was unable to get a proper stacktrace (or logs), but I manually traced the offending slice indexing to [this line](https://github.com/go-gitea/gitea/blob/014313134f72ebb2ca57eb2e5cd42726231f8fff/services/gitdiff/gitdiff.go#L867). The git-diff output filename extractor (`readFileName`) cannot properly split the old name from the new name (and panics because instead of finding `b/<filename>` to remove `b/` it finds the single letter bit of the old filename). git-diff outputs **unquoted** paths because Gitea [sets `core.quotePath` (git config) to `false`](https://github.com/go-gitea/gitea/blob/50208e903ae9aa931db3d93bde7032e766687cae/modules/git/git.go#L150). Interestingly, `readFileName` has code handling quoted paths already. It did work too when I changed the offending patch in the comments table to have quoted paths. ## Steps to reproduce: 1. Open a PR creating a file whose name is something like `sdafdsaf z` `1 - the first document.txt` (anything in the format`<any amount of characters that are not a space> <1 non-space character>[ <anything>]` works as far as I can tell) 2. Comment on any line of this new file in the PR 3. The conversation tab of this PR now gives a 500 error when accessed ## Fix suggestion Maybe there's a really good reason to keep it, but imho leaving all paths returned by git quoted (`core.quotePath=true`) is the best option for sane path handling everywhere. It **may** already be ok to do so: setting it to true didn't break any tests on my machine at least. - Gitea version (or commit ref): 1.13.2 - Git version: 2.26.2 - Operating system: Gentoo * using official ebuild with USE="acct build-client filecaps pam -sqlite" * started by OpenRC - Database (use `[x]`): - [ ] PostgreSQL - [x] MySQL - [ ] MSSQL - [x] SQLite (also tested) - Can you reproduce the bug at https://try.gitea.io: - [x] Yes [example](https://try.gitea.io/templatepanic/templatepanic/pulls/3) - [ ] No - Log gist: no logs appear (not even with Trace everything)
Author
Owner

@zeripath commented on GitHub (Feb 26, 2021):

I've said it before and I'll say it again I hate our diff infrastructure.

@zeripath commented on GitHub (Feb 26, 2021): I've said it before and I'll say it again I hate our diff infrastructure.
Author
Owner

@zeripath commented on GitHub (Feb 26, 2021):

OK so this is not rendering the diff pages themselves but rather a problem with rendering a comment on the pull itself right?

@zeripath commented on GitHub (Feb 26, 2021): OK so this is not rendering the diff pages themselves but rather a problem with rendering a comment on the pull itself right?
Author
Owner

@abread commented on GitHub (Feb 26, 2021):

I think so yes (at least that's what I gather from the error message)

@abread commented on GitHub (Feb 26, 2021): I think so yes (at least that's what I gather from the error message)
Author
Owner

@zeripath commented on GitHub (Feb 26, 2021):

yup - https://try.gitea.io/templatepanic/templatepanic/pulls/3/files this doesn't panic.

@zeripath commented on GitHub (Feb 26, 2021): yup - https://try.gitea.io/templatepanic/templatepanic/pulls/3/files this doesn't panic.
Author
Owner

@zeripath commented on GitHub (Feb 26, 2021):

OK so this is not a failure with the code I spent so effing long on making work for rendering of diffs in the first place.

@zeripath commented on GitHub (Feb 26, 2021): OK so this is not a failure with the code I spent so effing long on making work for rendering of diffs in the first place.
Author
Owner

@zeripath commented on GitHub (Feb 26, 2021):

No doubt the problem will be similar - in the end the reality is that there is no way to reliably split up: diff a/filename b/filename2 without forcing it to be always quoted.

@zeripath commented on GitHub (Feb 26, 2021): No doubt the problem will be similar - in the end the reality is that there is no way to reliably split up: diff a/filename b/filename2 without forcing it to be always quoted.
Author
Owner

@zeripath commented on GitHub (Feb 27, 2021):

Just to highlight this issue here is a particularly catastrophic example:

diff --git a/b b/b b/b b/b
deleted file mode 100644
index 92e798b..0000000
--- a/b b/b	
+++ /dev/null
@@ -1 +0,0 @@
-b b/b
diff --git a/b b/b b/b b/b b/b b/b b/b b/b
new file mode 100644
index 0000000..1bc3698
--- /dev/null
+++ b/b b/b b/b b/b	
@@ -0,0 +1 @@
+b b/b is now b b/b b/b b/b
diff --git a/b b/b b/b b/b b/b b/b
similarity index 100%
rename from b
rename to b b/b b/b b/b b/b
diff --git a/regular_text_file.rb b/regular_text_file.rb
new file mode 100644
index 0000000..6aa8f4e
--- /dev/null
+++ b/regular_text_file.rb
@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+
+puts 'Foobar!'

from the diff --git lines alone what are the new and old filenames...

@zeripath commented on GitHub (Feb 27, 2021): Just to highlight this issue here is a particularly catastrophic example: ```diff diff --git a/b b/b b/b b/b deleted file mode 100644 index 92e798b..0000000 --- a/b b/b +++ /dev/null @@ -1 +0,0 @@ -b b/b diff --git a/b b/b b/b b/b b/b b/b b/b b/b new file mode 100644 index 0000000..1bc3698 --- /dev/null +++ b/b b/b b/b b/b @@ -0,0 +1 @@ +b b/b is now b b/b b/b b/b diff --git a/b b/b b/b b/b b/b b/b similarity index 100% rename from b rename to b b/b b/b b/b b/b diff --git a/regular_text_file.rb b/regular_text_file.rb new file mode 100644 index 0000000..6aa8f4e --- /dev/null +++ b/regular_text_file.rb @@ -0,0 +1,3 @@ +# frozen_string_literal: true + +puts 'Foobar!' ``` from the `diff --git` lines alone what are the new and old filenames...
Author
Owner

@zeripath commented on GitHub (Feb 27, 2021):

oh yeah and this particular doozy:

diff --git a/b b/b b/b b/b b/b b/b
similarity index 100%
rename from b b/b b/b b/b b/b
rename to b
diff --git a/b b/b b/b b/b b/b b/b
similarity index 100%
rename from b b/b b/b b/b
rename to b b/b
@zeripath commented on GitHub (Feb 27, 2021): oh yeah and this particular doozy: ```diff diff --git a/b b/b b/b b/b b/b b/b similarity index 100% rename from b b/b b/b b/b b/b rename to b diff --git a/b b/b b/b b/b b/b b/b similarity index 100% rename from b b/b b/b b/b rename to b b/b ```
Author
Owner

@abread commented on GitHub (Feb 27, 2021):

Yesterday I bit the bullet and looked through all the code calling git (started by looking for everything using os/exec and went up from there). I found only 43 places that are not ready to handle quoted paths (1 looks like dead code).

I think I can get a PR fixing this (and potentially other issues with paths) that flips core.quotedPath to true done today

@abread commented on GitHub (Feb 27, 2021): Yesterday I bit the bullet and looked through all the code calling git (started by looking for everything using `os/exec` and went up from there). I found only ~~4~~3 places that are not ready to handle quoted paths (1 looks like dead code). I think I can get a PR fixing this (and potentially other issues with paths) that flips `core.quotedPath` to `true` done today
Author
Owner

@zeripath commented on GitHub (Feb 27, 2021):

No don't worry I'm there already

@zeripath commented on GitHub (Feb 27, 2021): No don't worry I'm there already
Author
Owner

@zeripath commented on GitHub (Feb 27, 2021):

Just thinking about fixing the comment as patch code to make it quote or not but I guess that needs to be done as a different PR.

@zeripath commented on GitHub (Feb 27, 2021): Just thinking about fixing the comment as patch code to make it quote or not but I guess that needs to be done as a different PR.
Author
Owner

@zeripath commented on GitHub (Feb 27, 2021):

Hmm. I think I'm gonna merge the fix into #14804

@zeripath commented on GitHub (Feb 27, 2021): Hmm. I think I'm gonna merge the fix into #14804
Author
Owner

@abread commented on GitHub (Feb 27, 2021):

I think it's easier to just assume paths are quoted in ParsePatch and change the ~3 callers to call git -c core.quotedpath=true ... no?

@abread commented on GitHub (Feb 27, 2021): I think it's easier to just assume paths are quoted in `ParsePatch` and change the ~3 callers to call `git -c core.quotedpath=true ...` no?
Author
Owner

@zeripath commented on GitHub (Feb 27, 2021):

ah you've made the mistake of assuming that these particular "diffs" came from git - they're from some code that hacks them out of git diffs.

The reality is that these diffs are stored in plaintext in your db so we still need to be able to parse them.

In any case core.quotedpath doesn't guarantee what you think it does. The answer for forcing every git from 1.7.2+ to quote its paths is to pass git diff --src-prefix=\\a/ --dst-prefix=\\b/ and then trim off the \\.

Honestly it's just crazy.

@zeripath commented on GitHub (Feb 27, 2021): ah you've made the mistake of assuming that these particular "diffs" came from git - they're from some code that hacks them out of git diffs. The reality is that these diffs are stored in plaintext in your db so we still need to be able to parse them. In any case core.quotedpath doesn't guarantee what you think it does. The answer for forcing every git from 1.7.2+ to quote its paths is to pass `git diff --src-prefix=\\a/ --dst-prefix=\\b/` and then trim off the `\\`. Honestly it's just crazy.
Author
Owner

@abread commented on GitHub (Feb 27, 2021):

Ah right I was forgetting those

At one point I had the idea of gitea doctoring them into sane quotedness, but now that I think about it the only reason they're in the DB is because git gc could throw away the originals right?

On February 27, 2021 1:30:42 PM UTC, zeripath notifications@github.com wrote:

ah you've made the mistake of assuming that these particular "diffs" came from git - they're from some code that hacks them out of git diffs.

The reality is that these diffs are stored in plaintext in your db so we still need to be able to parse them.

In any case core.quotedpath doesn't guarantee what you think it does. The answer for forcing every git from 1.7.2+ to quote its paths is to pass git diff --src-prefix=\\a/ --dst-prefix=\\b/ and then trim off the \\.

Honestly it's just crazy.

--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/go-gitea/gitea/issues/14812#issuecomment-787073079

@abread commented on GitHub (Feb 27, 2021): Ah right I was forgetting those At one point I had the idea of `gitea doctor`ing them into sane quotedness, but now that I think about it the only reason they're in the DB is because `git gc` could throw away the originals right? On February 27, 2021 1:30:42 PM UTC, zeripath <notifications@github.com> wrote: >ah you've made the mistake of assuming that these particular "diffs" came from git - they're from some code that hacks them out of git diffs. > >The reality is that these diffs are stored in plaintext in your db so we still need to be able to parse them. > >In any case core.quotedpath doesn't guarantee what you think it does. The answer for forcing every git from 1.7.2+ to quote its paths is to pass `git diff --src-prefix=\\a/ --dst-prefix=\\b/` and then trim off the `\\`. > >Honestly it's just crazy. > >-- >You are receiving this because you authored the thread. >Reply to this email directly or view it on GitHub: >https://github.com/go-gitea/gitea/issues/14812#issuecomment-787073079
Author
Owner

@skillcoder commented on GitHub (Mar 18, 2021):

I'm updatated gitea 1.13.2 -> 1.13.4
But I still have see broken PR

template: repo/issue/view_content/comments:470:20: executing "repo/issue/view_content/comments" at <CommentMustAsDiff (index $comms 0)>: error calling CommentMustAsDiff: runtime error: invalid memory address or nil pointer dereference

It's normal ?

@skillcoder commented on GitHub (Mar 18, 2021): I'm updatated gitea 1.13.2 -> 1.13.4 But I still have see broken PR ``` template: repo/issue/view_content/comments:470:20: executing "repo/issue/view_content/comments" at <CommentMustAsDiff (index $comms 0)>: error calling CommentMustAsDiff: runtime error: invalid memory address or nil pointer dereference ``` It's normal ?
Author
Owner

@miili commented on GitHub (Mar 18, 2021):

Same here. Is there any way to fix this?
Can we fix the bad PR or throw it away?

@miili commented on GitHub (Mar 18, 2021): Same here. Is there any way to fix this? Can we fix the bad PR or throw it away?
Author
Owner

@zeripath commented on GitHub (Mar 18, 2021):

Please open a new issue instead of commenting on old issues and replicate the issue on try.gitea.io, and give us some logs.

Please refer to this issue on the new issue so we can go from there.

@zeripath commented on GitHub (Mar 18, 2021): Please open a new issue instead of commenting on old issues and replicate the issue on try.gitea.io, and give us some logs. Please refer to this issue on the new issue so we can go from there.
Author
Owner

@zeripath commented on GitHub (Mar 18, 2021):

@miili without even giving us an example of what is breaking how can you expect there to be a fix? The provided testcase referred to in the in this issue is fixed and frankly it's extremely unhelpful of you to write things like: "Can we fix the bad PR or throw it away."

@zeripath commented on GitHub (Mar 18, 2021): @miili without even giving us an example of what is breaking how can you expect there to be a fix? The provided testcase referred to in the in this issue is fixed and frankly it's extremely unhelpful of you to write things like: "Can we fix the bad PR or throw it away."
Author
Owner

@miili commented on GitHub (Mar 19, 2021):

@miili without even giving us an example of what is breaking how can you expect there to be a fix? The provided testcase referred to in the in this issue is fixed and frankly it's extremely unhelpful of you to write things like: "Can we fix the bad PR or throw it away."

@zeripath, thank you for you respone. The exact issue and how to replicate it has been described in great detail in this issue. I don't feel the need to document or replicate it again.

template: repo/issue/view_content/comments:470:20: executing "repo/issue/view_content/comments" at <CommentMustAsDiff (index $comms 0)>: error calling CommentMustAsDiff: runtime error: invalid memory address or nil pointer dereference

The bad PR is refering to the broken Pull Request on the production system, which cannot be accessed anymore.
Here is a link, if that helps https://git.pyrocko.org/pyrocko/pyrocko/pulls/159#issuecomment-785

Can the broken PR be fixed?

@miili commented on GitHub (Mar 19, 2021): > @miili without even giving us an example of what is breaking how can you expect there to be a fix? The provided testcase referred to in the in this issue is fixed and frankly it's extremely unhelpful of you to write things like: "Can we fix the bad PR or throw it away." @zeripath, thank you for you respone. The exact issue and how to replicate it has been described in great detail in this issue. I don't feel the need to document or replicate it again. ``` template: repo/issue/view_content/comments:470:20: executing "repo/issue/view_content/comments" at <CommentMustAsDiff (index $comms 0)>: error calling CommentMustAsDiff: runtime error: invalid memory address or nil pointer dereference ``` The _bad PR_ is refering to the broken Pull Request on the production system, which cannot be accessed anymore. Here is a link, if that helps https://git.pyrocko.org/pyrocko/pyrocko/pulls/159#issuecomment-785 Can the broken PR be fixed?
Author
Owner

@zeripath commented on GitHub (Mar 24, 2021):

@miili could you update to 1.13.6 please we should get more information then

@zeripath commented on GitHub (Mar 24, 2021): @miili could you update to 1.13.6 please we should get more information then
Author
Owner

@zeripath commented on GitHub (Mar 24, 2021):

Also looking at:

https://git.pyrocko.org/pyrocko/pyrocko/pulls/159/files

I can't see an obvious thing that would cause the problem but I wonder
if you'd be able to help me out to figure out what the problem is or
could be.

Part of the problem is that the panic is occurring inside the template
executer so we're lacking more information as to exactly where the
panic is occuring. It would be possible to add a recover handler into
the function to help get information into why the result is failing
but I can't seem to replicate this locally at all.

If you would be able to give me a copy of your issue, pull_request and
comment tables as relating to this request I might be able to figure
it out.

1.13.6 has changed the code to give us more information about the panic but it would still be helpful to figure things out

@zeripath commented on GitHub (Mar 24, 2021): Also looking at: https://git.pyrocko.org/pyrocko/pyrocko/pulls/159/files I can't see an obvious thing that would cause the problem but I wonder if you'd be able to help me out to figure out what the problem is or could be. Part of the problem is that the panic is occurring inside the template executer so we're lacking more information as to exactly where the panic is occuring. It would be possible to add a recover handler into the function to help get information into why the result is failing but I can't seem to replicate this locally at all. If you would be able to give me a copy of your issue, pull_request and comment tables as relating to this request I might be able to figure it out. 1.13.6 has changed the code to give us more information about the panic but it would still be helpful to figure things out
Author
Owner

@zeripath commented on GitHub (Mar 29, 2021):

@miili @skillcoder Are either of you associated with pyrocko? If so the upgrade to 1.13.6 has revealed the source of the problem.

As I suspected it was not associated to this PR.

@zeripath commented on GitHub (Mar 29, 2021): @miili @skillcoder Are either of you associated with pyrocko? If so the upgrade to 1.13.6 has revealed the source of the problem. As I suspected it was not associated to this PR.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#6931