Panic when submitting PR due to bleve index error #6905

Closed
opened 2025-11-02 07:10:43 -06:00 by GiteaMirror · 7 comments
Owner

Originally created by @ldstein on GitHub (Feb 20, 2021).

  • Gitea version (or commit ref): 1.13.2 from official downloads

  • Git version: 2.18.0.windows.1

  • Operating system:
    Windows Server v6.3 (Build 9600) AMD64 running on Amazon EC2

  • Database (use [x]):

    • PostgreSQL (Amazon RDS)
    • MySQL
    • MSSQL
    • SQLite
  • Can you reproduce the bug at https://try.gitea.io:

    • Yes (provide example URL)
    • No
  • Log gist:

[Macaron] 2021-02-20 05:06:17: Started GET /serviceworker.js for ::ffff:49.179.25.120
[Macaron] [Static] Serving /serviceworker.js
[Macaron] 2021-02-20 05:06:17: Completed GET /serviceworker.js 304 Not Modified in 987.4µs
[Macaron] 2021-02-20 05:06:18: Completed GET /user/events 200 OK in 2.9470408s
[Macaron] 2021-02-20 05:06:19: Started POST /ldstein/crash-test/compare/master...crash-test for ::ffff:49.179.25.120
2021/02/20 05:06:19 ...s/context/context.go:332:func1() [D] Session ID: c19f1d16465be952
2021/02/20 05:06:19 ...s/context/context.go:333:func1() [D] CSRF Token: ufiGMWDBujT4IT_i2Z-3MNkYiTA6MTYxMzc5NDU1MzI0MTQ1MDMwMA
2021/02/20 05:06:20 ...er/issues/indexer.go:305:UpdateIssueIndexer() [D] Adding to channel: &{353 851 Added test2  [] false []}
[Macaron] 2021-02-20 05:06:20: Completed POST /ldstein/crash-test/compare/master...crash-test 302 Found in 1.4809377s
panic: invalid page type: 31: 10

goroutine 3366 [running]:
code.gitea.io/gitea/vendor/go.etcd.io/bbolt.(*Cursor).search(0xc002dc9cf0, 0xc0095ba000, 0x3, 0x1000, 0x1f)
        /go/src/code.gitea.io/gitea/vendor/go.etcd.io/bbolt/cursor.go:250 +0x345
code.gitea.io/gitea/vendor/go.etcd.io/bbolt.(*Cursor).searchPage(0xc002dc9cf0, 0xc0095ba000, 0x3, 0x1000, 0x32b18000)
        /go/src/code.gitea.io/gitea/vendor/go.etcd.io/bbolt/cursor.go:308 +0x16c
code.gitea.io/gitea/vendor/go.etcd.io/bbolt.(*Cursor).search(0xc002dc9cf0, 0xc0095ba000, 0x3, 0x1000, 0x58)
        /go/src/code.gitea.io/gitea/vendor/go.etcd.io/bbolt/cursor.go:265 +0x194
code.gitea.io/gitea/vendor/go.etcd.io/bbolt.(*Cursor).searchPage(0xc002dc9cf0, 0xc0095ba000, 0x3, 0x1000, 0x32b79000)
        /go/src/code.gitea.io/gitea/vendor/go.etcd.io/bbolt/cursor.go:308 +0x16c
code.gitea.io/gitea/vendor/go.etcd.io/bbolt.(*Cursor).search(0xc002dc9cf0, 0xc0095ba000, 0x3, 0x1000, 0xb9)
        /go/src/code.gitea.io/gitea/vendor/go.etcd.io/bbolt/cursor.go:265 +0x194
code.gitea.io/gitea/vendor/go.etcd.io/bbolt.(*Cursor).seek(0xc0064c9cf0, 0xc0095ba000, 0x3, 0x1000, 0xc002dc9d40, 0x40f9d0, 0x320108, 0xc00953b857, 0xc0095ba000, 0x326f4ce0, ...)
        /go/src/code.gitea.io/gitea/vendor/go.etcd.io/bbolt/cursor.go:159 +0x85
code.gitea.io/gitea/vendor/go.etcd.io/bbolt.(*Bucket).Get(0xc00947a980, 0xc0095ba000, 0x3, 0x1000, 0xc002dc9d70, 0x451b78, 0x1000)
        /go/src/code.gitea.io/gitea/vendor/go.etcd.io/bbolt/bucket.go:262 +0xc5
code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/store/boltdb.(*Reader).Get(0xc0095b2a00, 0xc0095ba000, 0x3, 0x1000, 0x100000001, 0x10, 0x10, 0x32b80025, 0xc002dc9df0)
        /go/src/code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/store/boltdb/reader.go:30 +0x58
code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/upsidedown.backIndexRowForDoc(0x46753e0, 0xc0095b2a00, 0xc002dc9ec8, 0x2, 0x20, 0x0, 0x0, 0x0)
        /go/src/code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/upsidedown/upsidedown.go:1071 +0x1ba
code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/upsidedown.(*UpsideDownCouch).Batch.func2(0xc00955e6c0, 0xc00319d480, 0xc0095b4810, 0xc0095b4820, 0xc0095b2980)
        /go/src/code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/upsidedown/upsidedown.go:858 +0x285
created by code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/upsidedown.(*UpsideDownCouch).Batch
        /go/src/code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/upsidedown/upsidedown.go:841 +0x2e9

Description

Recently upgraded from Gitea 1.12.5 to 1.13.2 and run into an issue where submitting a Pull Request will consistently crash Gitea with the above panic.

Gitea is run as a process (by double-clicking the exe).

When Gitea is restarted after the crash, the PR is successfully created, but will crash again on subsequent PR submissions for all repos.

Any pointers as to where the issue might lie?

Originally created by @ldstein on GitHub (Feb 20, 2021). - Gitea version (or commit ref): 1.13.2 from official downloads - Git version: 2.18.0.windows.1 - Operating system: Windows Server v6.3 (Build 9600) AMD64 running on Amazon EC2 - Database (use `[x]`): - [X] PostgreSQL (Amazon RDS) - [ ] MySQL - [ ] MSSQL - [ ] SQLite - Can you reproduce the bug at https://try.gitea.io: - [ ] Yes (provide example URL) - [X] No - Log gist: ``` [Macaron] 2021-02-20 05:06:17: Started GET /serviceworker.js for ::ffff:49.179.25.120 [Macaron] [Static] Serving /serviceworker.js [Macaron] 2021-02-20 05:06:17: Completed GET /serviceworker.js 304 Not Modified in 987.4µs [Macaron] 2021-02-20 05:06:18: Completed GET /user/events 200 OK in 2.9470408s [Macaron] 2021-02-20 05:06:19: Started POST /ldstein/crash-test/compare/master...crash-test for ::ffff:49.179.25.120 2021/02/20 05:06:19 ...s/context/context.go:332:func1() [D] Session ID: c19f1d16465be952 2021/02/20 05:06:19 ...s/context/context.go:333:func1() [D] CSRF Token: ufiGMWDBujT4IT_i2Z-3MNkYiTA6MTYxMzc5NDU1MzI0MTQ1MDMwMA 2021/02/20 05:06:20 ...er/issues/indexer.go:305:UpdateIssueIndexer() [D] Adding to channel: &{353 851 Added test2 [] false []} [Macaron] 2021-02-20 05:06:20: Completed POST /ldstein/crash-test/compare/master...crash-test 302 Found in 1.4809377s panic: invalid page type: 31: 10 goroutine 3366 [running]: code.gitea.io/gitea/vendor/go.etcd.io/bbolt.(*Cursor).search(0xc002dc9cf0, 0xc0095ba000, 0x3, 0x1000, 0x1f) /go/src/code.gitea.io/gitea/vendor/go.etcd.io/bbolt/cursor.go:250 +0x345 code.gitea.io/gitea/vendor/go.etcd.io/bbolt.(*Cursor).searchPage(0xc002dc9cf0, 0xc0095ba000, 0x3, 0x1000, 0x32b18000) /go/src/code.gitea.io/gitea/vendor/go.etcd.io/bbolt/cursor.go:308 +0x16c code.gitea.io/gitea/vendor/go.etcd.io/bbolt.(*Cursor).search(0xc002dc9cf0, 0xc0095ba000, 0x3, 0x1000, 0x58) /go/src/code.gitea.io/gitea/vendor/go.etcd.io/bbolt/cursor.go:265 +0x194 code.gitea.io/gitea/vendor/go.etcd.io/bbolt.(*Cursor).searchPage(0xc002dc9cf0, 0xc0095ba000, 0x3, 0x1000, 0x32b79000) /go/src/code.gitea.io/gitea/vendor/go.etcd.io/bbolt/cursor.go:308 +0x16c code.gitea.io/gitea/vendor/go.etcd.io/bbolt.(*Cursor).search(0xc002dc9cf0, 0xc0095ba000, 0x3, 0x1000, 0xb9) /go/src/code.gitea.io/gitea/vendor/go.etcd.io/bbolt/cursor.go:265 +0x194 code.gitea.io/gitea/vendor/go.etcd.io/bbolt.(*Cursor).seek(0xc0064c9cf0, 0xc0095ba000, 0x3, 0x1000, 0xc002dc9d40, 0x40f9d0, 0x320108, 0xc00953b857, 0xc0095ba000, 0x326f4ce0, ...) /go/src/code.gitea.io/gitea/vendor/go.etcd.io/bbolt/cursor.go:159 +0x85 code.gitea.io/gitea/vendor/go.etcd.io/bbolt.(*Bucket).Get(0xc00947a980, 0xc0095ba000, 0x3, 0x1000, 0xc002dc9d70, 0x451b78, 0x1000) /go/src/code.gitea.io/gitea/vendor/go.etcd.io/bbolt/bucket.go:262 +0xc5 code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/store/boltdb.(*Reader).Get(0xc0095b2a00, 0xc0095ba000, 0x3, 0x1000, 0x100000001, 0x10, 0x10, 0x32b80025, 0xc002dc9df0) /go/src/code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/store/boltdb/reader.go:30 +0x58 code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/upsidedown.backIndexRowForDoc(0x46753e0, 0xc0095b2a00, 0xc002dc9ec8, 0x2, 0x20, 0x0, 0x0, 0x0) /go/src/code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/upsidedown/upsidedown.go:1071 +0x1ba code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/upsidedown.(*UpsideDownCouch).Batch.func2(0xc00955e6c0, 0xc00319d480, 0xc0095b4810, 0xc0095b4820, 0xc0095b2980) /go/src/code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/upsidedown/upsidedown.go:858 +0x285 created by code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/upsidedown.(*UpsideDownCouch).Batch /go/src/code.gitea.io/gitea/vendor/github.com/blevesearch/bleve/index/upsidedown/upsidedown.go:841 +0x2e9 ``` ## Description Recently upgraded from Gitea 1.12.5 to 1.13.2 and run into an issue where submitting a Pull Request will consistently crash Gitea with the above panic. Gitea is run as a process (by double-clicking the exe). When Gitea is restarted after the crash, the PR is successfully created, but will crash again on subsequent PR submissions for all repos. Any pointers as to where the issue might lie?
Author
Owner

@zeripath commented on GitHub (Feb 20, 2021):

The panic is coming from inside bleve - presumably related to issue indexing.

The likelihood is that the underlying bleve index file is broken in some way.

I suspect the only answer is going to be to delete bleve's index files here - it will regenerate them.

@lunny can you remember where the default place is these go? Or is there another thing that can be tried to fix the index?

@zeripath commented on GitHub (Feb 20, 2021): The panic is coming from inside bleve - presumably related to issue indexing. The likelihood is that the underlying bleve index file is broken in some way. I suspect the only answer is going to be to delete bleve's index files here - it will regenerate them. @lunny can you remember where the default place is these go? Or is there another thing that can be tried to fix the index?
Author
Owner

@ldstein commented on GitHub (Feb 20, 2021):

Thanks for the tip @zeripath. Emptying the contents of gitea/data/indexers solved the issue.

I don't suppose there is a way to validate the integrity of the bleve index?

@ldstein commented on GitHub (Feb 20, 2021): Thanks for the tip @zeripath. Emptying the contents of `gitea/data/indexers` solved the issue. I don't suppose there is a way to validate the integrity of the bleve index?
Author
Owner

@lunny commented on GitHub (Feb 20, 2021):

I think we should have a version file to keep the bleve index version. Once you upgrade from an old index, Gitea will automatically delete old indexers and restart the new index. I don't know if this feature is still work. Otherwise, the only way is to delete old indexers and restart Gitea like what @zeripath said.

@lunny commented on GitHub (Feb 20, 2021): I think we should have a version file to keep the bleve index version. Once you upgrade from an old index, Gitea will automatically delete old indexers and restart the new index. I don't know if this feature is still work. Otherwise, the only way is to delete old indexers and restart Gitea like what @zeripath said.
Author
Owner

@zeripath commented on GitHub (Feb 20, 2021):

@ldstein were there any log warnings earlier? Anything that could be used to indicate that a problem was incoming?

@zeripath commented on GitHub (Feb 20, 2021): @ldstein were there any log warnings earlier? Anything that could be used to indicate that a problem was incoming?
Author
Owner

@ldstein commented on GitHub (Feb 22, 2021):

Nothing which stood out (unfortunately I cleared the logs as part of our debugging process).

I also ran gitea doctor -all which also reported everything OK. Maybe doctor is a good place to check the indexer intgerity?

@ldstein commented on GitHub (Feb 22, 2021): Nothing which stood out (unfortunately I cleared the logs as part of our debugging process). I also ran `gitea doctor -all` which also reported everything OK. Maybe doctor is a good place to check the indexer intgerity?
Author
Owner

@lunny commented on GitHub (Feb 22, 2021):

@ldstein Hm, currently no check for indexer in doctor.

@lunny commented on GitHub (Feb 22, 2021): @ldstein Hm, currently no check for indexer in doctor.
Author
Owner

@ldstein commented on GitHub (Feb 22, 2021):

@lunny I know :)

Was just a suggestion. gitea doctor seemed like a good place to check for things that could be wrong.

@ldstein commented on GitHub (Feb 22, 2021): @lunny I know :) Was just a suggestion. `gitea doctor` seemed like a good place to check for things that could be wrong.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#6905