Repo indexer - Max file size #1501

@lafriks commented on GitHub (Feb 5, 2018):

MAX_FILE_SIZE is not for index size but for up to how big files to index

@lafriks commented on GitHub (Feb 5, 2018): `MAX_FILE_SIZE` is not for index size but for up to how big files to index

GiteaMirror commented

@ghost commented on GitHub (Feb 5, 2018):

Thank you for your fast answer @lafriks . Then, what about the fixed "store" file size? I am trying to indexing a big project, and some of the files was indexed, but seems to be that not everyone, and I supposed that it depends maybe from the "file size"

@ghost commented on GitHub (Feb 5, 2018): Thank you for your fast answer @lafriks . Then, what about the fixed "store" file size? I am trying to indexing a big project, and some of the files was indexed, but seems to be that not everyone, and I supposed that it depends maybe from the "file size"

GiteaMirror commented

@lafriks commented on GitHub (Feb 5, 2018):

how big is file that was not indexed?

@lafriks commented on GitHub (Feb 5, 2018): how big is file that was not indexed?

GiteaMirror commented

@ghost commented on GitHub (Feb 5, 2018):

Only 9KB, and is a Java file

@ghost commented on GitHub (Feb 5, 2018): Only 9KB, and is a Java file

GiteaMirror commented

@lafriks commented on GitHub (Feb 5, 2018):

Do you see any errors in log file? and why do you think it was not indexed?

@lafriks commented on GitHub (Feb 5, 2018): Do you see any errors in log file? and why do you think it was not indexed?

GiteaMirror commented

@ghost commented on GitHub (Feb 5, 2018):

No errors in log file. I am sure that the file was not indexed because I am trying to look also for other strings inside this file, but no results appear.

@ghost commented on GitHub (Feb 5, 2018): No errors in log file. I am sure that the file was not indexed because I am trying to look also for other strings inside this file, but no results appear.

GiteaMirror commented

@ghost commented on GitHub (Feb 5, 2018):

I just replicated the problem. If I create a new repository with just this file and just this repo, then the file is indexed correctly. I think that the problem is the max "store" file size

@ghost commented on GitHub (Feb 5, 2018): I just replicated the problem. If I create a new repository with just this file and just this repo, then the file is indexed correctly. I think that the problem is the max "store" file size

GiteaMirror commented

@lafriks commented on GitHub (Feb 5, 2018):

I don't think there is maximal size for store file

@lafriks commented on GitHub (Feb 5, 2018): I don't think there is maximal size for store file

GiteaMirror commented

@ghost commented on GitHub (Feb 5, 2018):

You can try to replicate the problem by cloning the repo: https://github.com/Xilinx/u-boot-xlnx
and search for string: "dm_test_adc_bind" (that is present in file "test/dm/adc.c"). Even in this case, the store file goes to a size of 1.048.576 KB and stop working (stop to grow up and to index files)

@ghost commented on GitHub (Feb 5, 2018): You can try to replicate the problem by cloning the repo: https://github.com/Xilinx/u-boot-xlnx and search for string: "dm_test_adc_bind" (that is present in file "test/dm/adc.c"). Even in this case, the store file goes to a size of 1.048.576 KB and stop working (stop to grow up and to index files)

GiteaMirror commented

@lafriks commented on GitHub (Feb 5, 2018):

@ethantkoenig can you check this?

@lafriks commented on GitHub (Feb 5, 2018): @ethantkoenig can you check this?

GiteaMirror commented

@ghost commented on GitHub (Feb 5, 2018):

@ethantkoenig @lafriks Thank you. I also want to add an additional information: with the same repository occured also an access violation during repo indexing: Exception

@ghost commented on GitHub (Feb 5, 2018): @ethantkoenig @lafriks Thank you. I also want to add an additional information: with the same repository occured also an access violation during repo indexing: [Exception](https://github.com/go-gitea/gitea/files/1695767/error.txt)

GiteaMirror commented

@ethantkoenig commented on GitHub (Feb 6, 2018):

In my experience, bleve does not do well once index files approach/exceed 1 GB (although it may be different for other machines and OS's). I do recall seeing similar behavior at some point (where an index file remained the exact same size as more content was added to it), but wasn't able to figure out what was going on.

I have found that using sharded indices (not natively supported by bleve) allows bleve to scale more gracefully. Perhaps this is something to consider using in gitea.

@ethantkoenig commented on GitHub (Feb 6, 2018): In my experience, bleve does not do well once index files approach/exceed 1 GB (although it may be different for other machines and OS's). I do recall seeing similar behavior at some point (where an index file remained the exact same size as more content was added to it), but wasn't able to figure out what was going on. I have found that using sharded indices (not natively supported by bleve) allows bleve to scale more gracefully. Perhaps this is something to consider using in gitea.

GiteaMirror commented

@ghost commented on GitHub (Feb 6, 2018):

Thank you for your answer @ethantkoenig . Are there any workaround to avoid this behavior?

@ghost commented on GitHub (Feb 6, 2018): Thank you for your answer @ethantkoenig . Are there any workaround to avoid this behavior?

GiteaMirror commented

@ghost commented on GitHub (Feb 6, 2018):

I also want to add some external links to boltdb:
https://github.com/boltdb/bolt/issues/266
https://github.com/boltdb/bolt/issues/308
https://github.com/boltdb/bolt/blob/master/db.go (check comments at row 17, and starting from line 305)
Are those useful?

@ghost commented on GitHub (Feb 6, 2018): I also want to add some external links to boltdb: https://github.com/boltdb/bolt/issues/266 https://github.com/boltdb/bolt/issues/308 https://github.com/boltdb/bolt/blob/master/db.go (check comments at row 17, and starting from line 305) Are those useful?

GiteaMirror commented

@ghost commented on GitHub (Feb 8, 2018):

@ethantkoenig an additional information: your lasts commit on master, changed the store max size to 524.288 KB, instead of, for "v1.4.0-rc1" the size is fixed to 1.048.576 KB

@ghost commented on GitHub (Feb 8, 2018): @ethantkoenig an additional information: your lasts commit on master, changed the store max size to 524.288 KB, instead of, for "v1.4.0-rc1" the size is fixed to 1.048.576 KB

GiteaMirror commented

@ghost commented on GitHub (Feb 10, 2018):

@ethantkoenig @lafriks After hours of reverse engineering I think that I found the problem about max store sizing on gitea v1.4.0-rc1. On monday I will test mine tricky-hack on production gitea, and I will let you know about it here

@ghost commented on GitHub (Feb 10, 2018): @ethantkoenig @lafriks After hours of reverse engineering I think that I found the problem about max store sizing on gitea v1.4.0-rc1. On monday I will test mine tricky-hack on production gitea, and I will let you know about it here

GiteaMirror commented

@lafriks commented on GitHub (Feb 10, 2018):

@giudon that's great to hear

@lafriks commented on GitHub (Feb 10, 2018): @giudon that's great to hear

GiteaMirror commented

@ghost commented on GitHub (Feb 12, 2018):

@ethantkoenig @lafriks Now I can confirm my analisys: there is a problem in boltdb. In particular it happens in such particular cases, when is needed to remap the store file after 1GB. The problem is still present in the master version, because I seen that on gitea we are using this db.go version ccd680d8c1/db.go (this commit was done by @lunny , and after this there is only one commit more, that doesn't change the remap logic on db.go).
The following are the technical details (before start, I want to say that my OS pageSize, on a Windows 64 bit machine is equal to 4.096 bytes):
When boltdb needs to allocate new "junk" to store file, this method is called:

as you can see, when is needed to increase the store file is called the method "mmap" with a minimum size reallocation parameter. Inside "mmap" is called the method "mmapSize", that calculate the new store file size based on the minimum one passed as parameter:

as you can see from my red comments, there are 2 different calculation methods inside "mmapSize". The first one calculate the new store size following this rule, based on the minimum size parameter:
2^15 = 32.768 bytes
2^16 = 65.536 bytes
2^17 = 131.072 bytes
2^18 = 262.144 bytes
2^19 = 524.288 bytes
2^20 = 1.048.576 bytes
2^21 = 2.097.152 bytes
2^22 = 4.194.304 bytes
2^23 = 8.388.608 bytes
2^24 = 16.777.216 bytes
2^25 = 33.554.432 bytes
2^26 = 67.108.864 bytes
2^27 = 134.217.728 bytes
2^28 = 268.435.456 bytes
2^29 = 536.870.912 bytes
2^30 = 1.073.741.824 bytes
and this method works well.
The problems starts when the second calculation method take control over remap size, because the minimum size that is passed as parameter when 1GB is reached was "1.073.741.824", and not as I was expecting a value greater than "1.073.741.824".
So, when 1GB is reached, simply the mmapSize return the same minimum size passed as parameter (1.073.741.824), because following codes:

are not reached.

My little tricky-hack to let it works, is to comment the following red lines inside "mmapSize" method:

and so, when 1GB is reached, the new allocation size will be increased everytime by os.Getpagesize().
Now repo indexing work fine:

What do you think about it guys?

@ghost commented on GitHub (Feb 12, 2018): @ethantkoenig @lafriks Now I can confirm my analisys: there is a problem in boltdb. In particular it happens in such particular cases, when is needed to remap the store file after 1GB. The problem is still present in the master version, because I seen that on gitea we are using this db.go version https://github.com/boltdb/bolt/blob/ccd680d8c1a0179ac3d68f692b01e1a1589cbfc7/db.go (this commit was done by @lunny , and after this there is only one commit more, that doesn't change the remap logic on db.go). The following are the technical details (before start, I want to say that my OS pageSize, on a Windows 64 bit machine is equal to 4.096 bytes): When boltdb needs to allocate new "junk" to store file, this method is called: ![immagine](https://user-images.githubusercontent.com/6648129/36090559-14e7b03c-0fe1-11e8-832d-002d8673ff98.png) as you can see, when is needed to increase the store file is called the method "mmap" with a minimum size reallocation parameter. Inside "mmap" is called the method "mmapSize", that calculate the new store file size based on the minimum one passed as parameter: ![immagine](https://user-images.githubusercontent.com/6648129/36090740-d2c1add8-0fe1-11e8-969a-286b158e74ee.png) as you can see from my red comments, there are 2 different calculation methods inside "mmapSize". The first one calculate the new store size following this rule, based on the minimum size parameter: 2^15 = 32.768 bytes 2^16 = 65.536 bytes 2^17 = 131.072 bytes 2^18 = 262.144 bytes 2^19 = 524.288 bytes 2^20 = 1.048.576 bytes 2^21 = 2.097.152 bytes 2^22 = 4.194.304 bytes 2^23 = 8.388.608 bytes 2^24 = 16.777.216 bytes 2^25 = 33.554.432 bytes 2^26 = 67.108.864 bytes 2^27 = 134.217.728 bytes 2^28 = 268.435.456 bytes 2^29 = 536.870.912 bytes 2^30 = 1.073.741.824 bytes and this method works well. The problems starts when the second calculation method take control over remap size, because the minimum size that is passed as parameter when 1GB is reached was "1.073.741.824", and not as I was expecting a value greater than "1.073.741.824". So, when 1GB is reached, simply the mmapSize return the same minimum size passed as parameter (1.073.741.824), because following codes: ![immagine](https://user-images.githubusercontent.com/6648129/36091008-9c11d41a-0fe2-11e8-8080-871cf08016c8.png) are not reached. My little tricky-hack to let it works, is to comment the following red lines inside "mmapSize" method: ![immagine](https://user-images.githubusercontent.com/6648129/36091067-ca597954-0fe2-11e8-85ba-7353838c1819.png) and so, when 1GB is reached, the new allocation size will be increased everytime by os.Getpagesize(). Now repo indexing work fine: ![immagine](https://user-images.githubusercontent.com/6648129/36091259-77022cc8-0fe3-11e8-8413-b05e393f0e5b.png) What do you think about it guys?

GiteaMirror commented

@lafriks commented on GitHub (Feb 12, 2018):

@giudon great bug hunting :) can you report this upstream to boltdb?

@lafriks commented on GitHub (Feb 12, 2018): @giudon great bug hunting :) can you report this upstream to boltdb?

GiteaMirror commented

@ghost commented on GitHub (Feb 12, 2018):

@lafriks did it

@ghost commented on GitHub (Feb 12, 2018): @lafriks did it

GiteaMirror commented

@guillep2k commented on GitHub (Aug 1, 2019):

Bolt have become archived/unmaintained. Proposed alternative seems to be https://github.com/etcd-io/bbolt. Perhaps it's worth taking a look at it?

@guillep2k commented on GitHub (Aug 1, 2019): Bolt have become archived/unmaintained. Proposed alternative seems to be https://github.com/etcd-io/bbolt. Perhaps it's worth taking a look at it?

GiteaMirror commented