Attempt to compare rwlock and LMDB internal locking mechanism.
The benchmark first writes 1000 key/values (actually, the same key is re-used) in LMDB and read it 1000 times. Then it does the same thing using concurrency, on a 16 cores machine, with the following configurations: - 32 threads (29 readers and 3 writer) - 8 threads (6 readers and 2 writer) - 8 threads (8 readers and 0 writer) See [3] for LMDB internal locking result and [4] for rwlock result. What the result seems to say is that LMDB internal locking mechanism is doing way better (100x) when there are lot of concurrency involved. I'm kinda surprised by such factor and tried various things in case I've done something wrong, but eventually always end up with similar difference. The only explanation I have is that LMDB actually does uses an userspace RCU mechanism internally, so readers are never blocked. Using rwlock would block the readers while a writer lock is alive. LMDB doc seems to say [1] that if we use our own synchronization mechanism, we should only have one writer at the time, and hold a write lock from the beginning of the transaction up to the commit/revert (I tried holding a read lock on a write transaction then upgrade to a write lock only at commit time... but of course I end up in lot of odd crashes). I think this explain the massive congestion when using 32 threads, as lot of readers are waiting. When there is no thread congestion (the 2 later configuration with 8 threads) using the rwlock actually seems a little bit faster (same order of magnitude though). I guess this might be explained by the fact that, while LMDB doesn't lock readers, there is still a quick lock-unlock to maintains a reader table (and eventually free unused old versions once a writer commit). [2] [1] http://www.lmdb.tech/doc/group__mdb.html#:~:text=must%20ensure%20that%20no%20readers%20are%20using%20old%20transactions%20while%20a%20writer%20is%20active [2] http://www.lmdb.tech/doc/group__readers.html [3] two runs using LMDB internal locking (HEAD~2) sequential writes and read avg write 0.005878 ms avg read 0.001640 ms concurrent 29 readers and 3 writers avg write 0.746217 ms avg read 6.644507 ms concurrent 6 readers and 2 writers avg write 0.077576 ms avg read 0.177316 ms concurrent 8 readers and 0 writer avg write 0.000000 ms avg read 0.273563 ms sequential writes and read avg write 0.005799 ms avg read 0.001656 ms concurrent 29 readers and 3 writers avg write 0.737782 ms avg read 6.616162 ms concurrent 6 readers and 2 writers avg write 0.075893 ms avg read 0.178631 ms concurrent 8 readers and 0 writer avg write 0.000000 ms avg read 0.303209 ms [4] two runs using rwlock (HEAD~1) sequential writes and read avg write 0.005931 ms avg read 0.001658 ms concurrent 29 readers and 3 writers avg write 17.249417 ms avg read 163.805094 ms concurrent 6 readers and 2 writers avg write 0.040559 ms avg read 0.168439 ms concurrent 8 readers and 0 writer avg write 0.000000 ms avg read 0.200699 ms sequential writes and read avg write 0.007423 ms avg read 0.001656 ms concurrent 29 readers and 3 writers avg write 20.312584 ms avg read 190.565366 ms concurrent 6 readers and 2 writers avg write 0.026039 ms avg read 0.147365 ms concurrent 8 readers and 0 writer avg write 0.000000 ms avg read 0.174471 ms
This commit is contained in:
Colin Vidal