René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Reading variable sized data (a small entry followed by a large entry) leads to IndexOutOfBoundsException","articleBody":"This is a bit of a doozy.\r\n\r\nWe're in the process of moving an application to lmdbjava. We observe that when running the tests the following occurs:\r\n\r\n1. If the tests are all run in the same JVM process then we consistently get very strange errors related to ByteBuf validation constraints:\r\n\r\nFor example:\r\n\r\n```\r\nCaused by: java.lang.IndexOutOfBoundsException: readerIndex: 1297, writerIndex: 1095 (expected: 0 \u003c= readerIndex \u003c= writerIndex \u003c= capacity(1095))\r\n        at io.netty.buffer.AbstractByteBuf.checkIndexBounds(AbstractByteBuf.java:112)\r\n        at io.netty.buffer.AbstractByteBuf.writerIndex(AbstractByteBuf.java:135)\r\n        at org.lmdbjava.ByteBufProxy.out(ByteBufProxy.java:166)\r\n        at org.lmdbjava.ByteBufProxy.out(ByteBufProxy.java:41)\r\n        at org.lmdbjava.KeyVal.valOut(KeyVal.java:134)\r\n        at org.lmdbjava.Cursor.seek(Cursor.java:377)\r\n        at org.lmdbjava.Cursor.first(Cursor.java:126)\r\n        at org.lmdbjava.CursorIterable.executeCursorOp(CursorIterable.java:125)\r\n        at org.lmdbjava.CursorIterable.update(CursorIterable.java:172)\r\n        at org.lmdbjava.CursorIterable.access$100(CursorIterable.java:52)\r\n        at org.lmdbjava.CursorIterable$1.hasNext(CursorIterable.java:99)\r\n```\r\n\r\nThis occurs using the very standard code to iterate over all the keys in a table:\r\n\r\n```java\r\ntry (final CursorIterable\u003cByteBuf\u003e cursor = table.iterate(txn)) {\r\n\t\t\t\r\n\t\t\tfor (final KeyVal\u003cByteBuf\u003e kv : cursor) {\r\n\t\t\t\tfinal ByteBuf keyBuffer = kv.key();\r\n\t\t\t\tfinal ByteBuf valueBuffer = kv.val();\r\n```\r\n\r\n2. If the maven build is run using -DforkCount=1C -DreuseForks=false then all the tests run perfectly fine.  [These system properties tell maven to run each test class in its own JVM](https://www.baeldung.com/maven-junit-parallel-tests) and it eliminates all errors in the build at the expense of a much slower build.\r\n\r\nAdditionally if we run a single test using the -Dtest=$TestName syntax the test passes fine.\r\n\r\nClearly there is some erorr related to static state that is leading to inter-test dependencies. If we eliminate static state by running each Test in its own JVM all the tests pass. If we run the tests run the tests in the same JVM fails.\r\n\r\nI've tried to dig into this and get some idea of what's happening and I doi believe the problem is related to the ByteBufProxy class. There are some red flags in this class:\r\n\r\n1. This class maintains significant static state in a threadlocal BUFFERS cache. There seems to be no way to reset or clear this BUFFERS. As far I can tell the cache persists even if you call Env.close() and it might explain why we're getting ByteBufs whose readIndex is larger than their writeIndex.\r\n\r\n2. The ByteBufProxy class is also hardcoded to use PooledByteBufAllocator.DEFAULT. We've had problems in the past with multiple components (netty, netty extensions) all mucking around with PooledByteBufAllocator.DEFAULT. We tend to **avoid** using PooledByteBufAllocator.DEFAULT and instead having each component use its own explicit PooledByteBufAllocator. Unfortunately there's no way to configure ByteBufProxy to use a different ByteBufAllocator.\r\n\r\n3. The ByteBufProxy class makes several questionable assumptions. ByteBufs are allocated and not released, and no attempt is made to clear() ByteBufs returned from the pool. ","author":{"url":"https://github.com/buko","@type":"Person","name":"buko"},"datePublished":"2021-10-20T09:59:32.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":6},"url":"https://github.com/183/lmdbjava/issues/183"}