René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"LittleEndianByteBufAllocator uses UnpooledByteBufAllocator is not optimal and potentially causing OOM error","articleBody":"MySQLConnectionHandler uses the default LittleEndianByteBufAllocator for netty\n`this.bootstrap.option(ChannelOption.ALLOCATOR, LittleEndianByteBufAllocator.INSTANCE)`\n\nLittleEndianByteBufAllocator uses UnpooledByteBufAllocator as default\n```\nclass LittleEndianByteBufAllocator(private val allocator: UnpooledByteBufAllocator = UnpooledByteBufAllocator(false)) :\n    ByteBufAllocator by allocator {\n\n    companion object {\n        val INSTANCE = LittleEndianByteBufAllocator()\n    }\n```\n\nI guess the jasync-sql team want to use heap memory instead of direct memory here.\n\nHowever, on the netty side, netty's SSLHandler will choose to use direct memory or not by himself\n\nio.netty.handler.ssl.SSLHandler\n```\n    private ByteBuf allocate(ChannelHandlerContext ctx, int capacity) {\n        ByteBufAllocator alloc = ctx.alloc();\n        if (engineType.wantsDirectBuffer) {\n            return alloc.directBuffer(capacity);\n        } else {\n            return alloc.buffer(capacity);\n        }\n    }\n```\n\nAnd unfortunately, in some cases, like when using tcnative libaray, netty will choose to use directBuffer explicitly. This is not efficient to allocate and de-allocate directly using unpooled allocator. And also, using Unpooled direct memory sometimes causing OutOfMemory error because large amount of direct memory has already got pooled by netty.\n\n```\n\u003cmysql-connection-35\u003e Transport failure  (mysql.MySQLConnection:)\njava.lang.OutOfMemoryError: Cannot reserve 4110 bytes of direct buffer memory (allocated: 134217321, limit: 134217728)\n\tat java.base/java.nio.Bits.reserveMemory(Unknown Source) ~[?:?]\n\tat java.base/java.nio.DirectByteBuffer.\u003cinit\u003e(Unknown Source) ~[?:?]\n\tat java.base/java.nio.ByteBuffer.allocateDirect(Unknown Source) ~[?:?]\n\tat io.netty.util.internal.CleanerJava9.allocate(CleanerJava9.java:86) ~[netty-common-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.util.internal.PlatformDependent.allocateDirect(PlatformDependent.java:567) ~[netty-common-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.buffer.UnpooledDirectByteBuf.allocateDirectBuffer(UnpooledDirectByteBuf.java:121) ~[netty-buffer-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeDirectByteBuf.allocateDirectBuffer(UnpooledByteBufAllocator.java:207) ~[netty-buffer-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.buffer.UnpooledDirectByteBuf.\u003cinit\u003e(UnpooledDirectByteBuf.java:66) ~[netty-buffer-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.buffer.UnpooledUnsafeDirectByteBuf.\u003cinit\u003e(UnpooledUnsafeDirectByteBuf.java:42) ~[netty-buffer-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeDirectByteBuf.\u003cinit\u003e(UnpooledByteBufAllocator.java:202) ~[netty-buffer-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:93) ~[netty-buffer-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[netty-buffer-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[netty-buffer-4.2.3.Final.jar!/:4.2.3.Final]\n\tat com.github.jasync.sql.db.mysql.codec.LittleEndianByteBufAllocator.directBuffer(LittleEndianByteBufAllocator.kt:43) ~[jasync-mysql-2.2.4.jar!/:?]\n\tat io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2442) ~[netty-handler-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1502) ~[netty-handler-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1398) ~[netty-handler-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1449) ~[netty-handler-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[netty-codec-base-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[netty-codec-base-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-base-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:356) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1429) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:167) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.handle(AbstractNioChannel.java:445) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.channel.nio.NioIoHandler$DefaultNioRegistration.handle(NioIoHandler.java:381) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.channel.nio.NioIoHandler.processSelectedKey(NioIoHandler.java:575) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.channel.nio.NioIoHandler.processSelectedKeysOptimized(NioIoHandler.java:550) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.channel.nio.NioIoHandler.processSelectedKeys(NioIoHandler.java:491) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.channel.nio.NioIoHandler.run(NioIoHandler.java:468) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.channel.SingleThreadIoEventLoop.runIo(SingleThreadIoEventLoop.java:207) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:178) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1073) ~[netty-common-4.2.3.Final.jar!/:4.2.3.Final]\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.2.3.Final.jar!/:4.2.3.Final]\n\tat java.base/java.lang.Thread.run(Unknown Source) [?:?]\n```\n\nSo, I would like to suggest to use PooledByteBufAllocator for LittleEndianByteBufAllocator. Or at least make it configurable.\nThanks!\n\nMy environment:\nLinux\n```\n\u003cdependency\u003e\n\t\t\t\u003cgroupId\u003eio.netty\u003c/groupId\u003e\n\t\t\t\u003cartifactId\u003enetty-tcnative-boringssl-static\u003c/artifactId\u003e\n\t\t\t\u003cversion\u003e2.0.72.Final\u003c/version\u003e\n\t\t\u003c/dependency\u003e\n\n\u003cdependency\u003e\n\t\t\t\u003cgroupId\u003eio.netty\u003c/groupId\u003e\n\t\t\t\u003cartifactId\u003enetty-handler\u003c/artifactId\u003e\n\t\t\t\u003cversion\u003e4.2.3.Final\u003c/version\u003e\n\t\t\u003c/dependency\u003e\n\n\u003cdependency\u003e\n\t\t\t\u003cgroupId\u003ecom.github.jasync-sql\u003c/groupId\u003e\n\t\t\t\u003cartifactId\u003ejasync-r2dbc-mysql\u003c/artifactId\u003e\n\t\t\t\u003cversion\u003e2.2.4\u003c/version\u003e\n\t\t\u003c/dependency\u003e\n```","author":{"url":"https://github.com/knucle-infinity","@type":"Person","name":"knucle-infinity"},"datePublished":"2025-10-17T03:40:14.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/435/jasync-sql/issues/435"}