René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Netty IllegalReferenceCountException causes false upload failure despite successful upload","articleBody":"### problem\n\n**Related Issue**\nRelated to https://github.com/apache/cloudstack/issues/12486\n\n**Summary**\nWhen uploading templates/ISOs from local file via the UI, the upload completes successfully on the SSVM (file is written to disk, SSVM returns HTTP 200 \"upload successful\"), but a Netty IllegalReferenceCountException occurs during response handling. This causes the UI to display an error to the user even though the upload actually succeeded. The template eventually becomes \"Ready\" after background processing completes.\n\n### versions\n\nCloudStack Version 4.20.2.0\n\n### The steps to reproduce the bug\n\n1. Navigate to Images → Templates\n2. Click \"Upload Template from local\"\n4. Fill in the form and submit\n5. Watch browser Network tab - upload returns 200 OK\n6. Check browser Console - may show error despite 200 OK\n7. Check SSVM logs - see Netty IllegalReferenceCountException\n8. Wait - template eventually becomes Ready\n\nNote: This issue is intermittent.\n\n**Expected Behavior**\nWhen the upload succeeds (HTTP 200, file written to disk), the UI should show success to the user.\n\n**Actual Behavior**\n\n- Upload succeeds (HTTP 200, \"upload successful\")\n- File is written to disk correctly\n- Netty throws IllegalReferenceCountException during cleanup\n- UI may show error to user (HTTP 432 in some cases)\n- Template eventually becomes Ready after background processing\n\n\n**Impact**\n\n- User confusion: User sees error but upload actually worked\n- Duplicate uploads: User may retry, creating duplicate templates\n- Wasted time: User troubleshoots non-existent problem\n\n\n**Root Cause Analysis**\nThe IllegalReferenceCountException occurs in `HttpUploadServerHandler.channelUnregistered()` when trying to destroy the `HttpPostRequestDecoder`. The decoder's reference count has already reached 0, indicating it was already released elsewhere.\nThis is a reference counting bug in the Netty upload handler - the decoder is being released twice:\n\n- Once during normal processing\n- Again in `channelUnregistered()` cleanup\n\n**Evidence**\n\nBrowser Network Tab - Upload Succeeded (HTTP 200)\nRequest Headers (all correct):\n\n```\nX-Expires: 2026-01-21T16:46:06.391Z\nX-Metadata: 7d9tasmSFbD/i/a+7cIPVHUKLB5oZUTfHOf6nIwEo6gdgJ1IHcgXfSHe1TouKFGRARpVRJJyyfVX5fqGcsv14oYurocRB4KGOkuW6zfW3GZC3hxhM2vA5OjK5Q497cxlc/TkgD/BOYoyx3VS32w4jFFxfVty9YSrc2tx1/bo9kE4MFXbFiuGNHI799xK8WhEXiTuR/7iyxRqLM17+ql/JYxlPLSP7DWbVOMFwm7HACbmSSaUNyJhi2OiaOmmU0TC5jpezvLxXhVmDsl9Fe9/uhTG2PDtmd/d2L+QtQz8+zNqxwgrS8EJqJKsk7q7tPzoDFKWc7Qp0uYpPvMqpyd377kXA2qhKnphEKrB6rBvIHV0OaX071UMNSysfrYC+GOIi3smUK6p5x939GlNlVFI6kYM5pkVZ9AsuNzGii4SMDZc8UGYrYmhazT10+ye0IlAVw5cFu5hcwVmeGnK8O2xDQkHo1z5wFYxiqCzZSuVdrhk0jdV0ktZ0CAcQeZzjBblu4DWFCcdqRDHl+8SkeAwtUjV3ZDht0u4D1lQdgV4G/NeDPCPhkm61WhKcR4AHvq5pb1+bs9h/8+b2BkuhfNbdNCd/gjW2kMsl4KcbhQwB+3uQsxZ9srg+hKacPB2Cyd8kgdLzuC4zYBq8mv8pRg4gpj5fbVC5Vs7pYbzpDlfoeIAuCZS7pDJM4Jontlae/QoGGo5\nX-Signature: zw+vnaAEiYNsjUfHov44KNZ/GBw=\nContent-Length: 400621794\n```\n\n- Response:\n\n```\nStatus Code: 200 OK\nBody: upload successful.\n```\n\n- SSVM Logs (/var/log/cloud.log) - Upload Received and Processed\n\n```\n2026-01-21T15:46:06,551 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: SIGNATURE=zw+vnaAEiYNsjUfHov44KNZ/GBw=\n2026-01-21T15:46:06,552 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: METADATA=7d9tasmSFbD/i/a+7cIPVHUKLB5oZUT...\n2026-01-21T15:46:06,552 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: EXPIRES=2026-01-21T16:46:06.391Z\n2026-01-21T15:46:06,552 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: HOST=10.0.56.181\n2026-01-21T15:46:06,553 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: CONTENT_LENGTH=400621794\n2026-01-21T15:46:06,554 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) URI: uuid=6c80606a-8826-460f-88a8-c15ed3606a6c\n2026-01-21T15:46:06,555 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) base directory: /mnt/SecStorage/5f75abe3-3803-3595-89ad-52d23b3dea97/template/tmpl/2/203\n2026-01-21T15:46:06,611 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) BODY: FileUpload: name=\"file\"; filename=\"linux-debian-11-x86_64-gen2-v1.qcow2\"\n```\n\n- SSVM Logs - Netty Exception During Cleanup\n\n```\n2026-01-21T15:47:30,572 WARN  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) \nDecoder already destroyed \nio.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1\n    at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83)\n    at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:140)\n    at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:79)\n    at io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:293)\n    at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelUnregistered(HttpUploadServerHandler.java:169)\n```\n\n- File Successfully Written to SSVM\n\n```\nroot@s-1-VM:~# ls -la /mnt/SecStorage/5f75abe3-3803-3595-89ad-52d23b3dea97/template/tmpl/2/203/\ntotal 391236\ndrwxrwxrwx 2 root root        95 Jan 21  2026 .\ndrwxr-xr-x 3 root root        25 Jan 21  2026 ..\n-rw-rw-rw- 1 root root 400621568 Jan 21  2026 c620128e-31ef-31ec-a04a-24104a89fdf0.qcow2\n-rw-rw-rw- 1 root root       340 Jan 21  2026 template.properties\n```\n\n- Template Properties File Created\n\n```\nroot@s-1-VM:~# cat /mnt/SecStorage/5f75abe3-3803-3595-89ad-52d23b3dea97/template/tmpl/2/203/template.properties\n#\n#Wed Jan 21 15:47:30 UTC 2026\nuniquename=203-2-0cb72ddc-d73a-3f2c-9f1f-24af1d033391\nqcow2.filename=c620128e-31ef-31ec-a04a-24104a89fdf0.qcow2\nqcow2.virtualsize=5368709120\ndescription=test\nqcow2.size=400621568\nfilename=c620128e-31ef-31ec-a04a-24104a89fdf0.qcow2\nsize=400621568\npublic=true\nid=203\nhvm=true\nvirtualsize=5368709120\nqcow2=true\n```\n\n- Despite the error shown to the user, the template eventually became Ready:\n\n```\n{\n  \"id\": \"6c80606a-8826-460f-88a8-c15ed3606a6c\",\n  \"name\": \"Debian 11\",\n  \"isready\": true,\n  \"status\": \"Download Complete\",\n  \"downloaddetails\": [\n    {\n      \"datastore\": \"NFS://10.0.32.4/acs/secondary/ref-trl-10677-k-Mr8-rositsa-kyuchukova/ref-trl-10677-k-Mr8-rositsa-kyuchukova-sec1\",\n      \"downloadPercent\": \"100\",\n      \"downloadState\": \"DOWNLOADED\"\n    }\n  ],\n  \"physicalsize\": 400621568,\n  \"size\": 5368709120\n}\n```\n\n### What to do about it?\n\nIn H`ttpUploadServerHandler.java`, add a null check or try-catch around the decoder destruction in `channelUnregistered()`\n\n**Workaround**\n\nIgnore the error in the UI and wait - template will eventually become Ready\nRefresh the template list after a minute to see actual status","author":{"url":"https://github.com/RosiKyu","@type":"Person","name":"RosiKyu"},"datePublished":"2026-01-21T18:22:02.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/12489/cloudstack/issues/12489"}