René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"F-stack rack and BBR both causes PCB memory leak","articleBody":"My application suddenly start spamming `ff_socket: \"No buffer space available.\"` so I went to debug why. I figured out that setting tcp stack back to default freebsd stack solved the issue, it only happens on tcp stack to rack/bbr. Also does not happen unless socket is bound and connected (leak is in pcb), so test case must cover that.\r\n\r\nAfter debugging for a couple hours, i think it is stemming from tcphpts. \r\nI spewed logs in certain places to get insight. Hopefully the info can be helpful for you to get to the bottom of it.\r\n\r\nHere is logs when using freebsd stack.\r\nFirst log shows that a 4th pcb has been allocated, for total of 4 currently allocated. Middle lines show the socket creation and instant destruction, basically `ff_socket, ff_bind, ff_connect, ff_close` combo. Expected refcnt increases/decreases. Very last line shows that the pcb is getting freed, and down to 3 currently handed out. Looks good!\r\n```\r\nALLOC 4\r\ntcp_newtopcb do thing 0\r\nIN_PCBREF INCR REF: 1 -\u003e 2\r\ntcp_usr_detach\r\nDISCARD CB!\r\nDRAINCNT IS 0\r\nDO PCB RELE\r\nIN_PCBRELE_WLOCKED DECR REF: 2 -\u003e 1\r\nRELEASE RET ZERO! 1\r\nin_pcbfree CALLED! SCHEDULE DEFERRED.\r\nin_pcbfree_deferred!\r\nIN_PCBRELE_WLOCKED DECR REF: 1 -\u003e 0\r\nPCB RELE WLOCKED 3\r\n```\r\n\r\nHere's logs with rack/bbr stack enabled. Now we see tcp hpts is adding a refcnt, which didn't happen before, but does not seem to be  decreasing it. I have twiddled with tcp timewait/keepalive settings and brought them down to very low values, and let it sit there for hours in hopes to see it eventually release, but nope. Socket was not given nearly enough time to *actually* connect so i don't think that would be the issue anyways.\r\n\r\nso, in_pcbfree sets `INP_FREED` flag, which is picked up during in_pcbfree_deferred's call to in_pcbrele_wlocked, however during that call the refcnt is still 2, so somebody else is holding ref (appears to be tcphpts) meaning it can't be freed yet. \r\n\r\n```\r\nALLOC 4\r\ntcp_newtopcb do thing 0\r\nIN_PCBREF INCR REF: 1 -\u003e 2\r\nhpts do thing 0\r\nIN_PCBREF INCR REF: 2 -\u003e 3\r\ntcp_usr_detach\r\nDISCARD CB!\r\nDRAINCNT IS 0\r\nDO PCB RELE\r\nIN_PCBRELE_WLOCKED DECR REF: 3 -\u003e 2\r\nRELEASE RET ZERO! 2\r\nin_pcbfree CALLED! SCHEDULE DEFERRED.\r\nin_pcbfree_deferred!\r\nIN_PCBRELE_WLOCKED DECR REF: 2 -\u003e 1\r\nRELEASE RET ZERO! 1\r\nFLAG IS FREED!\r\n```\r\n\r\nTest case can be similar to #679 except also binds/connects to some host, then instant ff_close.\r\n\r\nWill run forever with freebsd stack without issue, will exit with these logs on bbr or rack stack:\r\n```\r\nff_socket: No buffer space available\r\nff_socket failed, fd:-1, errno:105, No buffer space available\r\nDone 262154\r\n```","author":{"url":"https://github.com/MrSonicMaster","@type":"Person","name":"MrSonicMaster"},"datePublished":"2022-10-14T19:49:07.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/702/f-stack/issues/702"}