René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Fuzzing reveals a number of parse errors","articleBody":"I'm the lead developer of Beautiful Soup, which has html5lib as an optional dependency. Over the past couple of years I've gotten a number of notifications from Google's oss-fuzz project about unhandled exceptions that actually turned out to be problems in html5lib. There wasn't much I could do with these errors, but now that it looks like html5lib maintenance is picking up, I can pass them on to you. (Sorry. :crying_cat_face:)\r\n\r\nI've incorporated the fuzz reports into [the Beautiful Soup test suite](https://git.launchpad.net/beautifulsoup/tree/bs4/tests/test_fuzz.py), and [the test cases themselves are here](https://git.launchpad.net/beautifulsoup/tree/bs4/tests/fuzz), but here's a general picture of what problems I see. In each case, I believe just parsing the bad markup is enough to trigger the error.\r\n\r\n**clusterfuzz-testcase-minimized-bs4_fuzzer-4999465949331456**\r\n\r\nMarkup: `b')\u003ca\u003e\u003cmath\u003e\u003cTR\u003e\u003ca\u003e\u003cmI\u003e\u003ca\u003e\u003cp\u003e\u003ca\u003e'`\r\n\r\nError:\r\n\r\n```\r\nself = \u003chtml\u003e, node = \u003cp\u003e, refNode = None\r\n\r\n    def insertBefore(self, node, refNode):\r\n\u003e       index = self.element.index(refNode.element)\r\nE       AttributeError: 'NoneType' object has no attribute 'element'\r\n```\r\n\r\n**clusterfuzz-testcase-minimized-bs4_fuzzer-5843991618256896**\r\n\r\nMarkup: `b'-\u003cmath\u003e\u003csElect\u003e\u003cmi\u003e\u003csElect\u003e\u003csElect\u003e'`\r\n\r\nError:\r\n\r\n```\r\n    def resetInsertionMode(self):\r\n    ...\r\n            # Check for conditions that should only happen in the innerHTML\r\n            # case\r\n            if nodeName in (\"select\", \"colgroup\", \"head\", \"html\"):\r\n\u003e               assert self.innerHTML\r\nE               AssertionError\r\n```\r\n\r\n**clusterfuzz-testcase-minimized-bs4_fuzzer-6241471367348224**\r\n\r\nMarkup: `b'ñ\u003ctable\u003e\u003csvg\u003e\u003chtml\u003e'`\r\n\r\nError:\r\n\r\n```\r\nself = \u003chtml5lib.html5parser.getPhases.\u003clocals\u003e.InTablePhase object at 0x7f8f405ad440\u003e\r\n\r\n    def processEOF(self):\r\n        if self.tree.openElements[-1].name != \"html\":\r\n            self.parser.parseError(\"eof-in-table\")\r\n        else:\r\n\u003e           assert self.parser.innerHTML\r\nE           AssertionError\r\n```\r\n\r\n**clusterfuzz-testcase-minimized-bs4_fuzzer-6600557255327744**\r\n\r\nMarkup: `b'\\t\u003cTABLE\u003e\u003c\u003c!\u003e;\u003c!\u003e\u003c\u003c!\u003e.\u003clec\u003e\u003cth\u003ei\u003e\u003ca\u003e\u003cmat\\x00\\x01\u003cmi\\x00a\u003e\u003cmath\u003e\u003e\u003cth\u003e\u003cmI\u003echardeta\\xff\\xff\\xff\\xff\u003c\u003e\u003cth\u003e\u003cmI\u003e\u003c||||||||A\u003cselect\u003e\u003c\u003equ?\\xbemath\u003e\u003cth\u003e\u003cmie\u003equ'`\r\n\r\nError:\r\n\r\n```\r\nself = \u003chtml5lib.html5parser.getPhases.\u003clocals\u003e.InTableBodyPhase object at 0x7f8f4184ce00\u003e\r\n\r\n    def clearStackToTableBodyContext(self):\r\n        while self.tree.openElements[-1].name not in (\"tbody\", \"tfoot\",\r\n                                                      \"thead\", \"html\"):\r\n            # self.parser.parseError(\"unexpected-implied-end-tag-in-table\",\r\n            #  {\"name\": self.tree.openElements[-1].name})\r\n            self.tree.openElements.pop()\r\n        if self.tree.openElements[-1].name == \"html\":\r\n\u003e           assert self.parser.innerHTML\r\nE           AssertionError\r\n```\r\n\r\nAlso reported to me recently was the issue that was reported to you as issue #557.","author":{"url":"https://github.com/leonardr","@type":"Person","name":"leonardr"},"datePublished":"2023-03-20T13:40:48.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/568/html5lib-python/issues/568"}