René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"lxml doesn’t like control characters","articleBody":"Same issue as #33, but with other non-whitespace C0 control characters: U+0001 to U+0008, U+000B, U+000C, U+000E to U+001F.\n\nEach of these trigger the exception below:\n\n```\nhtml5lib.parse('\u003cp\u003e\u0026#1;', treebuilder='lxml')\nhtml5lib.parse('\u003cp\u003e\\x01', treebuilder='lxml')\nhtml5lib.parse('\u003cp id=\"\u0026#1;\"\u003e', treebuilder='lxml')\nhtml5lib.parse('\u003cp id=\"\\x01\"\u003e', treebuilder='lxml')\n```\n\n```\nTraceback (most recent call last):\n  File \"/tmp/a.py\", line 4, in \u003cmodule\u003e\n    html5lib.parse('\u003cp\u003e\u0026#1;', treebuilder='lxml')\n  File \"/home/simon/.virtualenvs/weasyprint/lib/python3.3/site-packages/html5lib/html5parser.py\", line 28, in parse\n    return p.parse(doc, encoding=encoding)\n  File \"/home/simon/.virtualenvs/weasyprint/lib/python3.3/site-packages/html5lib/html5parser.py\", line 224, in parse\n    parseMeta=parseMeta, useChardet=useChardet)\n  File \"/home/simon/.virtualenvs/weasyprint/lib/python3.3/site-packages/html5lib/html5parser.py\", line 93, in _parse\n    self.mainLoop()\n  File \"/home/simon/.virtualenvs/weasyprint/lib/python3.3/site-packages/html5lib/html5parser.py\", line 183, in mainLoop\n    new_token = phase.processCharacters(new_token)\n  File \"/home/simon/.virtualenvs/weasyprint/lib/python3.3/site-packages/html5lib/html5parser.py\", line 991, in processCharacters\n    self.tree.insertText(token[\"data\"])\n  File \"/home/simon/.virtualenvs/weasyprint/lib/python3.3/site-packages/html5lib/treebuilders/_base.py\", line 320, in insertText\n    parent.insertText(data)\n  File \"/home/simon/.virtualenvs/weasyprint/lib/python3.3/site-packages/html5lib/treebuilders/etree_lxml.py\", line 240, in insertText\n    builder.Element.insertText(self, data, insertBefore)\n  File \"/home/simon/.virtualenvs/weasyprint/lib/python3.3/site-packages/html5lib/treebuilders/etree.py\", line 108, in insertText\n    self._element.text += data\n  File \"lxml.etree.pyx\", line 921, in lxml.etree._Element.text.__set__ (src/lxml/lxml.etree.c:41467)\n  File \"apihelpers.pxi\", line 652, in lxml.etree._setNodeText (src/lxml/lxml.etree.c:18888)\n  File \"apihelpers.pxi\", line 1335, in lxml.etree._utf8 (src/lxml/lxml.etree.c:24701)\nValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters\n```\n\nU+000C in text (but not in attribute values) is replaced by U+0020 with a warning:\n\n```\nDataLossWarning: Text cannot contain U+000C\n```\n\nlibxml2’s HTML parser replaces them with nothing, which I slightly prefer. Anyway, this is probably what should happen for every character that lxml doesn’t like.\n","author":{"url":"https://github.com/SimonSapin","@type":"Person","name":"SimonSapin"},"datePublished":"2013-07-22T10:17:29.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":26},"url":"https://github.com/96/html5lib-python/issues/96"}