René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"consider making html5lib.tokenizer public","articleBody":"Hello,\r\n\r\nIn version https://github.com/html5lib/html5lib-python/releases/tag/0.999999999 , `html5lib.tokenizer` was made private\r\n\r\nThe `wpull` project (https://github.com/ArchiveTeam/wpull ) uses this library, and if we were to ever migrate to using the 1.X versions, it would negatively impact the application, because instead of just tokenizing a webpage (see https://github.com/ArchiveTeam/wpull/blob/a4ff4a93f613ce18ad3c515aa3d4f5848a88b98c/wpull/document/htmlparse/html5lib_.py ), we would have to use the full tree parsing which is slower and uses more ram\r\n\r\nis there any reason this was made private when the 1.x branch was released?","author":{"url":"https://github.com/mgrandi","@type":"Person","name":"mgrandi"},"datePublished":"2021-04-12T05:36:33.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/532/html5lib-python/issues/532"}