René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"How do I get all the tokens for a statement?","articleBody":"I can't do `node.text.split(' ')`. \"Since tokens on hidden channels (e.g. whitespace or comments) are not added to the parse trees, they will not appear in the output of this method.\" So, I get `fromdatetimeimportdatetime,date` when traversing `from datetime import datetime, date`, for example.\r\n\r\nI'd want to do something like `node.getTokens()`, but `getTokens(x)` takes a `ttype` as a parameter that I haven't seen documentation for.\r\n\r\nI tried just plugging in incremental integers for `ttype`, but that didn't give me any tokens.\r\n```\r\nnode text fromdatetimeimportdatetime,date\r\nfor ttype -1 getTokens returns []\r\nfor ttype 0 getTokens returns []\r\nfor ttype 1 getTokens returns []\r\nfor ttype 2 getTokens returns []\r\nfor ttype 3 getTokens returns []\r\nfor ttype 4 getTokens returns []\r\nfor ttype 5 getTokens returns []\r\nfor ttype 6 getTokens returns []\r\nfor ttype 7 getTokens returns []\r\nfor ttype 8 getTokens returns []\r\nfor ttype 9 getTokens returns []\r\nfor ttype 10 getTokens returns []\r\nfor ttype 11 getTokens returns []\r\nfor ttype 12 getTokens returns []\r\nfor ttype 13 getTokens returns []\r\nfor ttype 14 getTokens returns []\r\nfor ttype 15 getTokens returns []\r\nfor ttype 16 getTokens returns []\r\nfor ttype 17 getTokens returns []\r\nfor ttype 18 getTokens returns []\r\nfor ttype 19 getTokens returns []\r\nfor ttype 20 getTokens returns []\r\nfor ttype 21 getTokens returns []\r\nfor ttype 22 getTokens returns []\r\nfor ttype 23 getTokens returns []\r\nfor ttype 24 getTokens returns []\r\nfor ttype 25 getTokens returns []\r\nfor ttype 26 getTokens returns []\r\nfor ttype 27 getTokens returns []\r\nfor ttype 28 getTokens returns []\r\n...\r\n```","author":{"url":"https://github.com/hamirmahal","@type":"Person","name":"hamirmahal"},"datePublished":"2023-03-23T16:52:20.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/4/python-ast/issues/4"}