Title: Tokenizer produces token '<>', missing in the language grammar (regardless on __future__.barry_as_FLUFL settings) · Issue #151464 · python/cpython · GitHub
Open Graph Title: Tokenizer produces token '<>', missing in the language grammar (regardless on __future__.barry_as_FLUFL settings) · Issue #151464 · python/cpython
X Title: Tokenizer produces token '<>', missing in the language grammar (regardless on __future__.barry_as_FLUFL settings) · Issue #151464 · python/cpython
Description: Long live to the SC! This is outcome of the #151303. In particular, that issue proposed a friendlier syntax error message for code like 1 <> 2. Now consider this: $ cat a.py 1 <> 2 $ cat a.py | python3.14 -m tokenize 1,0-1,1: NUMBER '1' ...
Open Graph Description: Long live to the SC! This is outcome of the #151303. In particular, that issue proposed a friendlier syntax error message for code like 1 <> 2. Now consider this: $ cat a.py 1 <> 2 $ cat a.py | pyt...
X Description: Long live to the SC! This is outcome of the #151303. In particular, that issue proposed a friendlier syntax error message for code like 1 <> 2. Now consider this: $ cat a.py 1 <> 2 $ ca...
Opengraph URL: https://github.com/python/cpython/issues/151464
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Tokenizer produces token '\u003c>', missing in the language grammar (regardless on __future__.barry_as_FLUFL settings)","articleBody":"Long live to the SC!\n\nThis is outcome of the https://github.com/python/cpython/issues/151303. In particular, that issue proposed a friendlier syntax error message for code like ``1 \u003c> 2``.\n\nNow consider this:\n```\n$ cat a.py\n1 \u003c> 2\n$ cat a.py | python3.14 -m tokenize\n1,0-1,1: NUMBER '1' \n1,2-1,4: OP '\u003c>' \n1,5-1,6: NUMBER '2' \n1,6-1,7: NEWLINE '\\n' \n2,0-2,0: ENDMARKER '' \n```\nvs\n```\n$ cat b.py\n1 !! 2\n$ cat b.py | python3.14 -m tokenize\n1,0-1,1: NUMBER '1' \n1,2-1,3: OP '!' \n1,3-1,4: OP '!' \n1,5-1,6: NUMBER '2' \n1,6-1,7: NEWLINE '\\n' \n2,0-2,0: ENDMARKER '' \n```\n\nThe Python grammar [has no official token](https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters) ``\u003c>``, but that's not true: the evil ``\u003c>`` (as synonym for ``!=``) was introduced by previous (unsuccessful) [evil attempt](https://github.com/python/cpython/commit/e3944a5e1ecf67aa722fd9ce0c0a4ee72ee5ba2d) to overthrow Guido (Long live the BDFL!).\n\nI think we are all in danger. The attached patch (enables different tokenization just in the REPL) shows that in principle it's possible to apply ``__future__`` settings to the tokenizer too. \n\n\u003cdetails>\n\n\u003csummary>The Holy Patch, That Save The World\u003c/summary>\n\n```diff\ndiff --git a/Parser/lexer/lexer.c b/Parser/lexer/lexer.c\nindex 7f25afec302..dd982a1f68d 100644\n--- a/Parser/lexer/lexer.c\n+++ b/Parser/lexer/lexer.c\n@@ -1280,7 +1280,7 @@ tok_get_normal_mode(struct tok_state *tok, tokenizer_mode* current_tok, struct t\n /* Check for two-character token */\n {\n int c2 = tok_nextc(tok);\n- int current_token = _PyToken_TwoChars(c, c2);\n+ int current_token = _PyToken_TwoChars(c, c2, tok->BARRY_AS_BDFL);\n if (current_token != OP) {\n int c3 = tok_nextc(tok);\n int current_token3 = _PyToken_ThreeChars(c, c2, c3);\ndiff --git a/Parser/lexer/state.c b/Parser/lexer/state.c\nindex 5cf9b4d768c..72daaa2a464 100644\n--- a/Parser/lexer/state.c\n+++ b/Parser/lexer/state.c\n@@ -63,6 +63,7 @@ _PyTokenizer_tok_new(void)\n #ifdef Py_DEBUG\n tok->debug = _Py_GetConfig()->parser_debug;\n #endif\n+ tok->BARRY_AS_BDFL = 0;\n return tok;\n }\n \ndiff --git a/Parser/lexer/state.h b/Parser/lexer/state.h\nindex 9cd196a114c..c367a4a0b7d 100644\n--- a/Parser/lexer/state.h\n+++ b/Parser/lexer/state.h\n@@ -137,6 +137,7 @@ struct tok_state {\n #ifdef Py_DEBUG\n int debug;\n #endif\n+ int BARRY_AS_BDFL;\n };\n \n int _PyLexer_type_comment_token_setup(struct tok_state *tok, struct token *token, int type, int col_offset,\ndiff --git a/Parser/pegen.c b/Parser/pegen.c\nindex 569f5afb312..9954231cb41 100644\n--- a/Parser/pegen.c\n+++ b/Parser/pegen.c\n@@ -1042,10 +1042,11 @@ _PyPegen_run_parser_from_string(const char *str, int start_rule, PyObject *filen\n PyCompilerFlags *flags, PyArena *arena, PyObject *module)\n {\n int exec_input = start_rule == Py_file_input;\n+ int parser_flags = compute_parser_flags(flags);\n \n struct tok_state *tok;\n if (flags != NULL && flags->cf_flags & PyCF_IGNORE_COOKIE) {\n- tok = _PyTokenizer_FromUTF8(str, exec_input, 0);\n+ tok = _PyTokenizer_FromUTF8(str, exec_input, 0, parser_flags & PyPARSE_BARRY_AS_BDFL);\n } else {\n tok = _PyTokenizer_FromString(str, exec_input, 0);\n }\n@@ -1062,7 +1063,6 @@ _PyPegen_run_parser_from_string(const char *str, int start_rule, PyObject *filen\n // We need to clear up from here on\n mod_ty result = NULL;\n \n- int parser_flags = compute_parser_flags(flags);\n int feature_version = flags && (flags->cf_flags & PyCF_ONLY_AST) ?\n flags->cf_feature_version : PY_MINOR_VERSION;\n Parser *p = _PyPegen_Parser_New(tok, start_rule, parser_flags, feature_version,\ndiff --git a/Parser/tokenizer/tokenizer.h b/Parser/tokenizer/tokenizer.h\nindex 8fbeb2d6ae6..bf04cf40a48 100644\n--- a/Parser/tokenizer/tokenizer.h\n+++ b/Parser/tokenizer/tokenizer.h\n@@ -4,7 +4,7 @@\n #include \"Python.h\"\n \n struct tok_state *_PyTokenizer_FromString(const char *, int, int);\n-struct tok_state *_PyTokenizer_FromUTF8(const char *, int, int);\n+struct tok_state *_PyTokenizer_FromUTF8(const char *, int, int, int);\n struct tok_state *_PyTokenizer_FromReadline(PyObject*, const char*, int, int);\n struct tok_state *_PyTokenizer_FromFile(FILE *, const char*,\n const char *, const char *);\n\ndiff --git a/Parser/tokenizer/utf8_tokenizer.c b/Parser/tokenizer/utf8_tokenizer.c\nindex 1a925f44540..f31a7fe606a 100644\n--- a/Parser/tokenizer/utf8_tokenizer.c\n+++ b/Parser/tokenizer/utf8_tokenizer.c\n@@ -28,7 +28,8 @@ tok_underflow_string(struct tok_state *tok) {\n \n /* Set up tokenizer for UTF-8 string */\n struct tok_state *\n-_PyTokenizer_FromUTF8(const char *str, int exec_input, int preserve_crlf)\n+_PyTokenizer_FromUTF8(const char *str, int exec_input, int preserve_crlf,\n+ int BARRY_AS_BDFL)\n {\n struct tok_state *tok = _PyTokenizer_tok_new();\n char *translated;\n@@ -51,5 +52,6 @@ _PyTokenizer_FromUTF8(const char *str, int exec_input, int preserve_crlf)\n tok->buf = tok->cur = tok->inp = translated;\n tok->end = translated;\n tok->underflow = &tok_underflow_string;\n+ tok->BARRY_AS_BDFL = BARRY_AS_BDFL;\n return tok;\n }\ndiff --git a/Tools/build/generate_token.py b/Tools/build/generate_token.py\nindex 9ee5ec86e75..c7603e212d2 100755\n--- a/Tools/build/generate_token.py\n+++ b/Tools/build/generate_token.py\n@@ -94,7 +94,7 @@ def update_file(file, content):\n // Export these 4 symbols for 'test_peg_generator'\n PyAPI_DATA(const char * const) _PyParser_TokenNames[]; /* Token names */\n PyAPI_FUNC(int) _PyToken_OneChar(int);\n-PyAPI_FUNC(int) _PyToken_TwoChars(int, int);\n+PyAPI_FUNC(int) _PyToken_TwoChars(int, int, int);\n PyAPI_FUNC(int) _PyToken_ThreeChars(int, int, int);\n \n #ifdef __cplusplus\n@@ -142,7 +142,7 @@ def make_h(infile, outfile='Include/internal/pycore_token.h'):\n }\n \n int\n-_PyToken_TwoChars(int c1, int c2)\n+_PyToken_TwoChars(int c1, int c2, int BARRY_AS_BDFL)\n {\n %s\\\n return OP;\n@@ -171,7 +171,10 @@ def generate_chars_to_token(mapping, n=1):\n write(indent)\n write(' break;\\n')\n else:\n- write(\"case '%s': return %s;\\n\" % (c, value))\n+ if c == '>' and value == 'NOTEQUAL':\n+ write(\"case '%s': return BARRY_AS_BDFL ? %s : OP;\\n\" % (c, value))\n+ else:\n+ write(\"case '%s': return %s;\\n\" % (c, value))\n write(indent)\n write('}\\n')\n return ''.join(result)\n```\n\n\u003c/details>\n\nShould we apply something like this? Of course, referenced issue could be solved and with the evil token. Though, @warsaw might overthrow the SC. BTW, Long live the FLUFL too! (I think it's safer to glorify all dictators, just in case.)\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-154854\n* gh-158189\n* gh-158190\n* gh-158191\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/skirpichev","@type":"Person","name":"skirpichev"},"datePublished":"2026-06-14T06:57:06Z","url":"https://github.com/python/cpython/issues/151464"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:edf90def-5fa4-dca0-fa89-0a7376ec3b58 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 93E8:1FE434:18EE03B:20E0984:6B020F64 |
| html-safe-nonce | 2f311f98cc9a1e61244cdb75229567353edc8c637caeb314a452744de0e1f8ca |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5M0U4OjFGRTQzNDoxOEVFMDNCOjIwRTA5ODQ6NkIwMjBGNjQiLCJ2aXNpdG9yX2lkIjoiMTA2NzA4MDM0MDEwOTEzNTcxNyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | f4d162430c1366a5b0b96ca45279890d1a13e5aac3fae73bf956c6069a9fd6a1 |
| hovercard-subject-tag | issue:4658223708 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/151464/issue_layout |
| twitter:image | https://opengraph.githubassets.com/56deb8c981d6cd1407050ef36a0c144b0e9685d6178ab7b5524b06fc83c75492/python/cpython/issues/151464 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/56deb8c981d6cd1407050ef36a0c144b0e9685d6178ab7b5524b06fc83c75492/python/cpython/issues/151464 |
| og:image:alt | Long live to the SC! This is outcome of the #151303. In particular, that issue proposed a friendlier syntax error message for code like 1 <> 2. Now consider this: $ cat a.py 1 <> 2 $ cat a.py | pyt... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | skirpichev |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width