René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Align the grammar documentation with Python's actual grammar","articleBody":"# Documentation\n\nThe current documentation of Python syntax (the later chapters of the [language reference](https://docs.python.org/3/reference/index.html)) uses hand-maintained *production lists*, like this:\n\nA)\n\n```\ncompound_stmt ::=  if_stmt\n                   | while_stmt\n                   | for_stmt\n                   | try_stmt\n                   | with_stmt\n                   | match_stmt\n                   | funcdef\n                   | classdef\n                   | async_with_stmt\n                   | async_for_stmt\n                   | async_funcdef\nsuite         ::=  stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENT\nstatement     ::=  stmt_list NEWLINE | compound_stmt\nstmt_list     ::=  simple_stmt (\";\" simple_stmt)* [\";\"]\n```\n\nThere is no mechanism to ensure that these are in sync with the [actual grammar](https://github.com/python/cpython/blob/main/Grammar/python.gram), and they inevitably do get out of sync.\nSee some of the [“docs” issues mentioning “grammar”](https://github.com/python/cpython/issues?q=is%3Aopen+is%3Aissue+label%3Adocs+grammar).\n\nIt's not easy to write an automatic tool to keep them in sync, because we *do* want to elide some details -- the parser rules, *unnecessary* lookaheads, cuts, etc. But, it's *possible* to write it, and we wrote a [proof of concept](https://github.com/encukou/cpython/blob/grammar-in-docs/Tools/peg_generator/docs_generator.py), which will need to be rewritten, tuned, and reviewed. Before introducing it, I'd like to go through all the docs, correct the existing documentation, bring it closer to what a tool could generate, and discuss what the *ideal* presentation would look like. That needs to be a manual process, and it will also need to touch the *prose* that's next to the grammar snippets.\n\nAs a first step, I propose an update to the tooling, which brings the *presentation* a bit closer to the `python.gram` syntax.\n\nFrom the *existing* ReST source, we can get this:\n\nB)\n\n```\ncompound_stmt: if_stmt\n               | while_stmt\n               | for_stmt\n               | try_stmt\n               | with_stmt\n               | match_stmt\n               | funcdef\n               | classdef\n               | async_with_stmt\n               | async_for_stmt\n               | async_funcdef\nsuite:         stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENT\nstatement:     stmt_list NEWLINE | compound_stmt\nstmt_list:     simple_stmt (\";\" simple_stmt)* [\";\"]\n```\n\nSince Sphinx hard-codes the *productionlist* formatting (the `::=` symbol and the aligning), we'll need to override the `productionlist` directive to achieve this.\n\nThen, by changing the ReST and using a different directive, we can get to something like:\n\nC)\n\n```\ncompound_stmt:\n    | if_stmt\n    | while_stmt\n    | for_stmt\n    | try_stmt\n    | with_stmt\n    | match_stmt\n    | funcdef\n    | classdef\n    | async_with_stmt\n    | async_for_stmt\n    | async_funcdef\nsuite:\n    | stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENT\nstatement:\n    | stmt_list NEWLINE | compound_stmt\nstmt_list:\n    | simple_stmt (\";\" simple_stmt)* [\";\"]\n```\n\nI propose to go from A) to B) at once (by overriding `productionlist`), and from B) to C) gradually, while also updating the content (including changing rule names to match the grammar, and adjusting/reorganizing nearby prose).\nI think that the B) and C) styles are similar enough that mixing them in a single version of the docs should not be jarring.\n\nBy the way, one additional benefit of a custom directive is that we can add syntax highlighting. (Ideally, with support from the theme.) I think that making strings stand out makes the listings more readable:\n\n![image](https://github.com/user-attachments/assets/731cc733-d3d8-4c0b-b4f7-83dcded43157)\n\n---\n\nAs a second step, I'd like to rewrite *token* documentation, and then the *lexical analysis* chapter, on which the grammar chapters build: #135676\n\nThen, continue with the grammar chapters in Language reference:\n\n* #141984\n* Simple statements\n* Compound statements\n* Top-level components\n* Full Grammar specification\n\nAnd after that, use generated grammar snippets wherever possible.\n\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-127835\n* gh-129689\n* gh-129690\n* gh-129692\n* gh-130376\n* gh-131468\n* gh-131474\n* gh-132407\n* gh-133632\n* gh-133633\n* gh-133652\n* gh-134423\n* gh-134443\n* gh-134850\n* gh-135301\n\u003c!-- /gh-linked-prs --\u003e\n\n### Other related PRs\n* gh-130588\n* gh-134034","author":{"url":"https://github.com/encukou","@type":"Person","name":"encukou"},"datePublished":"2024-12-11T16:49:03.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/127833/cpython/issues/127833"}