René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Publish C-level coverage of the CPython test suite","articleBody":"**Feature or enhancement**\r\n\r\nAutomatically publish C-level coverage of the CPython test suite.\r\n\r\n**Pitch**\r\n\r\nRecent issues, such as [the problems with frame.setlineno](https://github.com/python/cpython/issues/94438), have revealed that the C-level coverage of the test suite isn't perfect.  While source-level coverage isn't the only metric of a good test suite, paying closer attention to it should help us have more confidence when making changes to the CPython core.\r\n\r\n**Proposed design for MVP**\r\n\r\n- **Publish results nightly:** This cadence should be enough for most cases. While various tools like codecov.io can run on every pull request, coverage takes a long time to run, and IME, there is enough noise in the results that developers (especially new to the project) would spend a lot of time \"chasing\" coverage.\r\n- **Use clang/llvm-cov rather than gcc/gcov:** As @erlend-aasland [pointed out](https://github.com/faster-cpython/ideas/issues/426#issuecomment-1178091681) the LLVM toolchain coverage provides branch counts and coverage of subexpressions which seems pretty useful. There is a risk that this isn't the same tooling as most Linux developers would use, and not what `make coverage-report` does.\r\n- **Publish to Github Pages:** The easiest thing seems to be to publish to Github Pages in a new repository in the `python` org for this purpose. This keeps all of the credential management within Github.\r\n\r\n**Out of scope**\r\n\r\nCreating diffs between coverage runs seems like it would be useful, but I haven't found a good tool for that yet.\r\n\r\nPython-level coverage reports would likewise be useful, but should be a follow-on to this.\r\n\r\n**Previous discussion**\r\n\r\n@brettcannon had set up [coverage reports in the past](https://github.com/python/cpython/actions/workflows/coverage.yml), but they have been broken for a couple of releases. This aims to be simpler than that approach by using a pure Github stack (not using codecov.io) and would just run nightly, not as part of every pull request.\r\n\r\nMore recent discussion began here: https://github.com/faster-cpython/ideas/issues/426\n\n\u003c!-- gh-pr-number: gh-94760 --\u003e\n* PR: gh-94760\n\u003c!-- /gh-pr-number --\u003e\n","author":{"url":"https://github.com/mdboom","@type":"Person","name":"mdboom"},"datePublished":"2022-07-11T19:24:58.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/94759/cpython/issues/94759"}