René's URL Explorer Experiment

Title: Reduce TypeForm recognition slowdown by davidfstr · Pull Request #21585 · python/mypy · GitHub

Open Graph Title: Reduce TypeForm recognition slowdown by davidfstr · Pull Request #21585 · python/mypy

X Title: Reduce TypeForm recognition slowdown by davidfstr · Pull Request #21585 · python/mypy

Description: References #21262 Summary Enabling TypeForm by default (#21262) regressed mypy's self-check by ~2% on mypyc-compiled builds. Details The regression came entirely from SemanticAnalyzer.try_parse_as_type_expression, which is now invoked eagerly on ~2.84M expressions per self-check. Of those, ~3.3% reached the expensive full-parse block (expr_to_analyzed_type + isolated_error_analysis), and ~91% of those full parses failed — pure wasted work. This branch adds a series of fast-rejection filters to the offending function (SemanticAnalyzer.try_parse_as_type_expression), eliminating ~83% of the full-parse attempts (3,144 → 542) on mypy's self-check, with zero correctness regressions. End-to-end this recovers ~43% of the regression by paired median (21.6 ms of a 50.2 ms CPU-time regression, n=100, CI ±5 ms; consistent with ~34–40% on wall-clock). This branch also adds additional instrumentation to analyze what kinds of expressions pass through the fast-rejection filters and make it to a full-parse. This instrumentation is enabled via the MYPY_TYPEFORM_PROFILE_FULL_PARSE variable and outputs a .tsv file that can be processed using the new misc/analyze_typeform_full_parse_profile.py script to classify & aggregate expressions. Finally this branch extends misc/perf_compare.py with additional reporting options to reduce measured variance. Each commit on the PR branch has a detailed message giving more information about the specific changes made. Optimization Results CPU time — canonical; lowest-variance metric python misc/perf_compare.py --warmup-runs 3 --num-runs 100 -j 3 \ --metric cpu --workers1 \ 022d9bc96 16fef2515 HEAD Commit Mean Median Δ vs baseline (paired median ±95% CI) 022d9bc96 baseline 2.768 s 2.757 s — 16fef2515 regression 2.817 s 2.809 s +50.2 ms ±5.2 (+1.82%) 3f39cd753/HEAD branch (all filters) 2.797 s 2.789 s +28.6 ms ±5.5 (+1.04%) Details CPU time (user+sys) on a single worker is the lowest-variance estimator here (CV ≈ 3.3%, vs ≈ 7.5% for wall-clock at the same n) and the truest measure of the per-call work being attacked: with one worker the whole self-check runs serially, so all ~2.84M calls land in one CPU-time figure rather than being spread across workers and hidden behind the slowest one. The branch recovers 21.6 ms of the 50.2 ms regression — ~43% by paired median (~40% by trimmed mean), leaving +28.6 ms (~57%, still well outside the ±5 ms CI). The recovered fraction is the difference of two deltas measured against the same baseline within one interleaved run, so it inherits the run's low noise (a conservative independent-error bound is ≈ ±7.6 ms on the 21.6 ms numerator; the true band is tighter because the two deltas are positively correlated — same baseline, same per-round machine state). Wall-clock — default user-facing metric python misc/perf_compare.py --warmup-runs 3 --num-runs 100 -j 3 \ 022d9bc96 16fef2515 HEAD python misc/perf_compare.py --warmup-runs 3 --num-runs 400 -j 3 \ 022d9bc96 16fef2515 HEAD Run Commit Median Δ vs baseline (paired median ±95% CI) n=100 022d9bc96 baseline 1.342 s — 16fef2515 regression 1.369 s +27.2 ms ±6.7 (+2.03%) 3f39cd753 branch 1.357 s +16.3 ms ±6.0 (+1.21%) n=400 022d9bc96 baseline 1.351 s — 16fef2515 regression 1.369 s +17.6 ms ±3.3 (+1.30%) 3f39cd753 branch 1.362 s +11.6 ms ±3.0 (+0.86%) Details Wall-clock recovery is **~40% at n=100** (10.9 ms of 27.2 ms) and **~34% at n=400** (6.0 ms of 17.6 ms) by paired median — consistent with the CPU figure once you allow for wall-clock's higher variance and the fact that it is bounded by the slowest worker. The recovered fraction (~34–43% across metrics and run lengths) stays stable even though the absolute regression differs a lot between metrics (≈50 ms CPU vs ≈18–27 ms wall) — the reassuring sign that the recovery is real rather than a noise artifact. Using the new full-parse instrumentation Details # 1. Run instrumented self-check on this branch: MYPY_TYPEFORM_PROFILE_FULL_PARSE=/tmp/tf.log \ python -m mypy --config-file mypy_self_check.ini -p mypy --no-incremental # 2. Aggregate the profile: python misc/analyze_typeform_full_parse_profile.py --top 20 /tmp/tf.log.* Open Questions Is the MYPY_TYPEFORM_PROFILE_FULL_PARSE env-var name acceptable, or should it follow an existing naming convention? I did not run performance numbers against non-mypy repositories (like PyTorch and Black) as I originally planned. Would you like me to? Should the misc/perf_compare.py changes ship in this PR, or land separately? They are a general benchmarking-harness improvement (CPU metric, single-worker mode, median-based reporting, opt-in binary cache), independent of the TypeForm filters — splitting them into their own PR may be cleaner for review.

Open Graph Description: References #21262 Summary Enabling TypeForm by default (#21262) regressed mypy's self-check by ~2% on mypyc-compiled builds. Details The regression came entirely from SemanticAnalyzer.try_parse...

X Description: References #21262 Summary Enabling TypeForm by default (#21262) regressed mypy's self-check by ~2% on mypyc-compiled builds. Details The regression came entirely from SemanticAnalyzer.try_p...

Opengraph URL: https://github.com/python/mypy/pull/21585

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width