René's URL Explorer Experiment

Title: Python: switch dataflow library to new (shared) CFG + SSA by yoff · Pull Request #21925 · github/codeql · GitHub

Open Graph Title: Python: switch dataflow library to new (shared) CFG + SSA by yoff · Pull Request #21925 · github/codeql

X Title: Python: switch dataflow library to new (shared) CFG + SSA by yoff · Pull Request #21925 · github/codeql

Description: Summary The trunk-flip equivalent of the original draft PR #21894 (kept around as documentation of the end state), rebased on top of the four preparatory PRs in this stack: P1: #21919 — Remove AstNode.getAFlowNode() and rewrite callers. P2: #21920 — Qualify Flow.qll's AST references with Py:: prefix. P3: #21921 — Add new shared-CFG-backed control flow graph (additive). P4: #21923 — Add new shared-SSA-backed SSA adapter (additive). Based on #21923 — merge P1–P4 first. Flips the Python dataflow trunk from the legacy CFG (semmle/python/Flow.qll) and legacy ESSA SSA (semmle/python/essa/*) to the new shared CFG facade (semmle.python.controlflow.internal.Cfg) and the new SSA adapter (semmle.python.dataflow.new.internal.SsaImpl). What changes Dataflow library The Python dataflow library (semmle/python/dataflow/new/) now imports the new CFG facade and SSA adapter. All CFG-typed predicates (ControlFlowNode, CallNode, BasicBlock, NameNode, AttrNode, ...) are qualified with the Cfg:: prefix; SSA references switch from EssaVariable/EssaDefinition to SsaImpl::Definition/SourceVariable. GuardNode redesign GuardNode is redesigned to use the new CFG's outcome-node model (isAfterTrue/isAfterFalse) instead of the legacy ConditionBlock + flipped indirection. Only BarrierGuard<...> is preserved as public API — the rest of the legacy GuardNode surface (isSafeCheck, flipped) was Python-specific and had no callers outside this library. Framework updates Framework files (Bottle, FastApi, Django, Tornado, Pyramid, Stdlib, MarkupSafe, Pycurl, Pydantic, Gradio, ...) are updated to take CFG nodes from the new facade. Dataflow consistency tweaks for the new CFG Augmented-assignment targets are treated as both load and store. from X import * produces uncertain SSA writes for unknown names. CFG nodes are canonicalised so dataflow does not see equivalent pre/post-order pairs as distinct nodes. Two AST tweaks for the new CFG AstNodeImpl: omit PEP 695 type-parameter names from FunctionDefExpr/ClassDefExpr children (they belong to the type-params block, not the function/class header). ImportResolution: drop the legacy essa import. Test churn (~135 reblessed .expected) The reblessed .expected files fall into two buckets: Cosmetic node toString relabel (the dominant change). The new CFG node prints as X (or After X) where the legacy CFG printed ControlFlowNode for X. 118 of the 135 reblessed .expected are identical to the legacy output once this relabel is normalised away — i.e. a pure, alert-preserving rename. Slightly different CFG granularity. Flow-node-counting library tests (def-use-flow, use-use-flow, coverage, typetracking) see different counts because the new CFG has separate pre/post nodes per expression. These are node-count deltas, not alert deltas. Alert preservation — investigation To confirm the flip preserves results (not just node labels), the 135 reblessed .expected were audited against the legacy (pre-flip) output, normalising the node-toString relabel: 118 / 135 are byte-identical to legacy modulo the relabel — no result change at all. 17 / 135 differ in content. Each was inspected individually; all are either node-granularity count tests (above) or genuine precision improvements, namely: sqlalchemy / stdlib concept tests: more Concepts resolved (+20, +4). CallGraph type-tracking: calls that legacy could only resolve via points-to are now also resolved by type-tracking (MISSING: tt=… annotations removed). RequestWithoutValidation: the same alerts, with a richer message tracing verify=False back to its origin (… because $@ by $@). UnsafeUnpacking: identical #select; only an extra provenance hop in the path graph. No security alert is lost. Every #select/problem row present in the legacy output is present after the flip. Independent corroboration: 86 of the reblessed tests carry inline // $/# $ expectation annotations (via postprocess: utils/test/InlineExpectationsTestQuery.ql). Those annotations still match, witnessing preservation directly at the alert level rather than via the snapshot. Snapshot-freshness fix While auditing, a subset of the flip's .expected were found to be stale — they had been blessed at an earlier point in the stack, before the canonical (injects-only) ControlFlowNode representative and later dataflow refinements landed in the preparatory PRs, and had not been refreshed. The most visible symptom was CWE-022-PathInjection, whose stale snapshot recorded a Missing result: Source for a FastAPI Depends() parameter that the current code in fact reports correctly. All affected dirs were re-blessed; 51 .expected were refreshed (the rest already matched), and the audit above was run on the refreshed result. Verification All 367 lib/ + src/ + consistency-queries/ queries compile clean against the new trunk. All ControlFlow + PointsTo + dataflow + dataflow-new-ssa + essa + consistency library-tests pass, and the reblessed query-tests pass without --learn. CFG consistency verified at scale: consistency-queries/CfgConsistency.ql reports 0 violations on a full CPython database.

Open Graph Description: Summary The trunk-flip equivalent of the original draft PR #21894 (kept around as documentation of the end state), rebased on top of the four preparatory PRs in this stack: P1: #21919 — Remove Ast...

X Description: Summary The trunk-flip equivalent of the original draft PR #21894 (kept around as documentation of the end state), rebased on top of the four preparatory PRs in this stack: P1: #21919 — Remove Ast...

Opengraph URL: https://github.com/github/codeql/pull/21925

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width