René's URL Explorer Experiment

Title: Surface skipped conformance scenarios as baselined known failures by maxisbey · Pull Request #3030 · modelcontextprotocol/python-sdk · GitHub

Open Graph Title: Surface skipped conformance scenarios as baselined known failures by maxisbey · Pull Request #3030 · modelcontextprotocol/python-sdk

X Title: Surface skipped conformance scenarios as baselined known failures by maxisbey · Pull Request #3030 · modelcontextprotocol/python-sdk

Description: Conformance CI's green currently overstates coverage in two ways: 11 of the harness's server scenarios are never selected at all (the 10 extension-tagged tasks-* scenarios never match a --spec-version filter and sit on the harness's pending list, so the active, draft, and 2026 legs all exclude them; server-sse-polling is pending-listed and removed at 2026-07-28, so no leg can reach it), and several checks inside running scenarios skip silently when a prerequisite is missing — all 5 SEP-2243 Mcp-Param-* server-validation checks skip because no fixture tool carries an x-mcp-header annotation, and the SEP-2575 subscriptions/listen checks skip on -32601. This PR makes those surfaces visible as baselined known failures instead of silent exclusions, in three steps (one commit each): New server leg: bare --suite all (no --spec-version) — selects every scenario shipped with the pinned harness, including the extension-tagged and pending-listed ones that no existing leg reaches. With it, server-sse-polling turns out to already pass (3/3), and the 9 runnable tasks-* scenarios fail and are baselined in expected-failures.yml as a burn-down ledger — the harness's stale-entry rule forces each entry out as tasks support lands. tasks-status-notifications is deliberately not listed: the harness skips it unconditionally (pending its rewrite against subscriptions/listen), and a baseline entry for a scenario with no failing checks is itself flagged stale. This is the same pattern typescript-sdk uses (their extensions leg runs --suite all with no --spec-version and baselines the same nine tasks failures). test_x_mcp_header fixture tool in the everything-server (string param annotated x-mcp-header: Region via json_schema_extra, mirroring typescript-sdk's fixture) — the 5 Mcp-Param server-validation checks now execute: the 3 accept-path checks pass, the reject-path checks fail because server-side Mcp-Param validation isn't implemented yet, and the scenario is baselined in both expected-failures files. Conformance pin bump b18aa918 → 4944b268 (0.2.0-alpha.8, which includes conformance#372; not published to npm yet, so the same sha256-verified pkg.pr.new mechanics as before). #372 turns missing-prerequisite skips into "Not testable" failures, which surfaces one more unimplemented surface: server-stateless now fails its 3 subscriptions/listen MUST checks — the everything-server's legacy resources/subscribe handlers make it advertise resources.subscribe in server/discover while subscriptions/listen answers -32601, and advertised-but-rejected is now a failure. The scenario is baselined in both files until the listen runtime lands (#2804). The 2 listChanged SHOULD checks remain legitimately skipped (declared false, which #372 still treats as honest feature absence), and the tasks-* scenarios fail with higher check counts (their cascade skips now fail too) without needing entry changes. Net effect on the baseline: zero waivers for implemented surfaces is unchanged — every entry added here tracks a not-yet-built surface that previously produced no signal at all. Motivation and Context A green conformance run currently overstates coverage: unimplemented surfaces (tasks extension, server-side Mcp-Param validation, subscriptions/listen) produce no signal because their scenarios are filtered or their checks skip upstream of the expected-failures mechanism. Visible baselined failures are strictly better — they show up in the run output, they can't rot silently across harness pin bumps, and the stale-entry rule turns the baseline into a forced burn-down list as features land. How Has This Been Tested? Replicated all six CI legs locally against the new pinned tarball (sha256-verified), exactly as CI invokes them: the four server legs and both client legs exit 0 with "Baseline check passed" — --suite active → 42 passed, 0 failed (new entries inert — those scenarios aren't in the suite) --suite draft → 74 passed, 9 expected-failed (server-stateless 3 listen checks + http-custom-header-server-validation) --suite all --spec-version 2026-07-28 → 103 passed, 9 expected-failed (same two) --suite all (new leg) → 134 passed, 34 expected-failed (the above + 9 tasks-*); server-sse-polling and json-schema-2020-12 pass both client legs → 421/370 checks passed, 0 failed, 0 warnings Also verified the generated tools/list schema carries "x-mcp-header": "Region" through pydantic's json_schema_extra, and ran the discovery pass at the new pin before updating baselines to confirm server-stateless was the only newly-unexpected failure (no stale entries, no warnings anywhere). Breaking Changes None — CI configuration and a conformance fixture tool only. Types of changes Bug fix (non-breaking change which fixes an issue) New feature (non-breaking change which adds functionality) Breaking change (fix or feature that would cause existing functionality to change) Documentation update Checklist I have read the MCP Documentation My code follows the repository's style guidelines New and existing tests pass locally I have added appropriate error handling I have added or updated documentation as needed Additional context Suite choice for the new leg: --suite pending (14 scenarios) would cover the same job-level union slightly faster, since all = active ∪ draft ∪ pending at this harness and the other two suites already run in this job. Bare --suite all is preferred for the self-contained full-denominator run, robustness to scenarios moving between harness suite lists across pin bumps, and parity with typescript-sdk's CI. Why per-scenario waivers and not something finer: the harness baseline is keyed by scenario name only; one entry waives every failing check in that scenario. For tasks-* and http-custom-header-server-validation that's cheap (each tracks exactly one unimplemented surface). For server-stateless it carries a real masking cost, called out in the baseline comment: while listed, new failures in the scenario's other 25 (currently passing) envelope/error-handling checks are also absorbed. The trade is deliberate — a visible, forced-burn-down known failure for the listen gap beats a silent skip — but it makes landing the listen runtime (#2804) the way to get those 25 checks back under guard. The capability inconsistency the harness flags is real, not a harness artifact: on the 2026 wire the server advertises resources.subscribe while the subscribe methods themselves answer 404/-32601 there. Implementing listen resolves it from one side; era-aware capability reporting in server/discover would resolve it from the other. The bare-all leg re-runs the active/draft scenarios at their default wire versions (same connections as the existing legs), so most of its runtime is duplicate coverage; the harness has a 10s per-request timeout, and the tasks scenarios fail fast on -32601 rather than hanging. The one genuinely new timing-sensitive scenario is server-sse-polling (real-time SSE reconnect waits); it passed consistently in local runs and runs green in typescript-sdk's identical leg, but if it proves flaky on 2-core runners it cannot be pre-waived (a baseline entry for a passing scenario is flagged stale), so a flake would need a rerun or a harness-side fix. AI Disclaimer

Open Graph Description: Conformance CI's green currently overstates coverage in two ways: 11 of the harness's server scenarios are never selected at all (the 10 extension-tagged tasks-* scenarios never match a --s...

X Description: Conformance CI's green currently overstates coverage in two ways: 11 of the harness's server scenarios are never selected at all (the 10 extension-tagged tasks-* scenarios never mat...

Opengraph URL: https://github.com/modelcontextprotocol/python-sdk/pull/3030

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width