René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Flaky streamable-HTTP/SSE tests: TOCTOU port race under pytest -n auto","articleBody":"### Initial Checks\n\n- [x] I confirm that I'm using the latest version of MCP Python SDK\n- [x] I confirm that I searched for my issue in https://github.com/modelcontextprotocol/python-sdk/issues before opening this issue\n\n### Description\n\n### Summary\n\n  Tests in `tests/shared/test_streamable_http.py` intermittently fail in CI (non-deterministically, across different Python versions). The failures are not real regressions, they're caused by a time-of-check/time-of-use (TOCTOU) race in how the test server fixtures allocate ports, which collides when tests run in parallel under pytest -n auto.\n\n### Evidence\n  \n  The flakiness is intermittent and non-deterministic: the same commit, re-run across the CI matrix, fails on different tests and different Python versions, while passing locally and on most matrix entries. That pattern, a failure that moves around rather than reproducing on a specific test/version, is the signature of a parallelism race, not a code defect.\n\n  Two failure signatures have been observed, and both reduce to \"the client connected to the wrong server instance\":\n\n  1. Server can't bind the port (`test_streamable_http_client_session_termination_204`)\n\n  ERROR: [Errno 98] error while attempting to bind on address ('127.0.0.1', 35105): address already in use\n  AssertionError: assert 2 == 10\n  The intended server (10 tools) loses the bind race, so the client reaches a different test's server: `len(tools.tools)` comes back as `2` (the `echo_headers/echo_context server`) instead of 10.\n\n  2. Crossed responses (`test_streamable_http_client_respects_retry_interval`)\n\n  pydantic_core.ValidationError: 1 validation error for CallToolResult\n  content\n    Field required [type=missing, input_value={'tools': [...]}, input_type=dict]\n  A call_tool request is answered with a `ListToolsResult` payload (`{'tools': [...]}`), which fails to validate as `CallToolResult`. The client is talking to a server/stream that belongs to another test.\n  \n  Both symptoms are downstream of two servers contending for the same ephemeral port — see Root cause below.\n\n\n### Root cause\n\nThe port fixtures pick a port, then close the socket before the real server binds it:\n\n```python\n# tests/shared/test_streamable_http.py:474\n@pytest.fixture\ndef basic_server_port() -\u003e int:\n    with socket.socket() as s:\n        s.bind((\"127.0.0.1\", 0))      # OS assigns a free port\n        return s.getsockname()[1]      # ...socket closes here, freeing the port\n```\n\nThe server is then started in a separate `multiprocessing.Process` (`run_server`, line 435) and binds that port later. In the gap, another xdist worker's fixture can be handed the same port by the OS, so two servers race for it - one fails with `Errno 98`, and clients can reach the wrong server.\n\n### Affected files (same pattern)\n\n  - tests/shared/test_streamable_http.py\n  - tests/shared/test_sse.py\n  - tests/server/test_sse_security.py\n  - tests/server/test_streamable_http_security.py\n  - tests/client/test_http_unicode.py\n\n### Proposed fix\n\nReuse the existing race-free helper `run_uvicorn_in_thread` in `tests/test_helpers.py:15`. It binds and `listen()`s a socket, then hands that same socket to uvicorn (`server.run(sockets=[sock])`), so there is no window where another worker can claim the port. This pattern is already proven in tests/shared/test_ws.py, and its docstring documents exactly this race.\n\nMigrate the racy fixtures, e.g.:\n\n```python\n# before: basic_server_port + basic_server + basic_server_url (3 fixtures, racy)\n# after:\n@pytest.fixture\ndef basic_server_url() -\u003e Generator[str, None, None]:\n    app = create_app()\n    with run_uvicorn_in_thread(app, limit_concurrency=10, timeout_keep_alive=5, access_log=False) as url:\n        yield url\n```\n\nThis also removes the need for the `wait_for_server(port)` poll (the helper's pre-listen()ed socket means connections are accepted as soon as the fixture yields).\n\n### Considerations\n\n  - This converts the test servers from a subprocess to a background thread (as `test_ws.py` already does). Need to confirm no test relies on subprocess semantics (e.g. `proc.kill()`); the streamable-HTTP tests appear to test HTTP-level behavior, not process lifecycle.\n  - A subprocess-preserving alternative (pass a pre-bound listening socket into the child via `server.run(sockets=[sock]))` is harder cross-platform, Windows spawn can't easily inherit/pickle sockets, so the thread helper is preferred for the CI matrix.\n\n### Scope\n\nPrimary scope: `tests/shared/test_streamable_http.py`. The 4 sibling files share the root cause and can be migrated in the same PR or as follow-ups.\n\n### Acceptance criteria\n\n  - Racy `*_port` fixtures + `run_server/wait_for_server` subprocess pattern removed from the affected file(s).\n  - Tests pass reliably under `uv run pytest -n auto` across the CI matrix (3.10–3.14, ubuntu + windows).\n\n\n### Example Code\n\n```Python\nimport socket\n\ndef pick_free_port() -\u003e int:\n    # Exact pattern used by basic_server_port / json_server_port / event_server_port\n    # in tests/shared/test_streamable_http.py\n    with socket.socket() as s:\n        s.bind((\"127.0.0.1\", 0))\n        return s.getsockname()[1]  # \u003c- socket closes here, so the port is free again\n\n\n# A fixture assigns this port to \"server A\"...\nport = pick_free_port()\n\n# ...but server A is started later (in a separate multiprocessing.Process), so the\n# port sits free in between. Under `pytest -n auto`, another worker can claim it.\n# Simulate that intruder grabbing the port during the window:\nintruder = socket.socket()\nintruder.bind((\"127.0.0.1\", port))\nintruder.listen()\n\n# Now server A finally tries to bind the port it was handed:\nserver_a = socket.socket()\nserver_a.bind((\"127.0.0.1\", port))  # OSError: [Errno 98] Address already in use\n```\n\n### Python \u0026 MCP Python SDK\n\n```Text\nPython: CPython 3.10 and 3.13 (failures captured on ubuntu-latest in CI).\n        Not version-specific, it's a test-parallelism race, so it can surface anywhere on the 3.10–3.14 matrix.\n\nMCP Python SDK: main branch (the unreleased v2 line) — 1.25.1.dev builds.\n```","author":{"url":"https://github.com/Ar-maan05","@type":"Person","name":"Ar-maan05"},"datePublished":"2026-05-28T08:10:33.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":5},"url":"https://github.com/2704/python-sdk/issues/2704"}