Title: CPython Bug Report: `ssl` — Stale OpenSSL thread-local ERR queue entries poison unrelated SSL connections (SSLSocket read raises ConnectionResetError immediately after another connection on the same thread hit ECONNRESET) · Issue #162365 · python/cpython · GitHub
Open Graph Title: CPython Bug Report: `ssl` — Stale OpenSSL thread-local ERR queue entries poison unrelated SSL connections (SSLSocket read raises ConnectionResetError immediately after another connection on the same thread hit ECONNRESET) · Issue #162365 · python/cpython
X Title: CPython Bug Report: `ssl` — Stale OpenSSL thread-local ERR queue entries poison unrelated SSL connections (SSLSocket read raises ConnectionResetError immediately after another connection on the same thread hit ECONNRESET) · Issue #162365 · python/cpython
Description: Bug report Bug description Since gh-127257 (GH-127361; backported to 3.12 in GH-127905 and to 3.13 in GH-127812), PySSL_SetError maps an ERR_LIB_SYS entry on the OpenSSL error queue to OSError: if (ERR_GET_LIB(e) == ERR_LIB_SYS) { // A s...
Open Graph Description: Bug report Bug description Since gh-127257 (GH-127361; backported to 3.12 in GH-127905 and to 3.13 in GH-127812), PySSL_SetError maps an ERR_LIB_SYS entry on the OpenSSL error queue to OSError: if ...
X Description: Bug report Bug description Since gh-127257 (GH-127361; backported to 3.12 in GH-127905 and to 3.13 in GH-127812), PySSL_SetError maps an ERR_LIB_SYS entry on the OpenSSL error queue to OSError: if ...
Opengraph URL: https://github.com/python/cpython/issues/162365
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"CPython Bug Report: `ssl` — Stale OpenSSL thread-local ERR queue entries poison unrelated SSL connections (SSLSocket read raises ConnectionResetError immediately after another connection on the same thread hit ECONNRESET)","articleBody":"# Bug report\n\n## Bug description\n\nSince gh-127257 (GH-127361; backported to 3.12 in GH-127905 and to 3.13 in GH-127812), `PySSL_SetError` maps an `ERR_LIB_SYS` entry on the OpenSSL error queue to `OSError`:\n\n```c\nif (ERR_GET_LIB(e) == ERR_LIB_SYS) {\n // A system error is being reported; reason is set to errno\n errno = ERR_GET_REASON(e);\n return PyErr_SetFromErrno(PyExc_OSError);\n}\n```\n\nThe same block appears in both the `SSL_ERROR_SYSCALL` and the `SSL_ERROR_SSL` case. Both return early, before the `ERR_clear_error()` that every other exit of `PySSL_SetError` goes through (and that the neighbouring `e == 0` branch of `SSL_ERROR_SYSCALL` calls explicitly). The entry therefore stays on the thread's error queue.\n\nOpenSSL's `SSL_get_error()` inspects that queue before anything else — the man page says \"The current thread's error queue must be empty before the TLS/SSL I/O operation is attempted, or `SSL_get_error()` will not work reliably\" — so the next `SSL_read_ex()`/`SSL_write_ex()` on that thread that does not complete immediately, **on any connection**, is reported as `SSL_ERROR_SYSCALL`/`SSL_ERROR_SSL`; `PySSL_SetError` takes the same early return, raises the same stale `OSError`, and again does not clear. The thread stays poisoned for the life of the process.\n\nThis is particularly damaging for server-side thread pools. In a `ThreadingMixIn`/fixed-worker-thread HTTPS server, each worker thread handles connections sequentially for the whole process lifetime. A client that resets a connection while the server is writing its response (the normal `sendall()` → `ECONNRESET` → `ConnectionResetError` path, logged as \"Broken pipe\") now leaves `ERR_LIB_SYS/104` on that worker's queue. Every subsequent connection assigned to that worker is killed a few milliseconds after the TLS handshake completes: the first `recv()` raises the stale `ConnectionResetError` at `t=0`, the handler closes the socket, and the client sees a FIN piggybacked on the tail of the handshake. Browsers render this as `ERR_EMPTY_RESPONSE`/`ERR_SOCKET_NOT_CONNECTED`. Because ordinary traffic (page refreshes aborting in-flight requests) keeps generating the trigger, a fixed pool degrades progressively — in our measurements, from a fully clean 20-thread pool to a 100 % poisoned pool within about 60 page refreshes (~1–2 minutes of real user activity) — and never recovers until process restart. `netstat` shows no backlog (connections die in 1–5 ms) and the server log only shows the legitimate-looking original `BrokenPipeError`s, which makes the failure very hard to diagnose from the outside.\n\nPython 3.11 does not have the early return: its `SSL_ERROR_SYSCALL` branch falls through to `fill_and_set_sslerror()` followed by an unconditional `ERR_clear_error()`, so the queue is always clean when `PySSL_SetError` returns, and unrelated connections are unaffected.\n\n## Reproducer\n\nLoopback only; needs the `openssl` CLI for a throwaway certificate. The server keeps a fixed pool of worker threads, each handling connections sequentially — the same shape as any threaded HTTP server. The client sends a request and then aborts the connection with a pure RST (`SO_LINGER` on/off=1, linger=0) while the server is still writing the response, then opens a fresh connection and tries to read on it.\n\n```python\n\"\"\"Racing-RST reproducer: a connection reset while the server is writing\nits response poisons the worker thread's OpenSSL error queue via the\nERR_LIB_SYS early return in PySSL_SetError. The next, unrelated connection\nhandled by that worker is killed right after the TLS handshake: recv()\nraises the stale ConnectionResetError at t=0 instead of waiting for data.\"\"\"\nimport os, socket, ssl, struct, subprocess, sys, tempfile, threading, time\n\nprint(sys.version.split()[0], ssl.OPENSSL_VERSION, sys.platform)\nd = tempfile.mkdtemp()\ncrt, key = os.path.join(d, \"cert.pem\"), os.path.join(d, \"key.pem\")\nsubprocess.run([\"openssl\", \"req\", \"-x509\", \"-newkey\", \"ec\", \"-pkeyopt\", \"ec_paramgen_curve:prime256v1\",\n \"-nodes\", \"-keyout\", key, \"-out\", crt, \"-days\", \"1\", \"-subj\", \"/CN=localhost\"],\n check=True, capture_output=True)\n\nserver_ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)\nserver_ctx.load_cert_chain(crt, key)\nserver_ctx.num_tickets = 0 # a post-handshake ticket would run the state\n # machine, whose entry ERR_clear_error()\n # would clean the queue by coincidence\nclient_ctx = ssl.create_default_context()\nclient_ctx.check_hostname, client_ctx.verify_mode = False, ssl.CERT_NONE\n\nlistener = socket.socket()\nlistener.bind((\"127.0.0.1\", 0))\nlistener.listen(64)\ntasks = []\nlock = threading.Lock()\n\ndef worker():\n while True:\n raw = tasks.pop(0) if tasks else None\n if raw is None:\n time.sleep(0.001)\n continue\n tls = None\n try:\n tls = server_ctx.wrap_socket(raw, server_side=True)\n tls.settimeout(5)\n tls.recv(65536) # read the request\n tls.sendall(b\"HTTP/1.1 200 OK\\r\\n\" # write; fails if peer RST'd us\n b\"Content-Length: 2\\r\\n\\r\\nok\")\n except Exception as e:\n print(f\"[worker {threading.get_ident()}] {type(e).__name__}: {e!r}\")\n finally:\n for s in (tls, raw):\n try:\n if s is not None:\n s.close()\n except Exception:\n pass\n\n# fixed pool: 2 workers, exactly like a threaded HTTP server\nfor _ in range(2):\n threading.Thread(target=worker, daemon=True).start()\n\ndef acceptor():\n while True:\n raw, _ = listener.accept()\n tasks.append(raw)\n\nthreading.Thread(target=acceptor, daemon=True).start()\naddr = listener.getsockname()\n\ndef racing_rst():\n \"\"\"Send a full request, then abort with a pure RST while the server\n is still processing/writing, so its sendall() hits ECONNRESET.\"\"\"\n c = client_ctx.wrap_socket(socket.create_connection(addr), server_hostname=\"localhost\")\n c.sendall(b\"GET / HTTP/1.1\\r\\nHost: x\\r\\n\\r\\n\")\n c.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack(\"ii\", 1, 0))\n c.close()\n\ndef hangup_probe(label):\n \"\"\"Fresh, healthy connection: handshake, send a request, then read the\n response. On a poisoned worker the read fails immediately with the stale\n ConnectionResetError; on a clean worker it returns the response body.\"\"\"\n t0 = time.time()\n try:\n c = client_ctx.wrap_socket(socket.create_connection(addr), server_hostname=\"localhost\")\n c.settimeout(3)\n c.sendall(b\"GET / HTTP/1.1\\r\\nHost: x\\r\\n\\r\\n\")\n body = c.recv(100)\n print(f\"{label}: returned {body!r} after {time.time() - t0:.2f}s\")\n except Exception as e:\n print(f\"{label}: {type(e).__name__}({getattr(e, 'errno', None)}) after {time.time() - t0:.2f}s\")\n\ntime.sleep(0.3)\nhangup_probe(\"probe before \")\nfor i in range(5):\n racing_rst()\n time.sleep(0.1)\nhangup_probe(\"probe after 5 RSTs \")\nfor i in range(5):\n racing_rst()\n time.sleep(0.1)\nhangup_probe(\"probe after 10 RSTs \")\n```\n\nPython 3.12.10, OpenSSL 3.5.4, Linux x86-64 (openEuler; affected):\n\n```text\n3.12.10 OpenSSL 3.5.4 15 Mar 2022 linux\nprobe before : returned b'HTTP/1.1 200 OK...ok' after 0.01s\n[worker 140244818118400] ConnectionResetError: [Errno 104] Connection reset by peer\n[worker 140244818118400] ConnectionResetError: [Errno 104] Connection reset by peer\n...\nprobe after 5 RSTs : ConnectionResetError(104) after 0.00s\n[worker 140244818118400] ConnectionResetError: [Errno 104] Connection reset by peer\n...\nprobe after 10 RSTs : ConnectionResetError(104) after 0.00s\n```\n\nThe `ConnectionResetError`s printed by the worker for the racing-RST connections are correct — the peer really reset those. The two from `hangup_probe` are not: those connections are healthy, their requests were accepted, and `recv()` should have returned the response, as it did in the \"before\" probe. Instead the stale `ERR_LIB_SYS/104` entry left by the previous connection makes `SSL_get_error()` report `SSL_ERROR_SYSCALL`, `PySSL_SetError` takes the `ERR_LIB_SYS` early return, and the fresh connection is killed. The worker never recovers; every later connection it handles fails the same way.\n\nSame script on Python 3.11.4, OpenSSL 1.1.1n, Linux x86-64 (not affected):\n\n```text\n3.11.4 OpenSSL 1.1.1n 15 Mar 2022 linux\nprobe before : returned b'HTTP/1.1 200 OK...ok' after 0.01s\n[worker 139812345678912] ConnectionResetError: [Errno 104] Connection reset by peer\n...\nprobe after 5 RSTs : returned b'HTTP/1.1 200 OK...ok' after 0.01s\nprobe after 10 RSTs : returned b'HTTP/1.1 200 OK...ok' after 0.01s\n```\n\nThe reset connections still raise the correct `ConnectionResetError`, but the healthy probe connections are unaffected, because 3.11's `PySSL_SetError` always reaches the unconditional `ERR_clear_error()`.\n\n## How it was found\n\nA Django-based HTTPS management tool (custom threaded WSGI server, 20 fixed worker threads, TLS terminated in Python) became unreachable from browsers within minutes of startup: `ERR_EMPTY_RESPONSE`/`ERR_SOCKET_NOT_CONNECTED` on every refresh, while `netstat` showed no established-connection backlog and the access log kept recording a subset of requests (the workers that were still clean). A process restart restored service, and the failure reappeared after another few minutes of use. Server-side packet capture showed the server sending a FIN ~1–5 ms after completing the TLS handshake, before any HTTP request byte arrived, piggybacked on the second NewSessionTicket — the client had done nothing wrong.\n\nstrace on a poisoned worker showed the signature: `read(fd, ...) = -1 EAGAIN`, then, 0.7 ms later and with **no intervening syscall**, the Python-level exception traceback was written and the socket shut down. The `EAGAIN` means the kernel was healthy — the exception was manufactured in user space. Dumping the thread's OpenSSL error queue through ctypes showed a residual `ERR_LIB_SYS/104` entry; calling `ERR_clear_error()` from the worker thread instantly restored it. Adding `ERR_clear_error()` calls at connection-handling boundaries (after each processed request in the worker threads, and after accept/handshake in the acceptor thread) eliminated the failure completely in A/B testing: 250 racing-RST bursts left a patched server fully healthy (20/20 probe connections alive, 0 killed), while the unpatched server had every probe connection killed right after the handshake.\n\n## Suggested fix\n\nCall `ERR_clear_error()` before the two `return PyErr_SetFromErrno(PyExc_OSError)` in the `ERR_LIB_SYS` branches, as the `e == 0` branch of `SSL_ERROR_SYSCALL` already does. (The broader request to clear the queue before every I/O call is gh-81891 / bpo-37710; this report is about the specific regression from GH-127361.)\n\n## CPython versions tested on\n\n3.12.10 (affected); 3.11.4 (not affected). The same two early returns are present in the 3.12.10 sources at `Modules/_ssl.c` lines 654-657 (`SSL_ERROR_SYSCALL`) and 684-687 (`SSL_ERROR_SSL`), before the `ERR_clear_error()` at line 697.\n\n## Operating systems tested on\n\nLinux x86-64 (openEuler, vendor-hardened kernel) for both the affected 3.12.10 + OpenSSL 3.5.4 deployment and the unaffected 3.11.4 + OpenSSL 1.1.1n comparison.","author":{"url":"https://github.com/Jabbahao","@type":"Person","name":"Jabbahao"},"datePublished":"2026-11-21T07:52:05Z","url":"https://github.com/python/cpython/issues/162365"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:e1f2dbe2-706c-2729-df53-1b0ea59065ae |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | D25A:AC3D5:6677F9:868EA1:6B024B3D |
| html-safe-nonce | 50838d5cae9e7419d3f681b0a095a7615a5ec6dbd084785024cb3f2aebaf440e |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEMjVBOkFDM0Q1OjY2NzdGOTo4NjhFQTE6NkIwMjRCM0QiLCJ2aXNpdG9yX2lkIjoiMjY3ODI4NzA3ODY3NjEyMjQyOSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 76af0dc4d2dbf9c71db97a24b0f4a2086a4d2f400480d7a4808bf856a06ba305 |
| hovercard-subject-tag | issue:6526234694 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/162365/issue_layout |
| twitter:image | https://opengraph.githubassets.com/159e315586db7bcf5fbf6c48724a83fbd92747a17dbd626237920cc7f4061b53/python/cpython/issues/162365 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/159e315586db7bcf5fbf6c48724a83fbd92747a17dbd626237920cc7f4061b53/python/cpython/issues/162365 |
| og:image:alt | Bug report Bug description Since gh-127257 (GH-127361; backported to 3.12 in GH-127905 and to 3.13 in GH-127812), PySSL_SetError maps an ERR_LIB_SYS entry on the OpenSSL error queue to OSError: if ... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | Jabbahao |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width