Title: RetryContext.lastAttemptNumber is inflated when retriable task is awaited via ctx.anyOf(...) · Issue #290 · microsoft/durabletask-java · GitHub
Open Graph Title: RetryContext.lastAttemptNumber is inflated when retriable task is awaited via ctx.anyOf(...) · Issue #290 · microsoft/durabletask-java
X Title: RetryContext.lastAttemptNumber is inflated when retriable task is awaited via ctx.anyOf(...) · Issue #290 · microsoft/durabletask-java
Description: Summary RetryContext.getLastAttemptNumber() violates its documented contract when the underlying retriable task is awaited indirectly via a CompoundTask (e.g. TaskOrchestrationContext.anyOf(...)). The counter increments on every Compound...
Open Graph Description: Summary RetryContext.getLastAttemptNumber() violates its documented contract when the underlying retriable task is awaited indirectly via a CompoundTask (e.g. TaskOrchestrationContext.anyOf(...)). ...
X Description: Summary RetryContext.getLastAttemptNumber() violates its documented contract when the underlying retriable task is awaited indirectly via a CompoundTask (e.g. TaskOrchestrationContext.anyOf(...)). ...
Opengraph URL: https://github.com/microsoft/durabletask-java/issues/290
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"RetryContext.lastAttemptNumber is inflated when retriable task is awaited via ctx.anyOf(...)","articleBody":"## Summary\n\n`RetryContext.getLastAttemptNumber()` violates its documented contract when the underlying retriable task is awaited indirectly via a `CompoundTask` (e.g. `TaskOrchestrationContext.anyOf(...)`). The counter increments on every `CompoundTask.await()` regardless of whether any task has failed, so a custom `RetryHandler` sees a `lastAttemptNumber` much higher than the actual number of failed attempts.\n\n## Documented contract\n\nFrom `RetryContext.getLastAttemptNumber()`:\n\n> Gets the previous retry attempt number. This number starts at 1 and increments each time the retry handler is invoked for a particular task failure.\n\n## Observed behavior\n\nIn a bounded sliding-window fan-out pattern that awaits retriable activities via `ctx.anyOf(...)`, tasks that enter the in-flight window *after* the first batch has burned through its retries observe `lastAttemptNumber ≈ window size` in their retry handler — on the **first real failure**, before any retry has happened.\n\n## Root cause\n\nThe inflated counter originates somewhere in `RetriableTask.attemptNumber` accounting when a freshly-scheduled `RetriableTask` joins an `anyOf` membership that is already being drained. Relevant entry points in `TaskOrchestrationExecutor.java` (1.9.0):\n\n```java\n// line ~2055\nvoid init() {\n this.startTime = this.startTime == null ? this.context.getCurrentInstant() : this.startTime;\n this.attemptNumber++;\n}\n\n// line ~2194\nprivate void initSubTasks() {\n for (Task\u003cV> subTask : this.subTasks) {\n if (subTask instanceof RetriableTask) ((RetriableTask\u003cV>)subTask).init();\n }\n}\n```\n\nWhat we can pin down from the observed behaviour:\n\n- The retry handler is invoked exactly once per real failure (no double-firing).\n- `totalRetryTimeMs` is consistent with the real number of attempts — it is `0` on the inflated invocations, confirming zero real retries have happened.\n- `attemptNumber` on a `RetriableTask` scheduled mid-drain is already much greater than 1 by the time its first failure reaches the handler.\n- `RetriableTask`s that exist before the drain loop starts retain a correct counter, even though they participate in many `CompoundTask.await()` cycles while still pending. So `init()` cannot be incrementing unconditionally on every `initSubTasks()` pass; there is effectively an \"already initialised\" guard for them.\n\nThe remaining unknown is what causes a *newly constructed* `RetriableTask` to begin life with `attemptNumber` already inflated — or to be subjected to `init()` calls that skip the guard the existing tasks benefit from — when it is added to an `anyOf` membership that has already cycled through several `CompoundTask.await()` calls. That interaction is the failure surface.\n\n## Minimal reproduction\n\nThe bug surfaces under these conditions:\n\n1. **Bounded sliding window**: tasks are added to the in-flight list *mid-drain* (one new task scheduled each time one completes).\n2. **Outer `try/catch`** wrapping `ctx.anyOf(inFlight).await()`. The `CompoundTask` returned by `anyOf()` re-throws the underlying `TaskFailedException` from its `await()`, rather than returning the failed task. Without an outer catch the orchestrator dies on the first failure and the bug never gets a chance to manifest on later tasks.\n3. **Enough items to fill a second window** after the first has completed. With `N` total items and a window of `W`, you need `N > W`. The first window goes through retries normally (1→max); the second window is where the bug shows up.\n\nKotlin, Azure Functions Java worker:\n\n```kotlin\nclass LastAttemptReproOrchestratorFunction {\n companion object {\n const val ORCHESTRATOR_NAME = \"LastAttemptReproOrchestratorFunction\"\n const val ACTIVITY_NAME = \"AlwaysFailsActivityFunction\"\n private const val NUM_ACTIVITIES = 20\n private const val MAX_IN_FLIGHT = 10\n private const val MAX_ATTEMPTS = 5\n private const val SLOW_TASK_STEP_MILLIS = 200L\n }\n\n @FunctionName(ORCHESTRATOR_NAME)\n fun run(\n @DurableOrchestrationTrigger(name = \"ctx\") ctx: TaskOrchestrationContext,\n executionContext: ExecutionContext,\n ) {\n val log = executionContext.logger\n val opts = TaskOptions(\n RetryHandler { retryCtx ->\n if (!retryCtx.orchestrationContext.isReplaying) {\n log.log(\n Level.WARNING,\n \"RetryHandler invoked: lastAttemptNumber=${retryCtx.lastAttemptNumber} \" +\n \"(max=$MAX_ATTEMPTS), totalRetryTimeMs=${retryCtx.totalRetryTime.toMillis()}\",\n )\n }\n retryCtx.lastAttemptNumber \u003c MAX_ATTEMPTS\n },\n )\n\n // Bounded sliding-window fan-out: keep at most MAX_IN_FLIGHT in flight at any time,\n // and start a replacement each time one completes.\n val inFlight = mutableListOf\u003cTask\u003c*>>()\n var nextIndex = 0\n while (nextIndex \u003c MAX_IN_FLIGHT && nextIndex \u003c NUM_ACTIVITIES) {\n inFlight += ctx.callActivity(ACTIVITY_NAME, nextIndex, opts, String::class.java)\n nextIndex++\n }\n while (inFlight.isNotEmpty()) {\n try {\n val completedTask: Task\u003c*> = ctx.anyOf(inFlight).await()\n inFlight.remove(completedTask)\n try {\n completedTask.await()\n } catch (e: TaskFailedException) {\n // expected — every activity always fails\n }\n } catch (e: TaskFailedException) {\n // anyOf().await() can throw the underlying TaskFailedException directly\n // rather than returning the failed task — identify the done task and continue.\n val done = inFlight.firstOrNull { it.isDone } ?: throw e\n inFlight.remove(done)\n }\n if (nextIndex \u003c NUM_ACTIVITIES) {\n inFlight += ctx.callActivity(ACTIVITY_NAME, nextIndex, opts, String::class.java)\n nextIndex++\n }\n }\n }\n\n @FunctionName(ACTIVITY_NAME)\n fun alwaysFails(@DurableActivityTrigger(name = \"index\") index: Int): String {\n Thread.sleep(index * SLOW_TASK_STEP_MILLIS)\n throw RuntimeException(\"Always fails (for lastAttemptNumber repro)\")\n }\n}\n```\n\n(Plus a trivial HTTP trigger that calls `durableContext.client.scheduleNewOrchestrationInstance(ORCHESTRATOR_NAME)` to start it.)\n\nThe staggered `Thread.sleep(index * 200ms)` in the activity makes the first batch of activities fail quickly and the second batch slowly, ensuring many `anyOf().await()` iterations run with the second batch still pending.\n\n### Observed output\n\nWith `NUM_ACTIVITIES=20`, `MAX_IN_FLIGHT=10`, `MAX_ATTEMPTS=5`, all `RetryHandler invoked` log lines from one orchestration run:\n\n```\n\"20/05/2026, 12:41:38.178\",\"RetryHandler invoked: lastAttemptNumber=1 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:38.251\",\"RetryHandler invoked: lastAttemptNumber=2 (max=5), totalRetryTimeMs=570\"\n\"20/05/2026, 12:41:38.313\",\"RetryHandler invoked: lastAttemptNumber=1 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:38.313\",\"RetryHandler invoked: lastAttemptNumber=3 (max=5), totalRetryTimeMs=661\"\n\"20/05/2026, 12:41:38.465\",\"RetryHandler invoked: lastAttemptNumber=4 (max=5), totalRetryTimeMs=722\"\n\"20/05/2026, 12:41:38.554\",\"RetryHandler invoked: lastAttemptNumber=1 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:38.554\",\"RetryHandler invoked: lastAttemptNumber=5 (max=5), totalRetryTimeMs=873\"\n\"20/05/2026, 12:41:38.623\",\"RetryHandler invoked: lastAttemptNumber=2 (max=5), totalRetryTimeMs=722\"\n\"20/05/2026, 12:41:38.763\",\"RetryHandler invoked: lastAttemptNumber=1 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:38.891\",\"RetryHandler invoked: lastAttemptNumber=3 (max=5), totalRetryTimeMs=1030\"\n\"20/05/2026, 12:41:38.891\",\"RetryHandler invoked: lastAttemptNumber=1 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:39.072\",\"RetryHandler invoked: lastAttemptNumber=2 (max=5), totalRetryTimeMs=961\"\n\"20/05/2026, 12:41:39.135\",\"RetryHandler invoked: lastAttemptNumber=1 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:39.201\",\"RetryHandler invoked: lastAttemptNumber=4 (max=5), totalRetryTimeMs=1298\"\n\"20/05/2026, 12:41:39.340\",\"RetryHandler invoked: lastAttemptNumber=1 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:39.437\",\"RetryHandler invoked: lastAttemptNumber=2 (max=5), totalRetryTimeMs=1170\"\n\"20/05/2026, 12:41:39.556\",\"RetryHandler invoked: lastAttemptNumber=5 (max=5), totalRetryTimeMs=1608\"\n\"20/05/2026, 12:41:39.556\",\"RetryHandler invoked: lastAttemptNumber=3 (max=5), totalRetryTimeMs=1479\"\n\"20/05/2026, 12:41:39.556\",\"RetryHandler invoked: lastAttemptNumber=1 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:39.748\",\"RetryHandler invoked: lastAttemptNumber=1 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:39.818\",\"RetryHandler invoked: lastAttemptNumber=2 (max=5), totalRetryTimeMs=1298\"\n\"20/05/2026, 12:41:39.952\",\"RetryHandler invoked: lastAttemptNumber=1 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:40.021\",\"RetryHandler invoked: lastAttemptNumber=4 (max=5), totalRetryTimeMs=1964\"\n\"20/05/2026, 12:41:40.109\",\"RetryHandler invoked: lastAttemptNumber=3 (max=5), totalRetryTimeMs=1846\"\n\"20/05/2026, 12:41:40.215\",\"RetryHandler invoked: lastAttemptNumber=2 (max=5), totalRetryTimeMs=1543\"\n\"20/05/2026, 12:41:40.512\",\"RetryHandler invoked: lastAttemptNumber=5 (max=5), totalRetryTimeMs=2430\"\n\"20/05/2026, 12:41:40.620\",\"RetryHandler invoked: lastAttemptNumber=2 (max=5), totalRetryTimeMs=1748\"\n\"20/05/2026, 12:41:40.698\",\"RetryHandler invoked: lastAttemptNumber=3 (max=5), totalRetryTimeMs=2227\"\n\"20/05/2026, 12:41:40.782\",\"RetryHandler invoked: lastAttemptNumber=4 (max=5), totalRetryTimeMs=2518\"\n\"20/05/2026, 12:41:41.043\",\"RetryHandler invoked: lastAttemptNumber=2 (max=5), totalRetryTimeMs=1964\"\n\"20/05/2026, 12:41:41.283\",\"RetryHandler invoked: lastAttemptNumber=3 (max=5), totalRetryTimeMs=2623\"\n\"20/05/2026, 12:41:41.425\",\"RetryHandler invoked: lastAttemptNumber=2 (max=5), totalRetryTimeMs=2154\"\n\"20/05/2026, 12:41:41.503\",\"RetryHandler invoked: lastAttemptNumber=5 (max=5), totalRetryTimeMs=3191\"\n\"20/05/2026, 12:41:41.565\",\"RetryHandler invoked: lastAttemptNumber=4 (max=5), totalRetryTimeMs=3106\"\n\"20/05/2026, 12:41:41.828\",\"RetryHandler invoked: lastAttemptNumber=2 (max=5), totalRetryTimeMs=2361\"\n\"20/05/2026, 12:41:41.899\",\"RetryHandler invoked: lastAttemptNumber=3 (max=5), totalRetryTimeMs=3029\"\n\"20/05/2026, 12:41:42.356\",\"RetryHandler invoked: lastAttemptNumber=4 (max=5), totalRetryTimeMs=3692\"\n\"20/05/2026, 12:41:42.437\",\"RetryHandler invoked: lastAttemptNumber=5 (max=5), totalRetryTimeMs=3974\"\n\"20/05/2026, 12:41:42.557\",\"RetryHandler invoked: lastAttemptNumber=3 (max=5), totalRetryTimeMs=3452\"\n\"20/05/2026, 12:41:43.109\",\"RetryHandler invoked: lastAttemptNumber=3 (max=5), totalRetryTimeMs=3834\"\n\"20/05/2026, 12:41:43.179\",\"RetryHandler invoked: lastAttemptNumber=4 (max=5), totalRetryTimeMs=4308\"\n\"20/05/2026, 12:41:43.436\",\"RetryHandler invoked: lastAttemptNumber=5 (max=5), totalRetryTimeMs=4766\"\n\"20/05/2026, 12:41:43.717\",\"RetryHandler invoked: lastAttemptNumber=3 (max=5), totalRetryTimeMs=4238\"\n\"20/05/2026, 12:41:44.034\",\"RetryHandler invoked: lastAttemptNumber=4 (max=5), totalRetryTimeMs=4965\"\n\"20/05/2026, 12:41:44.486\",\"RetryHandler invoked: lastAttemptNumber=5 (max=5), totalRetryTimeMs=5589\"\n\"20/05/2026, 12:41:44.800\",\"RetryHandler invoked: lastAttemptNumber=4 (max=5), totalRetryTimeMs=5519\"\n\"20/05/2026, 12:41:45.517\",\"RetryHandler invoked: lastAttemptNumber=5 (max=5), totalRetryTimeMs=6444\"\n\"20/05/2026, 12:41:45.596\",\"RetryHandler invoked: lastAttemptNumber=4 (max=5), totalRetryTimeMs=6125\"\n\"20/05/2026, 12:41:46.480\",\"RetryHandler invoked: lastAttemptNumber=5 (max=5), totalRetryTimeMs=7206\"\n\"20/05/2026, 12:41:47.480\",\"RetryHandler invoked: lastAttemptNumber=5 (max=5), totalRetryTimeMs=8005\"\n\"20/05/2026, 12:41:49.563\",\"RetryHandler invoked: lastAttemptNumber=10 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:49.792\",\"RetryHandler invoked: lastAttemptNumber=10 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:50.045\",\"RetryHandler invoked: lastAttemptNumber=10 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:50.203\",\"RetryHandler invoked: lastAttemptNumber=10 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:50.405\",\"RetryHandler invoked: lastAttemptNumber=10 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:50.623\",\"RetryHandler invoked: lastAttemptNumber=10 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:50.837\",\"RetryHandler invoked: lastAttemptNumber=10 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:51.017\",\"RetryHandler invoked: lastAttemptNumber=10 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:51.255\",\"RetryHandler invoked: lastAttemptNumber=10 (max=5), totalRetryTimeMs=0\"\n\"20/05/2026, 12:41:51.463\",\"RetryHandler invoked: lastAttemptNumber=10 (max=5), totalRetryTimeMs=0\"\n```\n\nThe first batch of 10 activities (indices 0–9) retries normally — `lastAttemptNumber` progresses 1→5 and `totalRetryTimeMs` rises monotonically per task. The second batch (indices 10–19) each see `lastAttemptNumber=10` with `totalRetryTimeMs=0` on their **first** failure — equal to `MAX_IN_FLIGHT`, matching the root-cause analysis above. The retry handler returns `false` (10 \u003c 5 is false), so the second batch gets zero retries and their first failure propagates. The `totalRetryTimeMs=0` confirms no real retry has happened, even though `lastAttemptNumber` already says 10.\n\n**Expected**: each task's retry handler sees `lastAttemptNumber` rising from 1 to 5 as it actually retries.\n**Actual**: tasks entering a refilled window see `lastAttemptNumber ≈ window size` on first failure — exceeding the retry budget before any real retry happens.\n\n## Impact\n\nAny custom `RetryHandler` that gates retries on `lastAttemptNumber` (the obvious choice given the docs) silently rejects retries that should fire, under common parallel fan-out patterns. We hit this in production when running 10 items in parallel via `anyOf` with a 5-attempt cap: the last items in the window saw inflated `lastAttemptNumber` on their first failure, retries were rejected, the activity failure propagated to the orchestrator, and the orchestration failed instead of recovering.\n\nWorkaround: gate on `RetryContext.getTotalRetryTime()` instead — that field is only written inside `RetriableTask.tryRetry()` after the timer await and is not affected by `initSubTasks`.\n\n## Suggested fix\n\nWithout a precise diagnosis it's premature to prescribe a code change, but the goal is clear: `RetryContext.getLastAttemptNumber()` must reflect only real handler invocations for a given task, matching `totalRetryTime`. Whatever code path bumps `attemptNumber` for a freshly-scheduled `RetriableTask` joining an active `CompoundTask` needs to be removed or guarded so that the counter starts at `1` on the first real failure for every task, regardless of when in the orchestrator's lifetime the task was scheduled.\n\nA useful invariant to enforce in tests: for every retry-handler invocation, `lastAttemptNumber == 1 || totalRetryTime > 0`. Any invocation with `lastAttemptNumber > 1` and `totalRetryTime == 0` is by definition inconsistent with the documented contract.\n\n## Environment\n\n- `com.microsoft:durabletask-client:1.9.0`\n- Azure Functions Java worker on **Flex Consumption** plan, region `norwayeast`, **Maximum instance count = 10**\n- Relevant `host.json`:\n\n ```json\n {\n \"version\": \"2.0\",\n \"functionTimeout\": \"00:10:00\",\n \"extensions\": {\n \"durableTask\": {\n \"hubName\": \"MyHub\",\n \"storageProvider\": {\n \"partitionCount\": 16\n },\n \"maxConcurrentActivityFunctions\": 10,\n \"maxConcurrentOrchestratorFunctions\": 20\n }\n },\n \"extensionBundle\": {\n \"id\": \"Microsoft.Azure.Functions.ExtensionBundle\",\n \"version\": \"[4.*, 5.0.0)\"\n }\n }\n ```\n\n","author":{"url":"https://github.com/torenedre","@type":"Person","name":"torenedre"},"datePublished":"2026-05-20T06:33:16Z","url":"https://github.com/microsoft/durabletask-java/issues/290"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:5ee89059-cc21-5d37-a4eb-61f5274d5f4d |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | C214:25139E:73812:98D5E:6AFE31BC |
| html-safe-nonce | 7ac8ce79787fc13f837b7f48ef6575bb4336c666cb6c45935638a348c61010ec |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDMjE0OjI1MTM5RTo3MzgxMjo5OEQ1RTo2QUZFMzFCQyIsInZpc2l0b3JfaWQiOiI1OTk4OTEwMDk0MDQzOTE4NzgwIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | f9a6bce891ebd135e1e8429aa116939dd97c0fea2bac2d94a387b78d6fd5634d |
| hovercard-subject-tag | issue:4483804807 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/microsoft/durabletask-java/290/issue_layout |
| twitter:image | https://opengraph.githubassets.com/aa1b65d49d8a994ccd8c692a7414a948daadb8892ac0cef5fcddcd8a66c4961a/microsoft/durabletask-java/issues/290 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/aa1b65d49d8a994ccd8c692a7414a948daadb8892ac0cef5fcddcd8a66c4961a/microsoft/durabletask-java/issues/290 |
| og:image:alt | Summary RetryContext.getLastAttemptNumber() violates its documented contract when the underlying retriable task is awaited indirectly via a CompoundTask (e.g. TaskOrchestrationContext.anyOf(...)). ... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | torenedre |
| hostname | github.com |
| expected-hostname | github.com |
| None | d70e31220a67f6376c108d2416fa8e7df854c1065b1672ab7a88ab6dc3a809e2 |
| turbo-cache-control | no-preview |
| go-import | github.com/microsoft/durabletask-java git https://github.com/microsoft/durabletask-java.git |
| octolytics-dimension-user_id | 6154722 |
| octolytics-dimension-user_login | microsoft |
| octolytics-dimension-repository_id | 452823832 |
| octolytics-dimension-repository_nwo | microsoft/durabletask-java |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 452823832 |
| octolytics-dimension-repository_network_root_nwo | microsoft/durabletask-java |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | ccb88d9c92abe69245b429398c8a57ae87070c7c |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width