Title: RetryableTask.compute_next_delay uses wall-clock time causing non-deterministic failures on replay with retry_timeout · Issue #316 · microsoft/durabletask-python · GitHub
Open Graph Title: RetryableTask.compute_next_delay uses wall-clock time causing non-deterministic failures on replay with retry_timeout · Issue #316 · microsoft/durabletask-python
X Title: RetryableTask.compute_next_delay uses wall-clock time causing non-deterministic failures on replay with retry_timeout · Issue #316 · microsoft/durabletask-python
Description: Problem Description When an activity or sub-orchestration is scheduled with a RetryPolicy that specifies a retry_timeout, RetryableTask.compute_next_delay() checks if the retry deadline has passed using system wall-clock time (datetime.n...
Open Graph Description: Problem Description When an activity or sub-orchestration is scheduled with a RetryPolicy that specifies a retry_timeout, RetryableTask.compute_next_delay() checks if the retry deadline has passed ...
X Description: Problem Description When an activity or sub-orchestration is scheduled with a RetryPolicy that specifies a retry_timeout, RetryableTask.compute_next_delay() checks if the retry deadline has passed ...
Opengraph URL: https://github.com/microsoft/durabletask-python/issues/316
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"RetryableTask.compute_next_delay uses wall-clock time causing non-deterministic failures on replay with retry_timeout","articleBody":"### Problem Description\n\nWhen an activity or sub-orchestration is scheduled with a `RetryPolicy` that specifies a `retry_timeout`, `RetryableTask.compute_next_delay()` checks if the retry deadline has passed using system wall-clock time (`datetime.now(tz=UTC).replace(tzinfo=None)`):\n\nhttps://github.com/microsoft/durabletask-python/blob/97427b24b4f80733334e3f8985b9bde339605d42/durabletask/task.py#L829-L850\n\n```python\n def compute_next_delay(self) -> timedelta | None:\n if self._attempt_count >= self._retry_policy.max_number_of_attempts:\n return None\n\n retry_expiration: datetime = datetime.max\n if self._retry_policy.retry_timeout is not None and self._retry_policy.retry_timeout != datetime.max:\n retry_expiration = self._start_time + self._retry_policy.retry_timeout\n\n if self._retry_policy.backoff_coefficient is None:\n backoff_coefficient = 1.0\n else:\n backoff_coefficient = self._retry_policy.backoff_coefficient\n\n if datetime.now(tz=UTC).replace(tzinfo=None) \u003c retry_expiration:\n next_delay_f = math.pow(backoff_coefficient, self._attempt_count - 1) * self._retry_policy.first_retry_interval.total_seconds()\n\n if self._retry_policy.max_retry_interval is not None:\n next_delay_f = min(next_delay_f, self._retry_policy.max_retry_interval.total_seconds())\n\n return timedelta(seconds=next_delay_f)\n\n return None\n```\n\nThis leads to two critical issues:\n\n1. **Replay Non-determinism**: In Durable Task Framework, orchestrations must be deterministic and replay-safe. Orchestration histories are replayed whenever an instance resumes (e.g. after a worker restart, crash recovery, server failover, long timer expiration, or external event arrival). Because `compute_next_delay()` evaluates the current system wall-clock time (`datetime.now()`) instead of the orchestration's logical time (`ctx.current_utc_datetime` or the history event timestamp), any replay that occurs after `_start_time + retry_timeout` has elapsed in real time will evaluate `datetime.now() \u003c retry_expiration` to `False` and return `None`.\n - Consequently, in `worker.py` (lines 2876-2881 and 2987-2992), the worker erroneously treats retries as expired, marks the task as failed via `fail()`, and completes the orchestration with `ORCHESTRATION_STATUS_FAILED` rather than scheduling the retry timer. This causes replay divergence and crashes valid in-flight orchestrations.\n2. **Offset-naive vs Offset-aware `TypeError`**: `datetime.now(tz=UTC).replace(tzinfo=None)` strips timezone info. If `_start_time` is timezone-aware (e.g. if `current_utc_datetime` is set with a timezone or in custom providers), `retry_expiration` is offset-aware. Comparing `check \u003c retry_expiration` crashes with `TypeError: can't compare offset-naive and offset-aware datetimes`.\n\n---\n\n### Minimal Reproduction\n\n#### 1. Replay Divergence / Failure\nRun the following script simulating an orchestration history that was recorded 2 minutes ago where an activity with a 10s `retry_timeout` failed after 1 second:\n\n```python\nfrom datetime import datetime, timedelta\nimport logging\nfrom durabletask import task, worker\nfrom durabletask.serialization import JsonDataConverter\nimport durabletask.internal.helpers as helpers\n\ndef dummy_activity(ctx, input):\n return \"result\"\n\ndef orchestrator(ctx, _):\n res = yield ctx.call_activity(\n dummy_activity,\n retry_policy=task.RetryPolicy(\n first_retry_interval=timedelta(seconds=1),\n max_number_of_attempts=5,\n retry_timeout=timedelta(seconds=10),\n ),\n )\n return res\n\nregistry = worker._Registry()\nname = registry.add_orchestrator(orchestrator)\nregistry.add_activity(dummy_activity)\n\n# Execution originally started 2 minutes ago:\nt0 = datetime.utcnow() - timedelta(minutes=2)\nt1 = t0 + timedelta(seconds=1)\n\nold_history = [\n helpers.new_orchestrator_started_event(timestamp=t0),\n helpers.new_execution_started_event(name, \"test-inst\", encoded_input=None),\n helpers.new_task_scheduled_event(1, task.get_name(dummy_activity)),\n]\nnew_events = [\n helpers.new_orchestrator_started_event(timestamp=t1),\n helpers.new_task_failed_event(1, ValueError(\"Transient error\")),\n]\n\nexecutor = worker._OrchestrationExecutor(registry, logging.getLogger(), JsonDataConverter())\nres = executor.execute(\"test-inst\", old_history, new_events)\n\naction_type = res.actions[0].WhichOneof(\"orchestratorActionType\")\nprint(f\"Action produced on replay: {action_type}\")\n# Output:\n# Action produced on replay: completeOrchestration\n# BUG: Expected 'createTimer' (since only 1s of the 10s timeout had elapsed), but got 'completeOrchestration'!\n```\n\n#### 2. Timezone Comparison Crash\n```python\nfrom datetime import datetime, timezone, timedelta\nfrom durabletask import task\n\nrp = task.RetryPolicy(\n first_retry_interval=timedelta(seconds=1),\n max_number_of_attempts=5,\n retry_timeout=timedelta(seconds=30),\n)\nt = task.RetryableTask(\n retry_policy=rp,\n action=None,\n start_time=datetime.now(timezone.utc),\n is_sub_orch=False,\n)\nt.compute_next_delay()\n# Raises: TypeError: can't compare offset-naive and offset-aware datetimes\n```\n\n---\n\n### Expected Behavior\n\n1. `compute_next_delay` should evaluate retry expiration against the orchestration's deterministic logical clock (`ctx.current_utc_datetime` or event timestamp) rather than system wall-clock time, ensuring replay reproducibility.\n2. Timezone-aware start timestamps should be handled seamlessly without `TypeError`.\n\n---\n\n### Suggested Direction\n\n1. Accept an optional `current_time: datetime | None = None` parameter in `RetryableTask.compute_next_delay()`:\n ```python\n def compute_next_delay(self, current_time: datetime | None = None) -> timedelta | None:\n if self._attempt_count >= self._retry_policy.max_number_of_attempts:\n return None\n\n retry_expiration: datetime = datetime.max\n if self._retry_policy.retry_timeout is not None and self._retry_policy.retry_timeout != datetime.max:\n retry_expiration = self._start_time + self._retry_policy.retry_timeout\n\n if self._retry_policy.backoff_coefficient is None:\n backoff_coefficient = 1.0\n else:\n backoff_coefficient = self._retry_policy.backoff_coefficient\n\n check_time = current_time if current_time is not None else datetime.now(tz=UTC)\n if retry_expiration != datetime.max:\n if retry_expiration.tzinfo is None and check_time.tzinfo is not None:\n check_time = check_time.replace(tzinfo=None)\n elif retry_expiration.tzinfo is not None and check_time.tzinfo is None:\n check_time = check_time.replace(tzinfo=UTC)\n\n if check_time \u003c retry_expiration:\n next_delay_f = (\n math.pow(backoff_coefficient, self._attempt_count - 1)\n * self._retry_policy.first_retry_interval.total_seconds()\n )\n if self._retry_policy.max_retry_interval is not None:\n next_delay_f = min(next_delay_f, self._retry_policy.max_retry_interval.total_seconds())\n return timedelta(seconds=next_delay_f)\n\n return None\n ```\n2. In `worker.py` (lines 2875 and 2986), pass `ctx.current_utc_datetime` when calling `compute_next_delay`:\n ```python\n next_delay = retryable_activity_task.compute_next_delay(ctx.current_utc_datetime)\n ```\n and\n ```python\n next_delay = retryable_sub_orch_task.compute_next_delay(ctx.current_utc_datetime)\n ```\n","author":{"url":"https://github.com/hardikkaurani","@type":"Person","name":"hardikkaurani"},"datePublished":"2026-10-31T22:19:45Z","url":"https://github.com/microsoft/durabletask-python/issues/316"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:e0dba658-36fb-762e-938b-ff308fa7df40 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | ACD4:306EC8:C34972:112CA8A:6AFE2CF2 |
| html-safe-nonce | 75646283d34950140e2ca3b37688f3a6ec26f38440f1ce88f82976f51f5d23fd |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBQ0Q0OjMwNkVDODpDMzQ5NzI6MTEyQ0E4QTo2QUZFMkNGMiIsInZpc2l0b3JfaWQiOiI0MDg5ODc5OTcyMzAxMTIyODAzIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | bd58f691e02fa723590a6af6b14807d342794b46e690dcd034d36001a0398b4c |
| hovercard-subject-tag | issue:6140424763 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/microsoft/durabletask-python/316/issue_layout |
| twitter:image | https://opengraph.githubassets.com/9ff07726bb04b0a6e5576fbb02d568216f1f2c5d5d2bf9ad13b919a3573be533/microsoft/durabletask-python/issues/316 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/9ff07726bb04b0a6e5576fbb02d568216f1f2c5d5d2bf9ad13b919a3573be533/microsoft/durabletask-python/issues/316 |
| og:image:alt | Problem Description When an activity or sub-orchestration is scheduled with a RetryPolicy that specifies a retry_timeout, RetryableTask.compute_next_delay() checks if the retry deadline has passed ... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | hardikkaurani |
| hostname | github.com |
| expected-hostname | github.com |
| None | e973f9ec3625f3faad8cac5c2d11a214b7ffe45f3cc563b828459b290ecaeed3 |
| turbo-cache-control | no-preview |
| go-import | github.com/microsoft/durabletask-python git https://github.com/microsoft/durabletask-python.git |
| octolytics-dimension-user_id | 6154722 |
| octolytics-dimension-user_login | microsoft |
| octolytics-dimension-repository_id | 627664515 |
| octolytics-dimension-repository_nwo | microsoft/durabletask-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 627664515 |
| octolytics-dimension-repository_network_root_nwo | microsoft/durabletask-python |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 1f92c367595b9190e0be293896078033886b1913 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width