Title: RetryPolicy max_retry_interval does not prevent backoff overflow, failing capped retries · Issue #367 · microsoft/durabletask-python · GitHub
Open Graph Title: RetryPolicy max_retry_interval does not prevent backoff overflow, failing capped retries · Issue #367 · microsoft/durabletask-python
X Title: RetryPolicy max_retry_interval does not prevent backoff overflow, failing capped retries · Issue #367 · microsoft/durabletask-python
Description: Summary RetryPolicy.max_retry_interval does not prevent exponential-backoff overflow. RetryableTask.compute_next_delay() computes math.pow(coefficient, attempt - 1) before applying the configured cap. Once the intermediate power overflow...
Open Graph Description: Summary RetryPolicy.max_retry_interval does not prevent exponential-backoff overflow. RetryableTask.compute_next_delay() computes math.pow(coefficient, attempt - 1) before applying the configured c...
X Description: Summary RetryPolicy.max_retry_interval does not prevent exponential-backoff overflow. RetryableTask.compute_next_delay() computes math.pow(coefficient, attempt - 1) before applying the configured c...
Opengraph URL: https://github.com/microsoft/durabletask-python/issues/367
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"RetryPolicy max_retry_interval does not prevent backoff overflow, failing capped retries","articleBody":"## Summary\n\n`RetryPolicy.max_retry_interval` does not prevent exponential-backoff overflow. `RetryableTask.compute_next_delay()` computes `math.pow(coefficient, attempt - 1)` before applying the configured cap. Once the intermediate power overflows, the worker fails the orchestration with `OverflowError` instead of continuing bounded retries or delivering the original exhausted failure to the orchestrator.\n\nThis reproduces with the normal coefficient `2.0`, a 1ms maximum interval and 1,027 maximum attempts: the orchestration fails after 1,025 activity calls. The cap has already made the delay constant, but the unused intermediate power keeps growing. A large coefficient below accelerates the same failure to three calls.\n\nThis is a local SDK reproduction using the public in-memory testing backend, client and worker. No Azure account or service was used, and no production incident is claimed.\n\n## Reproduction\n\nRun against core `durabletask` 1.13.1 from current main `256ffbdca1fb97f5ed96e547810b5d63ebefaabe` with its runtime dependencies installed:\n\n```python\nimport logging\nimport socket\nfrom datetime import timedelta\nfrom durabletask import client, task, worker\nfrom durabletask.testing import create_test_backend\nlogging.basicConfig(level=logging.ERROR)\nwith socket.socket() as s:\n s.bind(('127.0.0.1',0))\n port=s.getsockname()[1]\nhost=f'localhost:{port}'\nb=create_test_backend(port=port)\ncalls=[]\ndef fail(ctx,_):\n calls.append(True)\n raise ValueError('downstream unavailable')\ndef orchestration(ctx,_):\n policy=task.RetryPolicy(first_retry_interval=timedelta(milliseconds=1),max_number_of_attempts=5,\n backoff_coefficient=1e200,max_retry_interval=timedelta(milliseconds=1))\n try:\n yield ctx.call_activity(fail,retry_policy=policy)\n except task.TaskFailedError:\n return 'handled expected downstream failure'\ntry:\n with worker.TaskHubGrpcWorker(host_address=host) as w:\n w.add_orchestrator(orchestration)\n w.add_activity(fail)\n w.start()\n with client.TaskHubGrpcClient(host_address=host) as c:\n id=c.schedule_new_orchestration(orchestration)\n state=c.wait_for_orchestration_completion(id,timeout=10)\n print('status:',state.runtime_status)\n print('activity calls:',len(calls))\n print('output:',state.serialized_output)\n print('failure:',state.failure_details)\nfinally:\n b.stop()\n b.reset()\n```\n\nActual: `status: FAILED`, `activity calls: 3`, no output, and failure type `builtins.OverflowError` with message `math range error` at `math.pow(...)` in `compute_next_delay()`.\n\nExpected: five calls at delays no greater than 1ms, followed by the orchestrator's `TaskFailedError` handler returning `handled expected downstream failure` and a `COMPLETED` instance. With `backoff_coefficient=1.0`, that control does complete with five calls.\n\nFor the normal-coefficient case, change `backoff_coefficient` to `2.0`, `max_number_of_attempts` to `1027` and the completion timeout to `60`. I ran this separately on Python 3.13: it fails after 1,025 calls with the same SDK overflow rather than reaching the configured attempt limit. The local run completed in about 31 seconds; timing is not a benchmark or a guarantee.\n\n## Validation\n\nLinux x86_64, CPython 3.11 and 3.13, independently:\n\n- Unmodified native orchestration-executor, task-composition, retry-timeout and failure-details suites: **344 passed** on each interpreter.\n- Twelve focused delay-calculation cases: **8 fail, 4 pass as controls**. Failing cases include coefficients 2, 10, 1000 and the accelerated large coefficient, with positive and zero caps.\n- Four public client/worker cases: **2 fail, 2 pass as controls**. Both activity and sub-orchestration retries fail for the accelerated overflow case; the constant-backoff versions exhaust normally and are caught by the parent.\n- The 1,025-call coefficient-2 public activity run was executed on Python 3.13 only.\n\nNo full repository/provider suite, hosted CI, Azure emulator or live-service validation is claimed. No production fix has been written or submitted.\n\n## Scope and compatibility\n\nI checked #96/#116 (absent maximum interval) and #316/#326 (retry-timeout logical clock); neither fixes capped-power overflow. The current open async stack retains this delay calculation too. The intended scope would be to avoid evaluating an already-saturated power while preserving attempt limits, timeout checks and ordinary delay rounding.\n\nCorrecting the failure can change replay behavior for histories that previously terminated with this overflow, so any eventual patch needs an explicit compatibility decision and upgrade guidance. I will wait for maintainer triage/discussion before preparing a PR. Would this focused capped-backoff fix be welcome?\n\nInvestigation and reproduction tests developed with AI assistance. The reproduction runs above were executed and their outputs inspected.","author":{"url":"https://github.com/Rajatendu1","@type":"Person","name":"Rajatendu1"},"datePublished":"2026-11-17T08:41:02Z","url":"https://github.com/microsoft/durabletask-python/issues/367"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:3b3d7652-850f-4061-f473-1ea04e80c283 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | A7D8:355A02:676CC:8AED2:6AFE1E85 |
| html-safe-nonce | 3aab22c5ba1a390a273a29ad6c544cd3d2f2984a9a97a8dd8d23fc4adc488063 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBN0Q4OjM1NUEwMjo2NzZDQzo4QUVEMjo2QUZFMUU4NSIsInZpc2l0b3JfaWQiOiI3MDIxMTE5NzA2NDI1NTk3NTczIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 243e00e4de00240ac1cfc35ea5e2ddc5c1f068667a53950a2c0913d7ceaa4879 |
| hovercard-subject-tag | issue:6440716844 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/microsoft/durabletask-python/367/issue_layout |
| twitter:image | https://opengraph.githubassets.com/e2d340023d6f310ecc9af1efacd8a717f105985ee9354e3ffdbc8b8d0c62c7ce/microsoft/durabletask-python/issues/367 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/e2d340023d6f310ecc9af1efacd8a717f105985ee9354e3ffdbc8b8d0c62c7ce/microsoft/durabletask-python/issues/367 |
| og:image:alt | Summary RetryPolicy.max_retry_interval does not prevent exponential-backoff overflow. RetryableTask.compute_next_delay() computes math.pow(coefficient, attempt - 1) before applying the configured c... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | Rajatendu1 |
| hostname | github.com |
| expected-hostname | github.com |
| None | d0525ab1464526bf4532a5551aa348cf100dff75ef808a96fc67a08b1d946aa9 |
| turbo-cache-control | no-preview |
| go-import | github.com/microsoft/durabletask-python git https://github.com/microsoft/durabletask-python.git |
| octolytics-dimension-user_id | 6154722 |
| octolytics-dimension-user_login | microsoft |
| octolytics-dimension-repository_id | 627664515 |
| octolytics-dimension-repository_nwo | microsoft/durabletask-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 627664515 |
| octolytics-dimension-repository_network_root_nwo | microsoft/durabletask-python |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | fef3993a71fa20f7be8e167132fbf9619bc66e12 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width