Title: Add durabletask.extensions.history_export by andystaples · Pull Request #147 · microsoft/durabletask-python · GitHub
Open Graph Title: Add durabletask.extensions.history_export by andystaples · Pull Request #147 · microsoft/durabletask-python
X Title: Add durabletask.extensions.history_export by andystaples · Pull Request #147 · microsoft/durabletask-python
Description: A new optional extension for exporting the event history of terminal orchestrations to an external destination (Azure Blob Storage out of the box; custom destinations via a one-method HistoryWriter protocol). Modeled after the .NET SDK's ExportHistory package and at functional parity with it for both Batch and Continuous modes. NoteBuilds on the core retrieval APIs (get_orchestration_history, list_instance_ids) added in PR #133. No core SDK changes in this PR. Why Operators frequently need to archive, audit, or analyse orchestration histories outside the task hub. Today there is no first-party way to do that in Python. The .NET SDK ships ExportHistory for this purpose; this PR provides the Python equivalent. What ships A new optional extension package: durabletask/extensions/history_export/ ├── __init__.py # public re-exports ├── _constants.py # ENTITY_NAME, ORCHESTRATOR_NAME, deterministic-ID helper ├── _logging.py # module-wide logger ├── activities.py # list_terminal_instances, export_instance_history ├── azure_blob.py # bundled writer (behind pip install durabletask[history-export-azure]) ├── client.py # ExportHistoryClient + ExportHistoryJobClient ├── entity.py # ExportJobEntity (state machine) ├── exceptions.py # ExportJobError + InvalidTransition / NotFound subclasses ├── models.py # dataclasses + enums (versioned, JSON-primitive) ├── orchestrator.py # export_job_orchestrator (driver) ├── serialization.py # JSON / JSONL_GZIP + OrchestrationState metadata ├── transitions.py # state-transition matrix └── writer.py # HistoryWriter Protocol Three architectural layers User-facing client (the 95% surface). ExportHistoryClient.create_job, get_job, list_jobs, wait_for_job, delete_job; plus a per-job ExportHistoryJobClient returned by get_job_client(job_id). Durable workflow. ExportJobEntity is the state machine — owns status (PENDING / ACTIVE / COMPLETED / FAILED), config, checkpoint cursor, counters, and accumulated ExportFailure records. All status changes go through a declarative transition matrix; illegal transitions raise ExportJobInvalidTransitionError. The entity schedules its own driver orchestrator in its run op with a deterministic instance ID (export-job-{job_id}). export_job_orchestrator is the driver — pages through terminal instances, fans out per-instance exports (capped by max_parallel_exports, retried by a per-activity RetryPolicy), signals commit_checkpoint with page totals, and continue-as-new's every 5 page cycles to keep its own history bounded. Re-fetches entity state at the top of every iteration so external delete_job / mark_failed signals stop the orchestrator cleanly. Two activities: list_terminal_instances (wraps list_instance_ids) and export_instance_history (fetches history + OrchestrationState, serializes, hands off to the writer). Pluggable destination. HistoryWriter is a typing.Protocol — one method write(*, instance_id, blob_name, payload, content_type, content_encoding). Bundled AzureBlobHistoryExportWriter (behind the history-export-azure extra). No inheritance required for custom destinations. Modes Mode Semantics ExportMode.BATCH Process a fixed completed_time_from .. _to window, then mark_completed. ExportMode.CONTINUOUS Tail terminal instances indefinitely; sleep CONTINUOUS_IDLE_DELAY on empty pages. Stops only via external delete_job or mark_failed. Output format Each exported instance becomes one self-describing blob: Format (ExportFormatKind) Extension Content-Type Content-Encoding Shape JSON .json application/json (none) {schema_version, instance_id, metadata: {OrchestrationState}, events: [...]} JSONL_GZIP .jsonl.gz application/x-ndjson gzip metadata line first, then one event per line, gzipped Both formats are byte-deterministic (sorted keys, mtime=0 gzip header). Safe persisted state Persisted entity state goes through the SDK's default JSON encoder, which only handles JSON primitives. To avoid embedding Python class metadata in the persisted payload (a known deserialization-attack vector), the on-disk shape is owned by ExportJobState, a versioned dataclass with explicit _to_dict / _from_dict methods. Every persisted dict carries an explicit schema_version (currently "1.0"); unknown versions are rejected at load time. No __class__, __type__, or module-path fields appear in the serialized form. Distributed deployment A single dedicated export-worker process can serve a task hub of any size without interfering with the workers running live orchestrations: Workers only pull work items whose names they have registered. App workers that don't call export_client.register_worker(w) never see the export entity, orchestrator, or activities. The export worker reads history via the standard TaskHubGrpcClient.get_orchestration_history RPC — backend-mediated, so it can export anything in the hub. Scale export throughput horizontally by adding more export workers; the durable entity + deterministic orchestrator instance ID ensure exactly one driver runs per job at any moment. Dependency model Core extension has no third-party runtime deps. from durabletask.extensions.history_export import ExportHistoryClient works in a clean environment. The bundled Azure writer is opt-in: pip install durabletask[history-export-azure]. Same gating pattern as azure_blob_payloads. Per-backend naming (history-export-azure) leaves room for future first-party destinations (history-export-s3, etc.) without ambiguity. Public surface Importable from durabletask.extensions.history_export: # Clients ExportHistoryClient, ExportHistoryJobClient # Extension points HistoryWriter, HistoryExportContext # Configuration / read view ExportJobCreationOptions, ExportJobConfiguration ExportJobDescription, ExportJobState, ExportJobQuery ExportDestination, ExportFilter, ExportCheckpoint, ExportFailure ExportFormat, ExportFormatKind, ExportMode, ExportJobStatus # Errors ExportJobError, ExportJobInvalidTransitionError, ExportJobNotFoundError # Constants / helpers STATE_SCHEMA_VERSION, ENTITY_NAME, ORCHESTRATOR_NAME orchestrator_instance_id_for The bundled Azure writer is in the optional submodule: from durabletask.extensions.history_export.azure_blob import ( AzureBlobHistoryExportWriter, AzureBlobHistoryExportWriterOptions, ) Docs and example New "Orchestration history export" section in docs/features.md covering installation, configuration, output formats, modes, listing/managing jobs, and custom destinations (with a worked local-filesystem example). New runnable example at examples/history_export/app.py that uses the in-memory backend + Azurite (no Azure resources required). Changelog Single user-facing entry under ## Unreleased in CHANGELOG.md describing the new public extension and its capabilities. Out of scope (deliberately deferred) The following items would be additive follow-ups; none would change the public contract introduced here: First-party S3 / GCS / local-filesystem writers. This PR contains the HistoryWriter Protocol as the extension point; with Azure writer as the only concrete Structured logging helpers analogous to .NET's logger.ExportJobOperationInfo / Warning / Error. Today we use the standard logger directly.
Open Graph Description: A new optional extension for exporting the event history of terminal orchestrations to an external destination (Azure Blob Storage out of the box; custom destinations via a one-method HistoryWriter...
X Description: A new optional extension for exporting the event history of terminal orchestrations to an external destination (Azure Blob Storage out of the box; custom destinations via a one-method HistoryWriter...
Opengraph URL: https://github.com/microsoft/durabletask-python/pull/147
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:7a813e5c-c90e-c879-fc87-0cbb94f4f02f |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | E850:D6613:5BE2F:7CBDF:6AFE3F82 |
| html-safe-nonce | 5c34b8c80fdf6b6109ba1700ea2910300d67256c78435b05920d0f095d69a4f2 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFODUwOkQ2NjEzOjVCRTJGOjdDQkRGOjZBRkUzRjgyIiwidmlzaXRvcl9pZCI6IjQ3NzIxODQ2OTYwNTYzMzIxNjIiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | a9219ff24aa146172299fd77f3266a773c27743dee3762c2d531ea4af15c8b82 |
| hovercard-subject-tag | pull_request:3797037233 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/microsoft/durabletask-python/pull/147/files |
| twitter:image | https://avatars.githubusercontent.com/u/77818326?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/77818326?s=400&v=4 |
| og:image:alt | A new optional extension for exporting the event history of terminal orchestrations to an external destination (Azure Blob Storage out of the box; custom destinations via a one-method HistoryWriter... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | f1a5901e2377b0af2a0b303f9528678f1475d04b9219dca1e993344196eaa07c |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/microsoft/durabletask-python git https://github.com/microsoft/durabletask-python.git |
| octolytics-dimension-user_id | 6154722 |
| octolytics-dimension-user_login | microsoft |
| octolytics-dimension-repository_id | 627664515 |
| octolytics-dimension-repository_nwo | microsoft/durabletask-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 627664515 |
| octolytics-dimension-repository_network_root_nwo | microsoft/durabletask-python |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | c4447fdce9ed0f33af959d2b6680d0bc18212a6a |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width