Title: DuckDB delta extension install races across parallel tests · Issue #6743 · feast-dev/feast · GitHub
Open Graph Title: DuckDB delta extension install races across parallel tests · Issue #6743 · feast-dev/feast
X Title: DuckDB delta extension install races across parallel tests · Issue #6743 · feast-dev/feast
Description: Expected Behavior The DuckDB offline integration suite should run reliably with 8 pytest-xdist workers on a fresh runner. Current Behavior DuckDBDelta tests intermittently fail while several workers auto-install the delta extension into ...
Open Graph Description: Expected Behavior The DuckDB offline integration suite should run reliably with 8 pytest-xdist workers on a fresh runner. Current Behavior DuckDBDelta tests intermittently fail while several worker...
X Description: Expected Behavior The DuckDB offline integration suite should run reliably with 8 pytest-xdist workers on a fresh runner. Current Behavior DuckDBDelta tests intermittently fail while several worker...
Opengraph URL: https://github.com/feast-dev/feast/issues/6743
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"DuckDB delta extension install races across parallel tests","articleBody":"## Expected Behavior\n\nThe DuckDB offline integration suite should run reliably with 8 pytest-xdist workers on a fresh runner.\n\n## Current Behavior\n\nDuckDBDelta tests intermittently fail while several workers auto-install the `delta` extension into the shared `~/.duckdb/extensions` directory. The losing worker reports:\n\n```text\n_duckdb.IOException: IO Error: Could not set lock on file \".../delta.duckdb_extension.tmp-...duckdb_extension.info\": Conflicting lock is held in .../python3.10 (PID ...)\n```\n\nI found the same failure in 3 unrelated Linux CI runs:\n\n- https://github.com/feast-dev/feast/actions/runs/31946521769\n- https://github.com/feast-dev/feast/actions/runs/30604216643\n- https://github.com/feast-dev/feast/actions/runs/30477658759\n\nThe failures affected 2 different DuckDBDelta tests, so this is not tied to one test's behavior.\n\n## Steps to reproduce\n\n1. Start with an empty DuckDB extension cache on Linux.\n2. Install the locked `duckdb-tests` Pixi environment.\n3. Run `pixi run -e duckdb-tests test`, which executes the offline integration suite with `pytest -n 8`.\n4. Several workers may observe that `delta` is not installed and attempt to install it into the same extension directory.\n\nThe race is intermittent. I could not reproduce the lock failure on Apple Silicon macOS, including 80 synchronized cold-cache extension loads, but the 3 GitHub-hosted Linux runs above provide independent positive reproductions.\n\n### Specifications\n\n- Version: current `master` at f771ea47d\n- Platform: GitHub-hosted Linux, `linux_amd64`\n- Subsystem: DuckDB offline integration tests\n- Locked dependencies: Python 3.10.20, DuckDB 1.5.4, Ibis 12.0.0, pytest-xdist 3.8.0\n\n## Possible Solution\n\nInstall `delta` once before starting the parallel test workers. For example, add a serial Pixi task dependency that runs:\n\n```sh\npython -c \"import duckdb; duckdb.connect().install_extension('delta')\"\n```\n\nI validated this against a separate cold cache. A fresh Ibis connection reported `delta` as installed but not loaded, then `load_extension(\"delta\")` changed only the loaded state. This avoids Ibis's per-worker `INSTALL` path. The full 8-worker Feast suite then passed with 84 tests passed and 5 skipped.\n\nWorker-specific extension directories would also avoid contention, but would download and install the same extension once per worker.\n","author":{"url":"https://github.com/larrysingleton007","@type":"Person","name":"larrysingleton007"},"datePublished":"2026-08-16T12:44:47Z","url":"https://github.com/feast-dev/feast/issues/6743"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:3497e783-987b-d84f-0432-b4ff5b63988a |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | D8A0:2893E9:2C18A0:3CB496:6AFE980C |
| html-safe-nonce | 995bd7473848e46c3f139ad2d1bc96f29280b3e23fb25c1539b4cbdbc6d42b0e |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEOEEwOjI4OTNFOToyQzE4QTA6M0NCNDk2OjZBRkU5ODBDIiwidmlzaXRvcl9pZCI6IjQzOTI0NTA3MDIwNjQwODcxNiIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | fbfdf5f49117e25db18b4f143079a962b3f885b23e28c4ff636626f3d8b41a4c |
| hovercard-subject-tag | issue:5164307176 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/6743/issue_layout |
| twitter:image | https://opengraph.githubassets.com/7100e792b4c322090254f1256a3be13d67e18ba5fe9f5d648c3d13fd60d7372a/feast-dev/feast/issues/6743 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/7100e792b4c322090254f1256a3be13d67e18ba5fe9f5d648c3d13fd60d7372a/feast-dev/feast/issues/6743 |
| og:image:alt | Expected Behavior The DuckDB offline integration suite should run reliably with 8 pytest-xdist workers on a fresh runner. Current Behavior DuckDBDelta tests intermittently fail while several worker... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | larrysingleton007 |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width