Title: fix(pii): bound custom-pattern matching time per request by waleedlatif1 · Pull Request #9584 · simstudioai/sim · GitHub
Open Graph Title: fix(pii): bound custom-pattern matching time per request by waleedlatif1 · Pull Request #9584 · simstudioai/sim
X Title: fix(pii): bound custom-pattern matching time per request by waleedlatif1 · Pull Request #9584 · simstudioai/sim
Description: Summary Presidio's REGEX_TIMEOUT_SECONDS (2s) bounds one custom pattern on one text, so a single batch request could cost texts x patterns x 2s (2,000 texts x 20 patterns = ~22h on one handler thread). A pathological pattern like (a|aa)+$ hits the per-match cap on every text, and the sync handler thread keeps running after the client gives up, so each retry ties up another thread Custom-pattern recognizers now charge their matching time to a shared per-request 60s budget and reject the request with a 400 once it's spent. The cost of one request is now at most budget + one match The budget counts only custom-pattern time, not spaCy NER or built-in recognizers, so slow legitimate batches can't trip it. Benign patterns use well under 1s per 256 KB chunk, and a single text with 20 timed-out patterns stays under the budget, so single-text behavior is unchanged A 400 maps to PiiServiceRejectedError → route 422 → no retry in mask-client, and it isn't treated as an outage, so the chunk fails closed without retry amplification Updated the validate-regex.ts TSDoc that named Presidio's timeout as the only bound Type of Change Bug fix Testing Drove apps/pii/server.py with the pinned Presidio 2.2.362 / spaCy 3.8.14 / FastAPI stack through FastAPI TestClient (en_core_web_sm in place of the 5 large models), pre-fix vs post-fix: Report shape (2,000 texts x 20 (a|aa)+$ patterns, timeout 2s): post-fix /redact_batch (regex-only and full NLP paths) and /analyze_batch all return 400 in 60.4-63.0s Across the budget line (120 texts x 1 pattern, timeout 1s): pre-fix 200 after 121-165s, post-fix 400 after 60.2-60.3s Single text with a pathological pattern: /redact and /analyze still return 200 after one ~2s timeout, same as before Benign regression (2,000 realistic texts x 20 benign patterns + built-ins, all 6 endpoints incl. invalid-regex 400): pre vs post outputs are identical under a pinned PYTHONHASHSEED on 6 seeds. The only run-to-run variation is the winner of exact equal-score span ties, which varies by seed on the unmodified server too (Presidio dedups via set()) biome check on the TS file; JS gates otherwise left to CI (the diff is Python + one TSDoc sentence) Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing (new tests pass the test-audit authoring gate) No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)
Open Graph Description: Summary Presidio's REGEX_TIMEOUT_SECONDS (2s) bounds one custom pattern on one text, so a single batch request could cost texts x patterns x 2s (2,000 texts x 20 patterns = ~22h on one handler...
X Description: Summary Presidio's REGEX_TIMEOUT_SECONDS (2s) bounds one custom pattern on one text, so a single batch request could cost texts x patterns x 2s (2,000 texts x 20 patterns = ~22h on one han...
Opengraph URL: https://github.com/simstudioai/sim/pull/9584
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:eb166635-a1d4-f611-e83e-0574573e04a2 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | C06A:2EC0A1:B278D:106E7C:6AFB0BC0 |
| html-safe-nonce | 6fd355ef50f51e61123c0d803139b2a1cb4f3d38840feb2555845c073f32a034 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDMDZBOjJFQzBBMTpCMjc4RDoxMDZFN0M6NkFGQjBCQzAiLCJ2aXNpdG9yX2lkIjoiNTE1OTgxMTY1MzUwMzIyNDc2OCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 2d9357c2100befbf198d17513edc0d67112b4edc8b70f3dec45fd432821fb586 |
| hovercard-subject-tag | pull_request:5127195388 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/9584/files |
| twitter:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| og:image:alt | Summary Presidio's REGEX_TIMEOUT_SECONDS (2s) bounds one custom pattern on one text, so a single batch request could cost texts x patterns x 2s (2,000 texts x 20 patterns = ~22h on one handler... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 4275ec98d19a530a209480d02854be81de24ad5a05b648333242fc516bfb573c |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | ba13208acc32a4e7a4a3b54cb2f0dea6001ba5f8 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width