René's URL Explorer Experiment


Title: feat(pii): build & own combined PII (analyzer + anonymizer) image by TheodoreSpeaks · Pull Request #5176 · simstudioai/sim · GitHub

Open Graph Title: feat(pii): build & own combined PII (analyzer + anonymizer) image by TheodoreSpeaks · Pull Request #5176 · simstudioai/sim

X Title: feat(pii): build & own combined PII (analyzer + anonymizer) image by TheodoreSpeaks · Pull Request #5176 · simstudioai/sim

Description: Summary Build and own a single Presidio image (docker/pii.Dockerfile) replacing the stock mcr.microsoft.com/presidio-analyzer + presidio-anonymizer sidecars. A thin FastAPI service (docker/pii/server.py) builds one AnalyzerEngine + one AnonymizerEngine at startup and serves both on one port (3000): /health, /supportedentities, /analyze, /anonymize (stock-compatible shapes) — so the app needs one PRESIDIO_URL. Multi-language (en/es/it/pl/fi) via a NlpEngineProvider with five pinned spaCy lg models: en_core_web_lg + es/it/pl/fi_core_news_lg, all 3.8.0. Pinned presidio==2.2.362, spacy==3.8.14, fastapi/uvicorn on python:3.12-slim-bookworm, non-root. Models fetched with curl retry/resume (direct pip wheel URLs truncate). Native VIN recognizer (ISO-3779 check digit) ported from validate_pii.py, registered under all five languages (it's language-agnostic). Plus explicit registration of recognizers Presidio ships but does NOT load by default (confirmed: they don't auto-load even with the NLP model present): en locale ids: UK_NINO, AU_ABN/ACN/TFN/MEDICARE, IN_PAN/AADHAAR/VEHICLE_REGISTRATION/VOTER/PASSPORT, SG_NRIC_FIN, SG_UEN national ids: ES_NIF/NIE (es), IT_FISCAL_CODE/DRIVER_LICENSE/VAT_CODE/PASSPORT/IDENTITY_CARD (it), PL_PESEL (pl), FI_PERSONAL_IDENTITY_CODE (fi) Wire CI: ci.yml (build-dev, build-amd64, ghcr-arm64, manifests + both ECR ternaries) and images.yml; new secret ECR_PII, GHCR ghcr.io/simstudioai/pii. AMD64 for staging/prod, ARM64+manifest on main — mirrors app/db/realtime. Verification (full 5-model image built + run locally) Image: 6.07 GB. Cold start: ~9s (5 lg models load at import). Warm RAM: ~3.3 GiB idle. VIN: detected under en AND es (the multi-language fix); invalid check digit rejected. National ids fire in-language: ES_NIF, IT_FISCAL_CODE, PL_PESEL (checksum-validated), FI_PERSONAL_IDENTITY_CODE all detect with score 1.0; absent before explicit registration. Per-language /supportedentities returns the expected sets (en includes VIN + UK/AU/IN/SG; es/it/pl/fi include VIN + their national ids; global pattern recognizers — email/phone/IP/IBAN/etc. — present in every language automatically). Note: NER-derived labels (PERSON/LOCATION/ORGANIZATION/AGE/ID) vary per model. Contract: missing text → 422 (not 500); /anonymize default → . bun run lint + bun run check:api-validation:strict pass; both workflow YAMLs parse. Review feedback addressed (Bugbot + Greptile) VIN registered per-language (was en-only → missed under non-English routing). HEALTHCHECK --start-period 40s → 180s (measured cold start ~9s locally, but generous for slower ECS CPUs). Dropped --no-cache-dir so the pip cache mount works. Pydantic request models → missing text returns 422; operator type defaults to replace instead of KeyError→500. Required infra companion changes (NOT in this PR — infra repo) Create ECR repo sim/pii; add GitHub secret ECR_PII=sim/pii (dev/staging/prod accounts). deployment-templates/app/taskdef.json: collapse the two presidio-* sidecars into one container pii on the new ECR image, single containerPort: 3000, health /health; drop 5001/5002. Sizing: image is ~6 GB and idles at ~3.3 GiB RAM with five lg models — set container memory accordingly (≥4 GiB reservation, headroom for load) and bump ephemeral storage for the image. App container env: replace PRESIDIO_ANALYZER_URL/PRESIDIO_ANONYMIZER_URL with PRESIDIO_URL=http://localhost:3000. App follow-ups (separate PR — out of scope here) Current app talks to Presidio via a Python subprocess (validate_pii.ts → validate_pii.py), not HTTP. Rewire to call PRESIDIO_URL over HTTP; add PRESIDIO_URL to env.ts; drop the guardrails venv install from app.Dockerfile. Remove the TS VIN path (vin.ts, recognizers.ts) — VIN is now native (and multi-language) in the image. Workflow-log redaction language: logger.ts:623 calls redactPIIFromExecution(payload, { entityTypes }) with no language, so it's hardcoded to 'en' (the default in pii-redaction.ts). Add a language to the Data Retention config (PiiRedactionConfig, default 'en') and thread it through logger.ts → redactPIIFromExecution → maskPIIBatch. This also fixes a latent consistency gap: an org can select a national-id entity (e.g. ES_NIF) for redaction, but under 'en' its recognizer never loads, so it's silently never redacted. (The guardrails block already passes piiLanguage per call — leave it as-is.) pii-entities.ts / piiLanguage align with the image; drop orphaned entities with no selectable language: KR_RRN (ko) and TH_TNIN (th). Type of Change New feature (build infra / CI) Testing Tested manually — full 5-model image built and run locally (see "Verification"). Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Open Graph Description: Summary Build and own a single Presidio image (docker/pii.Dockerfile) replacing the stock mcr.microsoft.com/presidio-analyzer + presidio-anonymizer sidecars. A thin FastAPI service (docker/pii/ser...

X Description: Summary Build and own a single Presidio image (docker/pii.Dockerfile) replacing the stock mcr.microsoft.com/presidio-analyzer + presidio-anonymizer sidecars. A thin FastAPI service (docker/pii/ser...

Opengraph URL: https://github.com/simstudioai/sim/pull/5176

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:bf582cb4-0608-f84e-b954-78b45371cb1b
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idCBB8:34FC0A:C1BA3F:108AAAC:6AFB568D
html-safe-nonce8efce244a21375e2b517776f76cf65ef200a3d77e59ee098e9961d8dd94a1773
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDQkI4OjM0RkMwQTpDMUJBM0Y6MTA4QUFBQzo2QUZCNTY4RCIsInZpc2l0b3JfaWQiOiI4MDEzODAwMDM1NjY3NjI5NzA5IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmacbf891ae6a4d2f3a6307c23e8326a52776880331b600bdb86719de8b3557c7c39
hovercard-subject-tagpull_request:3916632054
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/5176/files
twitter:imagehttps://avatars.githubusercontent.com/u/19564703?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/19564703?s=400&v=4
og:image:altSummary Build and own a single Presidio image (docker/pii.Dockerfile) replacing the stock mcr.microsoft.com/presidio-analyzer + presidio-anonymizer sidecars. A thin FastAPI service (docker/pii/ser...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None7122bb414b542dba14d71a3f0b9f6fd2abce715b9f5912d082800563e6c806d1
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/simstudioai/sim git https://github.com/simstudioai/sim.git
octolytics-dimension-user_id199344406
octolytics-dimension-user_loginsimstudioai
octolytics-dimension-repository_id912559512
octolytics-dimension-repository_nwosimstudioai/sim
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id912559512
octolytics-dimension-repository_network_root_nwosimstudioai/sim
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releasecbf66c024bb6d10e6955baa7413c54cf13a201bb
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/simstudioai/sim/pull/5176/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F5176%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F5176%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=simstudioai%2Fsim
Reloadhttps://github.com/simstudioai/sim/pull/5176/files
Reloadhttps://github.com/simstudioai/sim/pull/5176/files
Reloadhttps://github.com/simstudioai/sim/pull/5176/files
simstudioaihttps://github.com/simstudioai
simhttps://github.com/simstudioai/sim
Codehttps://github.com/simstudioai/sim
Issues129 (129)https://github.com/simstudioai/sim/issues
Pull requests197 (197)https://github.com/simstudioai/sim/pulls
Actionshttps://github.com/simstudioai/sim/actions
Projectshttps://github.com/simstudioai/sim/projects
Security and qualityhttps://github.com/simstudioai/sim/security
Insightshttps://github.com/simstudioai/sim/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
TheodoreSpeakshttps://github.com/TheodoreSpeaks
staginghttps://github.com/simstudioai/sim/tree/staging
feat/presidio-imagehttps://github.com/simstudioai/sim/tree/feat/presidio-image
Conversation 16 https://github.com/simstudioai/sim/pull/5176
Commits 5 https://github.com/simstudioai/sim/pull/5176/commits
Checks 15 https://github.com/simstudioai/sim/pull/5176/checks
Files changed https://github.com/simstudioai/sim/pull/5176/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/5176/files
feat(pii): build & own combined PII (analyzer + anonymizer) image https://github.com/simstudioai/sim/pull/5176/files#top
Show all changes 5 commits https://github.com/simstudioai/sim/pull/5176/files
1001ca9 feat(presidio): build & own combined analyzer+anonymizer image TheodoreSpeaks Jun 23, 2026 https://github.com/simstudioai/sim/pull/5176/commits/1001ca93344835cf50f61e8e3eaf2f9c53d50692
815d875 feat(presidio): add multi-language support (es/it/pl/fi) TheodoreSpeaks Jun 23, 2026 https://github.com/simstudioai/sim/pull/5176/commits/815d8750cda4a2560e9a828e2b463122fbf06f7e
77b8e57 improvement(presidio): address review feedback TheodoreSpeaks Jun 23, 2026 https://github.com/simstudioai/sim/pull/5176/commits/77b8e57c686dc0f56206a145335032e8a9e306ff
147e6f7 refactor(pii): rename presidio image artifacts to pii TheodoreSpeaks Jun 23, 2026 https://github.com/simstudioai/sim/pull/5176/commits/147e6f7be0bf7b442ae576241633a9d0583113c5
2b3bce1 refactor(pii): move service to apps/pii, make image ECR-only TheodoreSpeaks Jun 23, 2026 https://github.com/simstudioai/sim/pull/5176/commits/2b3bce173de0ad376a27b58dccf769634890824c
Clear filters https://github.com/simstudioai/sim/pull/5176/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/5176/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/5176/files
ci.yml https://github.com/simstudioai/sim/pull/5176/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03f
images.yml https://github.com/simstudioai/sim/pull/5176/files#diff-11afbb4eae27849c9f06ecefe10854fd951a54476c5f9ddc0284c0d7f0ea0c6c
package.json https://github.com/simstudioai/sim/pull/5176/files#diff-9c1fc4844e45167d843cde290dbe1abb26400fca0f1eeaf90d093b6d48e54439
requirements.txt https://github.com/simstudioai/sim/pull/5176/files#diff-6efb8fa1e8b3e1c05dd5caa63d77c5bbe9244748ce5d16a77987ef5502322947
server.py https://github.com/simstudioai/sim/pull/5176/files#diff-4447adf1c9c684d32a5fd31a17e05677749ef10d0ca6bce34ac9b6263e781a40
bun.lock https://github.com/simstudioai/sim/pull/5176/files#diff-bfd0ef82a01108fa1e836c2500b98b7de8a92bb1ae352a8efcbd9d6ffcaabd60
pii.Dockerfile https://github.com/simstudioai/sim/pull/5176/files#diff-9ffed6e2379c4cb446d31f12ffaf134779b805cf63f2083664be12267918a8e2
.github/workflows/ci.ymlhttps://github.com/simstudioai/sim/pull/5176/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03f
View file https://github.com/simstudioai/sim/blob/2b3bce173de0ad376a27b58dccf769634890824c/.github/workflows/ci.yml
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/5176/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/5176/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03f
https://github.com/simstudioai/sim/pull/5176/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03f
https://github.com/simstudioai/sim/pull/5176/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03f
https://github.com/simstudioai/sim/pull/5176/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03f
https://github.com/simstudioai/sim/pull/5176/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03f
https://github.com/simstudioai/sim/pull/5176/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03f
https://github.com/simstudioai/sim/pull/5176/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03f
https://github.com/simstudioai/sim/pull/5176/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03f
https://github.com/simstudioai/sim/pull/5176/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03f
.github/workflows/images.ymlhttps://github.com/simstudioai/sim/pull/5176/files#diff-11afbb4eae27849c9f06ecefe10854fd951a54476c5f9ddc0284c0d7f0ea0c6c
View file https://github.com/simstudioai/sim/blob/2b3bce173de0ad376a27b58dccf769634890824c/.github/workflows/images.yml
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/5176/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/5176/files#diff-11afbb4eae27849c9f06ecefe10854fd951a54476c5f9ddc0284c0d7f0ea0c6c
https://github.com/simstudioai/sim/pull/5176/files#diff-11afbb4eae27849c9f06ecefe10854fd951a54476c5f9ddc0284c0d7f0ea0c6c
https://github.com/simstudioai/sim/pull/5176/files#diff-11afbb4eae27849c9f06ecefe10854fd951a54476c5f9ddc0284c0d7f0ea0c6c
https://github.com/simstudioai/sim/pull/5176/files#diff-11afbb4eae27849c9f06ecefe10854fd951a54476c5f9ddc0284c0d7f0ea0c6c
https://github.com/simstudioai/sim/blob/staging/.github/CODEOWNERS#L36
apps/pii/package.jsonhttps://github.com/simstudioai/sim/pull/5176/files#diff-9c1fc4844e45167d843cde290dbe1abb26400fca0f1eeaf90d093b6d48e54439
View file https://github.com/simstudioai/sim/blob/2b3bce173de0ad376a27b58dccf769634890824c/apps/pii/package.json
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/5176/{{ revealButtonHref }}
apps/pii/requirements.txthttps://github.com/simstudioai/sim/pull/5176/files#diff-6efb8fa1e8b3e1c05dd5caa63d77c5bbe9244748ce5d16a77987ef5502322947
View file https://github.com/simstudioai/sim/blob/2b3bce173de0ad376a27b58dccf769634890824c/apps/pii/requirements.txt
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/5176/{{ revealButtonHref }}
apps/pii/server.pyhttps://github.com/simstudioai/sim/pull/5176/files#diff-4447adf1c9c684d32a5fd31a17e05677749ef10d0ca6bce34ac9b6263e781a40
View file https://github.com/simstudioai/sim/blob/2b3bce173de0ad376a27b58dccf769634890824c/apps/pii/server.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/5176/{{ revealButtonHref }}
Please reload this pagehttps://github.com/simstudioai/sim/pull/5176/files
https://github.com/simstudioai/sim/blob/staging/.github/CODEOWNERS#L38
bun.lockhttps://github.com/simstudioai/sim/pull/5176/files#diff-bfd0ef82a01108fa1e836c2500b98b7de8a92bb1ae352a8efcbd9d6ffcaabd60
View file https://github.com/simstudioai/sim/blob/2b3bce173de0ad376a27b58dccf769634890824c/bun.lock
Open in desktop https://desktop.github.com
how customized files appear on GitHubhttps://docs.github.com/github/administering-a-repository/customizing-how-changed-files-appear-on-github
Please reload this pagehttps://github.com/simstudioai/sim/pull/5176/files
docker/pii.Dockerfilehttps://github.com/simstudioai/sim/pull/5176/files#diff-9ffed6e2379c4cb446d31f12ffaf134779b805cf63f2083664be12267918a8e2
View file https://github.com/simstudioai/sim/blob/2b3bce173de0ad376a27b58dccf769634890824c/docker/pii.Dockerfile
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/5176/{{ revealButtonHref }}
Please reload this pagehttps://github.com/simstudioai/sim/pull/5176/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/5176/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.