René's URL Explorer Experiment

Title: fix(pii): mask offloaded large payloads chunk-by-chunk instead of aborting at 16MB by TheodoreSpeaks · Pull Request #5810 · simstudioai/sim · GitHub

Open Graph Title: fix(pii): mask offloaded large payloads chunk-by-chunk instead of aborting at 16MB by TheodoreSpeaks · Pull Request #5810 · simstudioai/sim

X Title: fix(pii): mask offloaded large payloads chunk-by-chunk instead of aborting at 16MB by TheodoreSpeaks · Pull Request #5810 · simstudioai/sim

Description: Summary Block outputs past the 16MB inline materialization ceiling aborted the run before masking even started — the redaction path hydrated the whole offloaded value at once and the pre-flight size assert fired on the manifest's total byteSize Large-array manifests now redact chunk-by-chunk: page one stored chunk at a time, mask, re-store, rebuild via the manifest writer (preview regenerated from masked items, bookkeeping recomputed + asserted). Peak heap stays ~one chunk regardless of payload size Single refs up to the 64MB durable cap hydrate with a raised budget and run serially outside the concurrency pool (one 30MB hydration holds ~2-3x its size in transient heap; 4 at once would OOM the medium-1x trigger.dev machines) Mask-batch chunk requests retry transient failures (network errors, 408/429/5xx honoring Retry-After, runtime-level timeouts, socket closes) with jittered backoff — a single ALB blip or Presidio pod restart no longer fails a whole payload's redaction, and on the execution-altering stages no longer aborts the run Fixed a latent double-mask: a nested ref whose masked value shrinks back under the offload threshold got re-inlined and masked twice (wrong for custom patterns); string pass now runs before ref substitution Sidecar: NER_ENTITIES now intersects the SpacyRecognizer's claimed entities with what the loaded models' NER labels can actually produce. Presidio's default mapping claims PHONE_NUMBER/AGE/ID/EMAIL for transformer backends spaCy can't emit, so any rule naming PHONE_NUMBER silently forced the full spaCy pass and the regex-only fast path never fired Type of Change Bug fix Testing End-to-end against a live stack (local Presidio + S3 + dev DB): a 26.2MB / 40k-record function output that previously aborted instantly now masks in ~3.2min on a single sidecar worker with the realistic entity set (all 265 batches on the regex fast path; ~15min projected on full NER before the sidecar fix). All 40k items verified: order/structure preserved across chunks, emails/cards/SSNs/phones masked, manifest preview regenerated from masked content, skip-vs-full span parity confirmed for phones. Unit suites updated: manifest branch had zero coverage (isLargeArrayManifest was stubbed false) — now covered chunk-wise incl. partial-chunk failure in both failure modes. lint, check:api-validation:strict, and full lib/execution+logs+guardrails suites pass. Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA) 🤖 Generated with Claude Code https://claude.ai/code/session_01A1JYstmLHk9qMGyBDqYRcJ

Open Graph Description: Summary Block outputs past the 16MB inline materialization ceiling aborted the run before masking even started — the redaction path hydrated the whole offloaded value at once and the pre-flight si...

X Description: Summary Block outputs past the 16MB inline materialization ceiling aborted the run before masking even started — the redaction path hydrated the whole offloaded value at once and the pre-flight si...

Opengraph URL: https://github.com/simstudioai/sim/pull/5810

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width