Title: fix(logger): never let structured serialization throw into the caller by waleedlatif1 · Pull Request #6331 · simstudioai/sim · GitHub
Open Graph Title: fix(logger): never let structured serialization throw into the caller by waleedlatif1 · Pull Request #6331 · simstudioai/sim
X Title: fix(logger): never let structured serialization throw into the caller by waleedlatif1 · Pull Request #6331 · simstudioai/sim
Description: Context Production structured logging went fully silent for hours across all five sim-production-us-east-1-app ECS tasks — INFO/WARN/ERROR all dropped to zero per 30 min, and a raw substring search for \"module\": returned nothing. Confirmed as genuine loss via AWS/Logs IncomingLogEvents (~25k → ~2-6k per 30 min), not a query artifact. Non-JSON output (Next.js internals, Better Auth, raw stack traces) kept shipping, and the socket service on the same cluster and same awslogs driver was unaffected — so it was localized to the app containers' own output, not the driver or the account. This PR fixes one concrete, reproducible defect on that path. It is not claimed to be the whole root cause. The bug In the production JSON branch, Logger.log merged caller-supplied arguments into the entry with Object.assign and then called JSON.stringify, with no error handling on either step. If that metadata contained a cyclic reference, a BigInt, or a throwing getter, the call raised a TypeError out of logger.info(...) into the caller — the line was lost and whatever was being logged about was aborted. Verified against the current staging logger under NODE_ENV=production: RESULT: circular -> THREW: TypeError | JSON.stringify cannot serialize cyclic structures. RESULT: bigint -> THREW: TypeError | JSON.stringify cannot serialize BigInt. Cyclic metadata is easy to hit unintentionally — errors carrying request/response/cause chains, ORM rows with back-references, socket handles. The development path was never affected: the colorized branch routes objects through formatObject, which already has a try/catch. So this class of bug is invisible locally and only appears in production, where its signature is exactly structured logs vanish while raw stack traces keep shipping. The fix Entry construction and serialization now go through serializeEntry, which degrades instead of throwing: normal JSON.stringify retry with a replacer that tolerates cycles ([Circular]) and BigInt (stringified) minimal entry preserving timestamp/level/module/message, flagged serializationError: true A logger must never be able to break its caller. Tests Three regression tests in packages/logger/src/index.test.ts covering cyclic, BigInt, and throwing-getter metadata. All three were confirmed to fail against the unfixed source (git stash on index.ts → 3 failed | 25 passed) and pass with it (28 passed). bun run lint and tsc --noEmit are clean. Follow-ups (not in this repo) Alarms and ECS task definitions live in the private sibling CDK repo, not here. Two gaps found there and reported separately: No alarm exists on AWS/Logs IncomingLogEvents for the app log group. Every metric filter over that group sees zero matches during a blackout and sits in OK — a silent alarm indistinguishable from healthy. A log-absence alarm with treatMissingData: BREACHING is specified in the report. Both app and socket LogDrivers.awsLogs({ ... }) calls omit mode, so both default to blocking.
Open Graph Description: Context Production structured logging went fully silent for hours across all five sim-production-us-east-1-app ECS tasks — INFO/WARN/ERROR all dropped to zero per 30 min, and a raw substring search...
X Description: Context Production structured logging went fully silent for hours across all five sim-production-us-east-1-app ECS tasks — INFO/WARN/ERROR all dropped to zero per 30 min, and a raw substring search...
Opengraph URL: https://github.com/simstudioai/sim/pull/6331
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:17a1ee29-efeb-4eae-5aea-5a71857f8ead |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | DBE4:C4B81:99FCA4:D4A1D3:6AFB4DFB |
| html-safe-nonce | ab86030c84e35e4edb0bc0a043554d85d7f488227c1be1e9042b7128f614f86d |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEQkU0OkM0QjgxOjk5RkNBNDpENEExRDM6NkFGQjRERkIiLCJ2aXNpdG9yX2lkIjoiNDM0NjAyMTg5NjE2NDk1MzU5NSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 39fc5010aa361c41f25d9ff9bea8a6e948ab54f11c8f996cd2c47291ab29f3a0 |
| hovercard-subject-tag | pull_request:4223263426 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/6331/files |
| twitter:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| og:image:alt | Context Production structured logging went fully silent for hours across all five sim-production-us-east-1-app ECS tasks — INFO/WARN/ERROR all dropped to zero per 30 min, and a raw substring search... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | acf7616d06cc051644705b379878c9bb13dc5e70ecf4ffd9b34140f8ac01d7e1 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 26504db649b58525278f77a52b80ba5f10d5f4de |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width