Title: Optionally raise instead of swallowing JSON decode errors by jakobovski-arb · Pull Request #1027 · massive-com/client-python · GitHub
Open Graph Title: Optionally raise instead of swallowing JSON decode errors by jakobovski-arb · Pull Request #1027 · massive-com/client-python
X Title: Optionally raise instead of swallowing JSON decode errors by jakobovski-arb · Pull Request #1027 · massive-com/client-python
Description: The problem _get and _paginate_iter both decode responses like this: try: obj = self._decode(resp) except ValueError as e: logger.error("Error decoding json response: %s", e) return [] A truncated or malformed body therefore becomes an empty result, which a caller cannot distinguish from "the API had nothing to return". The only trace is a log line, and the call itself succeeds. In _paginate_iter this is worse than an empty result: the bare return ends the generator early, so a partial page silently truncates a result set that still looks complete to the caller. This is not hypothetical. In a single multi-hour backfill over /v3/trades, we logged 12 of these: RESTClient ERROR: Error decoding json response: Unterminated string starting at: line 1 column 570332 (char 570331) RESTClient ERROR: Error decoding json response: Expecting ',' delimiter: line 1 column 190527 (char 190526) ... Each one produced an empty page that our pipeline recorded as a real absence — "this ticker had no auction print that day" — and cached permanently, because an empty response from a working API is a legitimate answer in our domain. We only found out by reading the log afterwards. For data ordered newest-first, where the row of interest sits on the last page, a truncated early page can drop exactly the record you were querying for. The change RESTClient(raise_on_decode_error=True) raises a new ResponseDecodeError from both decode sites instead of returning []. The flag threads to VXClient; other BaseClient subclasses inherit it through RESTClient's super().__init__(). Default is False, so existing behaviour is unchanged and this is not a breaking change. Caveats, in the interest of full disclosure We think raising should eventually be the default. Converting an undecodable body into a successful empty result is a correctness hazard for any consumer that treats "empty" as data, and there is currently no way to detect it programmatically at all. We made it opt-in only to keep the change non-breaking — if you'd prefer it default-on for a major version, or gated by an env var / module-level default instead of a constructor kwarg, we're happy to rework it. The _paginate_iter half is the more important one, and it is arguably a bug rather than a design choice: silently returning a truncated result set is different in kind from returning an empty one, and no flag setting makes a partially-consumed generator obvious to the caller. If you'd rather fix only that site, or always raise there while keeping _get lenient, that would still address the dangerous case. A retry would be the natural follow-up. A truncated body is usually transient, so the SDK could reasonably retry a failed decode the way urllib3 retries a failed connection, rather than pushing that on every caller. We didn't include it here to keep this diff to one concern. Naming is yours to pick. raise_on_decode_error / ResponseDecodeError seemed closest to the existing BadResponse, but rename freely. Test note: test_rest/models/test_requests.py::RequestTest::test_clint_headers_concat already fails on a clean master checkout in our environment (verified by stashing this change), so it is unrelated to this PR. The other 47 tests pass, including the 4 added here. Tests test_rest/test_decode_errors.py covers: the default still returns an empty result, the opt-in raises ResponseDecodeError, the flag reaches client.vx, and a well-formed body is unaffected either way. Context: we're the authors of #1026 (maxsize on the connection pool). Both issues surfaced from the same workload — a high-concurrency historical backfill of /v3/trades — where silent data loss is expensive and hard to detect after the fact. 🤖 Generated with Claude Code
Open Graph Description: The problem _get and _paginate_iter both decode responses like this: try: obj = self._decode(resp) except ValueError as e: logger.error("Error decoding json response: %s", e) ...
X Description: The problem _get and _paginate_iter both decode responses like this: try: obj = self._decode(resp) except ValueError as e: logger.error("Error decoding json response: %s",...
Opengraph URL: https://github.com/massive-com/client-python/pull/1027
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:ec71c2f9-f5b6-d527-4ee3-6d710f649a4e |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 9B0A:A7C31:BD8E32:F8BA70:6AFE8AA3 |
| html-safe-nonce | f913d60657e7bc64109e4f1a989a157a30a7df8fac3e4d03b0a17d5eb94370f0 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5QjBBOkE3QzMxOkJEOEUzMjpGOEJBNzA6NkFGRThBQTMiLCJ2aXNpdG9yX2lkIjoiNjk1MDIwMTEwMzM1NDg1ODE0NyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 87bac3d9903e05acd4b0890002fd1800995f4d29e3bcc45c758f11dd6c68447e |
| hovercard-subject-tag | pull_request:4203698600 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/massive-com/client-python/pull/1027/files |
| twitter:image | https://avatars.githubusercontent.com/u/201493996?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/201493996?s=400&v=4 |
| og:image:alt | The problem _get and _paginate_iter both decode responses like this: try: obj = self._decode(resp) except ValueError as e: logger.error("Error decoding json response: %s", e) ... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/massive-com/client-python git https://github.com/massive-com/client-python.git |
| octolytics-dimension-user_id | 191946194 |
| octolytics-dimension-user_login | massive-com |
| octolytics-dimension-repository_id | 216660192 |
| octolytics-dimension-repository_nwo | massive-com/client-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 216660192 |
| octolytics-dimension-repository_network_root_nwo | massive-com/client-python |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width