René's URL Explorer Experiment
Title: gh-150875: Speed up JSON string encoding for long ASCII strings by gaborbernat · Pull Request #150876 · python/cpython · GitHub
Open Graph Title: gh-150875: Speed up JSON string encoding for long ASCII strings by gaborbernat · Pull Request #150876 · python/cpython
X Title: gh-150875: Speed up JSON string encoding for long ASCII strings by gaborbernat · Pull Request #150876 · python/cpython
Description: json.dumps escapes each string by first scanning it one character at a time to size the escaped output (ascii_escape_size), after which write_escaped_ascii copies the string verbatim when nothing needs escaping. For a long string with no characters that need escaping, which is the common case for text values, log messages, and other long content, that per-character sizing scan is pure overhead before the verbatim copy.
This detects the no-escape case on the one-byte (ASCII/Latin-1) representation eight bytes at a time, so it returns the verbatim size after about one eighth of the work. It is the encode-side counterpart to #150872; the two touch different code paths and are separate changes.
What we do now (scalar, one code point at a time)
for (i = 0, output_size = 2; i < input_chars; i++) {
Py_UCS4 c = PyUnicode_READ(kind, input, i);
if (S_CHAR(c)) { output_size += 1; } // ordinary char, needs no escaping
else { /* compute the escaped width */ }
}
S_CHAR is printable ASCII except " and \, so a byte needs escaping when c < 0x20 || c > 0x7e || c == '"' || c == '\\'. For a long escape-free string this reads and tests every byte just to learn that the output equals the input plus two quotes.
What SWAR does (8 bytes at a time, in one register)
SWAR is "SIMD within a register": load 8 bytes into a single uint64_t and test all 8 lanes at once with ordinary integer ops.
for (; j + 8 <= input_chars; j += 8) {
memcpy(&w, p + j, 8);
mq = haszero(w ^ (0x22 * 0x0101010101010101)); // any lane == '"' ?
ms = haszero(w ^ (0x5c * 0x0101010101010101)); // any lane == '\\' ?
mlo = haszero(w & 0xE0E0E0E0E0E0E0E0); // any lane < 0x20 ?
mhi = (w & 0x8080808080808080) // any lane >= 0x80 ?
| haszero(w ^ (0x7f * 0x0101010101010101)); // any lane == 0x7f ?
if (mq | ms | mlo | mhi) { needs_escape = 1; break; } // escape needed -> scalar
}
// no escape found anywhere -> output is the input plus two quotes
if (!needs_escape) return input_chars + 2;
haszero(v) = (v - 0x0101…) & ~v & 0x8080… lights the high bit of exactly the zero lanes, with no false positives or negatives. Broadcasting a byte (b * 0x0101…) and XOR-ing turns "equals b" into "is zero". The range checks < 0x20 and > 0x7e reuse the same idea. When all 8 lanes are ordinary, the loop advances 8 bytes; at the first lane that needs escaping it breaks and the existing per-character loop computes the exact size and does the work. A length guard keeps short strings (the common dict key) on the original loop, where the fast path's setup would not pay off.
These are the same 0x0101… / 0x8080… masks that Objects/unicodeobject.c and Objects/stringlib/find_max_char.h already use for ASCII scanning.
When and how this changes performance
json.dumps, current encoder versus this change:
Document shape
Effect
One long text field (~11 KB string)
5.3x faster
Many 200-character ASCII string values
3.1x faster
Realistic mixed records (short and medium strings)
1.3x faster
Short keys, strings that need escaping, the pyperformance document
no change
Strings with emoji or other non-Latin-1 text
no change (scalar path)
The gain scales with string length. The short-string guard keeps key-heavy documents unaffected; an earlier guardless version measured about 1.18x slower on a 2000-short-key document, which the guard removes.
Correctness
Output is byte-identical to the current encoder. Verified against the full test_json suite and a 199-case differential corpus that places each escape-relevant character (", \\, control chars, 0x7f, and non-Latin-1 characters) at every offset across the eight-byte window, in both ensure_ascii=True and ensure_ascii=False modes. Every output matched.
Benchmark
import json, pyperf
long_ascii = [("x"*200) for _ in range(200)]
text_blob = {"body": "lorem ipsum dolor sit amet " * 400}
escaped = [('a"b\\c\n'*30) for _ in range(200)]
short_keys = {f"k{i}": i for i in range(2000)}
mixed_real = [{"id":i,"name":f"user_{i}","email":f"u{i}@x.com","bio":"hello world "*10} for i in range(300)]
nonascii = ["café 😀 中文 "*20 for _ in range(200)]
objs = {"long_ascii": long_ascii, "text_blob": text_blob, "escaped": escaped,
"short_keys": short_keys, "mixed_real": mixed_real, "nonascii": nonascii}
runner = pyperf.Runner()
for n, o in objs.items():
runner.bench_func(f"dumps/{n}", lambda o=o: json.dumps(o))
References for the bit tricks: Sean Anderson, Bit Twiddling Hacks (zero byte, byte equal to n, byte less than n); Henry S. Warren Jr., Hacker's Delight, 2nd ed., chapter 6.
It is not the SIMD parsing backend from #142915: it adds no intrinsics, no CPU detection, and no build configuration, and it does not depend on #125022.
Resolves #150875.
Issue: gh-150875
Open Graph Description: json.dumps escapes each string by first scanning it one character at a time to size the escaped output (ascii_escape_size), after which write_escaped_ascii copies the string verbatim when nothing n...
X Description: json.dumps escapes each string by first scanning it one character at a time to size the escaped output (ascii_escape_size), after which write_escaped_ascii copies the string verbatim when nothing n...
Opengraph URL: https://github.com/python/cpython/pull/150876
X: @github
direct link
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:2eb9334d-b30d-e016-00c9-6879323a5a32 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 9D14:23E48:41E077:52BE8A:6B020EF3 |
| html-safe-nonce | 497956e9725b49a84f1d885bd88bd244e91ebce7ddae7365e12c6d50452ac0bb |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5RDE0OjIzRTQ4OjQxRTA3Nzo1MkJFOEE6NkIwMjBFRjMiLCJ2aXNpdG9yX2lkIjoiMTA5NjcwNDc1Njc3NDIxMTMxNSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 0c558c8e4116374d18b4258127eeadd2294e6691db268c8661bb727feff71a41 |
| hovercard-subject-tag | pull_request:3797585688 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | ///pull_requests/show/files |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/python/cpython/pull/150876/files |
| twitter:image | https://avatars.githubusercontent.com/u/690238?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/690238?s=400&v=4 |
| og:image:alt | json.dumps escapes each string by first scanning it one character at a time to size the escaped output (ascii_escape_size), after which write_escaped_ascii copies the string verbatim when nothing n... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
| Skip to content | https://github.com/python/cpython/pull/150876/files#start-of-content |
| https://github.com/ |
| Sign in | https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F150876%2Ffiles |
| GitHubRepository hosting, code review, and collaboration | https://github.com/platform |
| CopilotAI coding agent for tasks, projects, and automated work | https://github.com/features/copilot |
| ActionsCI/CD and automation | https://github.com/features/actions |
| Code QualityQuality analysis, fixes, and enforcement | https://github.com/features/code-quality |
| CodespacesCloud development environments | https://github.com/features/codespaces |
| Code SecurityVulnerability detection and remediation | https://github.com/security/advanced-security/code-security |
| Secret ProtectionSecret scanning and leak prevention | https://github.com/security/advanced-security/secret-protection |
| All Features | https://github.com/features |
| Documentation | https://docs.github.com/en |
| Training | https://learn.github.com/skills |
| Certifications | https://learn.github.com/credentials |
| Build with Copilot | https://github.com/features/ai |
| Customize Copilot | https://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview |
| Integrations | https://github.com/marketplace |
| Trending repos | https://github.com/trending |
| GitHub Sponsors | https://github.com/open-source/sponsors |
| Developer articles | https://github.com/resources/articles |
| ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place. | https://github.blog/changelog/ |
| Enterprises | https://github.com/enterprise |
| Team | https://github.com/team |
| Startups | https://github.com/enterprise/startups |
| Nonprofits | https://github.com/solutions/industry/nonprofits |
| Healthcare | https://github.com/solutions/industry/healthcare |
| Financial services | https://github.com/solutions/industry/financial-services |
| Manufacturing | https://github.com/solutions/industry/manufacturing |
| Government | https://github.com/solutions/industry/government |
| See all industries | https://github.com/solutions/industry |
| Why GitHub | https://github.com/why-github |
| Customer Stories | https://github.com/customer-stories |
| Business insights | https://github.com/solutions/executive-insights |
| Trust Center | https://github.com/trust-center |
| GitHub Advanced SecurityEnterprise-grade code security | https://github.com/security/advanced-security |
| Copilot for businessEnterprise-grade AI features | https://github.com/features/copilot/copilot-business |
| Premium SupportEnterprise-grade 24/7 support | https://github.com/enterprise/premium-support |
| Documentation | https://docs.github.com/en |
| Resource center | https://github.com/resources |
| Blog | https://github.blog/ |
| Events & webinars | https://github.com/resources/events |
| Community Forum | https://github.com/orgs/community/discussions |
| Customer Stories | https://github.com/customer-stories |
| Integrations | https://github.com/marketplace |
| Customer support | https://help.github.com/ |
| Training | https://learn.github.com/skills |
| GitHub Status | https://www.githubstatus.com/ |
| About | https://github.com/about |
| Partners | https://github.com/partners |
| Changelog | https://github.blog/changelog/ |
| GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub. | https://githubuniverse.com/ |
| Pricing | https://github.com/pricing |
| Sign in | https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F150876%2Ffiles |
| Sign up | https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=python%2Fcpython |
| Reload | https://github.com/python/cpython/pull/150876/files |
| Reload | https://github.com/python/cpython/pull/150876/files |
| Reload | https://github.com/python/cpython/pull/150876/files |
| python | https://github.com/python |
| cpython | https://github.com/python/cpython |
| Code | https://github.com/python/cpython |
| Issues5k+ (5k+) | https://github.com/python/cpython/issues |
| Pull requests2.4k (2.4k) | https://github.com/python/cpython/pulls |
| Actions | https://github.com/python/cpython/actions |
| Projects | https://github.com/python/cpython/projects |
| Security and quality | https://github.com/python/cpython/security |
| Insights | https://github.com/python/cpython/insights |
| Sign up for GitHub
| https://github.com/signup?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose |
| terms of service | https://docs.github.com/terms |
| privacy statement | https://docs.github.com/privacy |
| Sign in | https://github.com/login?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose |
| gaborbernat | https://github.com/gaborbernat |
| python:main | https://github.com/python/cpython/tree/main |
| gaborbernat:opt/json-swar-encode | https://github.com/gaborbernat/cpython/tree/opt/json-swar-encode |
|
Conversation
4
| https://github.com/python/cpython/pull/150876 |
|
Commits
2
| https://github.com/python/cpython/pull/150876/commits |
|
Checks
52
| https://github.com/python/cpython/pull/150876/checks |
|
Files changed
3
| https://github.com/python/cpython/pull/150876/files |
|
gh-150875: Speed up JSON string encoding for long ASCII strings
| https://github.com/python/cpython/pull/150876/files#top |
|
Show all changes
2 commits
| https://github.com/python/cpython/pull/150876/files |
|
36bd06b
gh-150875: Speed up JSON string encoding for long ASCII strings
gaborbernat Jun 3, 2026
| https://github.com/python/cpython/pull/150876/commits/36bd06b749bc18cf2951eb5b4f1ac103e5d0c88e |
|
7d10318
Add tests exercising the ensure_ascii encoder paths
gaborbernat Jun 3, 2026
| https://github.com/python/cpython/pull/150876/commits/7d10318c13671a29b5021adf2c1a0e852e6b3cf0 |
|
Clear filters
| https://github.com/python/cpython/pull/150876/files |
| Please reload this page | https://github.com/python/cpython/pull/150876/files |
| Please reload this page | https://github.com/python/cpython/pull/150876/files |
|
test_dump.py
| https://github.com/python/cpython/pull/150876/files#diff-c48f208a13a0f4cbaced66756fdc25388aedc34dfa4c6da5679d39a241d7a0af |
|
2026-06-03-11-10-06.gh-issue-150875.WK9OTj.rst
| https://github.com/python/cpython/pull/150876/files#diff-84ef3a6b254714772370e872b543162f895a63fa0e48306b3f565a2fd26a7da7 |
|
_json.c
| https://github.com/python/cpython/pull/150876/files#diff-efe183ae0b85e5b8d9bbbc588452dd4de80b39fd5c5174ee499ba554217a39ed |
| Lib/test/test_json/test_dump.py | https://github.com/python/cpython/pull/150876/files#diff-c48f208a13a0f4cbaced66756fdc25388aedc34dfa4c6da5679d39a241d7a0af |
|
View file
| https://github.com/gaborbernat/cpython/blob/7d10318c13671a29b5021adf2c1a0e852e6b3cf0/Lib/test/test_json/test_dump.py |
|
Open in desktop
| https://desktop.github.com |
| https://github.co/hiddenchars |
| https://github.com/python/cpython/pull/150876/{{ revealButtonHref }} |
|
| https://github.com/python/cpython/pull/150876/files#diff-c48f208a13a0f4cbaced66756fdc25388aedc34dfa4c6da5679d39a241d7a0af |
|
| https://github.com/python/cpython/pull/150876/files#diff-c48f208a13a0f4cbaced66756fdc25388aedc34dfa4c6da5679d39a241d7a0af |
| Misc/NEWS.d/next/Library/2026-06-03-11-10-06.gh-issue-150875.WK9OTj.rst | https://github.com/python/cpython/pull/150876/files#diff-84ef3a6b254714772370e872b543162f895a63fa0e48306b3f565a2fd26a7da7 |
|
View file
| https://github.com/gaborbernat/cpython/blob/7d10318c13671a29b5021adf2c1a0e852e6b3cf0/Misc/NEWS.d/next/Library/2026-06-03-11-10-06.gh-issue-150875.WK9OTj.rst |
|
Open in desktop
| https://desktop.github.com |
| https://github.co/hiddenchars |
| https://github.com/python/cpython/pull/150876/{{ revealButtonHref }} |
| Modules/_json.c | https://github.com/python/cpython/pull/150876/files#diff-efe183ae0b85e5b8d9bbbc588452dd4de80b39fd5c5174ee499ba554217a39ed |
|
View file
| https://github.com/gaborbernat/cpython/blob/7d10318c13671a29b5021adf2c1a0e852e6b3cf0/Modules/_json.c |
|
Open in desktop
| https://desktop.github.com |
| https://github.co/hiddenchars |
| https://github.com/python/cpython/pull/150876/{{ revealButtonHref }} |
|
| https://github.com/python/cpython/pull/150876/files#diff-efe183ae0b85e5b8d9bbbc588452dd4de80b39fd5c5174ee499ba554217a39ed |
|
| https://github.com/python/cpython/pull/150876/files#diff-efe183ae0b85e5b8d9bbbc588452dd4de80b39fd5c5174ee499ba554217a39ed |
| Please reload this page | https://github.com/python/cpython/pull/150876/files |
|
| https://github.com |
| Terms | https://docs.github.com/site-policy/github-terms/github-terms-of-service |
| Privacy | https://docs.github.com/site-policy/privacy-policies/github-privacy-statement |
| Security | https://github.com/security |
| Status | https://www.githubstatus.com/ |
| Community | https://github.community/ |
| Docs | https://docs.github.com/ |
| Contact | https://support.github.com?tags=dotcom-footer |
Viewport: width=device-width
URLs of crawlers that visited me.