Title: perf: size metadata ids and offsets per app by edusperoni · Pull Request #2113 · NativeScript/android · GitHub
Open Graph Title: perf: size metadata ids and offsets per app by edusperoni · Pull Request #2113 · NativeScript/android
X Title: perf: size metadata ids and offsets per app by edusperoni · Pull Request #2113 · NativeScript/android
Description: Description Stacked on #2112. Removes the 65,535-node limit on the metadata and makes it smaller again, by letting each app's metadata use only as many bytes per id and per offset as it needs. What changes. Three kinds of field were a fixed width: a node id (2 bytes, which is where the 65,535-node limit came from), an offset into the names (4 bytes) and a signature reference (3 bytes, added in #2112). The generator now gives each the fewest bytes, 1 to 4, that hold the largest value this app's metadata has, and writes the three widths in the value stream's header: Offset Size 0 4 format word (version 3) 4 1 bytes per node id 5 1 bytes per name offset 6 1 bytes per signature reference 7 1 reserved 8 4 length of the signature region Every record keeps the structure it has in #2112, with those widths substituted. A node record in the tree stream is two ids, a name offset and a 4-byte value offset (that one stays fixed, since its top range marks arrays). What an app gets: ids names signatures three streams Test app before #2112 2 4 – 3,264 KB Test app with #2112 2 4 3 2,838 KB Test app with this 2 3 3 2,687 KB Test app, every width forced one wider 3 4 4 3,085 KB An ordinary app gets 3-byte name offsets instead of 4, which is where this PR's saving comes from (−5% here, −18% against the original format). An app with more than 65,535 nodes gets 3-byte ids instead of a failed build. The limit is now 2^30 nodes. The 8 MB cap on the signature region that #2112 introduced is gone. The download does not shrink, and is in fact about 2 KB larger than with #2112 (release APK 34,933 KB against 34,931 KB): a 3-byte offset compresses slightly worse than a 4-byte one whose top byte is zero. The saving is on disk after extraction and in memory. Runtime. One helper reads every such field: it loads four bytes and masks. The widths are fixed for the process, so there is no branch on them. The generator pads the value stream with four bytes so the last field can be read that way, and a node record ends in a full four-byte field. Field records are read through MetadataReader rather than a packed struct cast onto the buffer, the way method records already were. Nodes created at run time are numbered from 2^30, above any id the static metadata can use (it was 65536). Generator. FileStreamWriter passed every write straight to the file, one system call per field and one more for each query of the file's position. It now buffers its output and counts its own position: about 1.7 s instead of 2.9 s on the test app, with byte-identical output. Speed, Galaxy S20 FE, Android 13, optimized release build, warm starts, 16 launches each, medians: #2112 This PR native metadata load 1,334 µs 1,366 µs first access, total of the harness's 17 cold rows 6,101 µs 6,130 µs native heap (PSS) 7,598 KB 7,598 KB mapped files (PSS) 2,304 KB 2,184 KB Both timing differences are well inside run-to-run noise. Testing the widths nobody normally gets. Widths above the smallest only occur in very large apps, which is the kind of path that went unnoticed until #2111. So runtests now builds the test app with -PwidenMetadataFields, which makes the generator write every width one byte wider than needed (3, 4 and 4 here). CI therefore runs the suite on the wide layouts, and every ordinary build uses the narrow ones. ./gradlew runtestsAndVerifyResults -PnarrowMetadataFields runs the suite on the widths that ship. The widen property is in app/build.gradle, which also ships to apps; it does nothing unless set. Not covered: 4-byte ids (more than 16.7 million nodes) and 1-byte widths never occur in any build here, and CI no longer runs the suite at the shipping widths unless asked to. Does your commit message include the wording below to reference a specific issue in this repo? No issue. Related Pull Requests #2112, which this is stacked on. #2111, where the 16-bit limit first caused trouble. Does your pull request have unit tests? WriterFormatTest (generator, 4 tests) writes a small hand-built tree and reads it back with a parser that knows only the format and fails unless every byte of the value stream belongs to exactly one record. It runs at three sets of widths (one-byte fields, the 2/3-byte ones ordinary apps get, and the widened ones) and covers signatures shared between methods, a 256-type signature and a refused 257-type one, a class with only static members, getter-only and setter-only properties, and own versus inherited static fields. Breaking the writer on purpose fails it. This is the test suggested in the review of #2112; it is here because this PR changes the layout again. The generator's unit tests were not run in CI before. ./gradlew runMetadataGeneratorTests runs them, and the pull request workflow now calls it; the release workflow is unchanged. No new device specs; every existing one reads the metadata. runtestsAndVerifyResults passes 1598 / 1598 in each of: the forced-wide configuration it now uses (ids 3, names 4, signatures 4); the default widths (2, 3, 3), with -PnarrowMetadataFields; a local build with the tree padded to 81,726 nodes, so that every real class has an id above 70,000 and the generator picks 3-byte ids on its own. The padding is not part of this PR. Both outputs were also checked with a parser written separately from the format description, which requires every byte of the value stream to belong to exactly one record and validates every id, name offset and signature reference against the header's widths. Summary by CodeRabbit New Features Metadata supports variable-width identifiers and records, with an option to widen fields. Improvements Runtime metadata reading supports the updated format and checks field widths and structure. Class members, fields, and method signatures are read using the metadata’s configured widths, supporting a wider range of record sizes.
Open Graph Description: Description Stacked on #2112. Removes the 65,535-node limit on the metadata and makes it smaller again, by letting each app's metadata use only as many bytes per id and per offset as it needs. ...
X Description: Description Stacked on #2112. Removes the 65,535-node limit on the metadata and makes it smaller again, by letting each app's metadata use only as many bytes per id and per offset as it nee...
Opengraph URL: https://github.com/NativeScript/android/pull/2113
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:a44ce7ed-11e7-f1a5-cb8d-0ac9c8240cb6 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | DEC6:269AD9:1353675:19D98F3:6B012854 |
| html-safe-nonce | 22221894f40524267feca80808a6d1ec68f98345b6671ad3b2bc6a0601ce3aa2 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJERUM2OjI2OUFEOToxMzUzNjc1OjE5RDk4RjM6NkIwMTI4NTQiLCJ2aXNpdG9yX2lkIjoiNzM5NzkxNDU0NTg5ODk4OTY1MyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | ca802a318a7bc9ca01486c9c3e85099e99d0d2c39a2a51a22aceed718bacb9a1 |
| hovercard-subject-tag | pull_request:5342954674 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/NativeScript/android/pull/2113/files |
| twitter:image | https://avatars.githubusercontent.com/u/4061767?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/4061767?s=400&v=4 |
| og:image:alt | Description Stacked on #2112. Removes the 65,535-node limit on the metadata and makes it smaller again, by letting each app's metadata use only as many bytes per id and per offset as it needs. ... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | cc5dfdd32f62ebd3223cab3af3aa2b268d3605218237326a48a99d1efd89495c |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/NativeScript/android git https://github.com/NativeScript/android.git |
| octolytics-dimension-user_id | 7392261 |
| octolytics-dimension-user_login | NativeScript |
| octolytics-dimension-repository_id | 29137950 |
| octolytics-dimension-repository_nwo | NativeScript/android |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 29137950 |
| octolytics-dimension-repository_network_root_nwo | NativeScript/android |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 8c77c9e8439031ad54c75a250694842892f693e1 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width