Title: Read, format and write each file on a worker thread, and keep only what the output needs · Issue #228 · openjavaformat/open-java-format · GitHub
Open Graph Title: Read, format and write each file on a worker thread, and keep only what the output needs · Issue #228 · openjavaformat/open-java-format
X Title: Read, format and write each file on a worker thread, and keep only what the output needs · Issue #228 · openjavaformat/open-java-format
Description: What happens Main.formatFiles reads every file on the main thread, as new String(Files.readAllBytes(path), UTF_8), keeps the text in a map and submits it to the pool. It then takes the results on the main thread, in the order the files w...
Open Graph Description: What happens Main.formatFiles reads every file on the main thread, as new String(Files.readAllBytes(path), UTF_8), keeps the text in a map and submits it to the pool. It then takes the results on t...
X Description: What happens Main.formatFiles reads every file on the main thread, as new String(Files.readAllBytes(path), UTF_8), keeps the text in a map and submits it to the pool. It then takes the results on t...
Opengraph URL: https://github.com/openjavaformat/open-java-format/issues/228
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Read, format and write each file on a worker thread, and keep only what the output needs","articleBody":"### What happens\n\n`Main.formatFiles` reads every file on the main thread, as `new String(Files.readAllBytes(path), UTF_8)`, keeps the text in a map and submits it to the pool. It then takes the results on the main thread, in the order the files were given, and compares, writes or prints each one there. Every input and every output stays referenced until the loop is over, and a file that is done waits for all the files before it to be written or printed.\n\n### google-java-format\n\nThe same, and it waits longer. It reads on the main thread too, and an `ExecutorCompletionService` hands back the results as they finish, but all of them go into a list, which is sorted by path, and only then is anything written or printed.\n\n### Where the time goes now\n\n865 files from `java.base` of JDK 21, 16 MB of source, formatted with `--replace` on 10 cores with 21.0.2:\n\n| | Wall | CPU | Peak RSS |\n|---|---|---|---|\n| Native binary | 7.0 s | 15 s, about 2 cores busy | 2.4 GB |\n| `java -cp ... Main`, JDK 21 | 6.2 s | 44 s, about 7 cores busy | 4.3 GB |\n\nReading the 865 files on one thread takes 0.04 s, and their text with the formatted text comes to some tens of megabytes. Neither is what holds the run back. In the native run, `-XX:+PrintGC` shows 40 collections that paused it for 5.0 of its 6.9 s: 17 full ones for 3.05 s and 23 incremental ones for 1.94 s, with the heap at up to 1.9 GB before a collection. The image is built without `--gc`, so it has the serial collector, which stops every thread: 20 threads format on about 2 cores.\n\n### Proposal\n\n- **One task does all of one file and hands back an immutable result.** It reads the file on its worker thread, formats it, writes it for `--replace`, and returns the path, whether the file changed, the diagnostics or the exception, and the formatted text only where it is printed, to standard output without `--replace` or `--dry-run`.\n- **The main thread passes on paths and prints results.** It takes the results in the order the files were given for what it prints, the formatted text, the paths of `--dry-run` and the messages, so that the output keeps its order, and lets go of each result once it is printed. With `--replace` no file waits for an earlier one to be written.\n- **A bound on what is in flight**, a small multiple of the pool size, so that memory does not grow with the number of files.\n\n### To measure first\n\nFor the native binary the collector is the bigger lever: a larger young generation or another collector where GraalVM offers one for the platform, and fewer than 20 threads under a serial collector. Each change can be measured on the run above.\n","author":{"url":"https://github.com/abashev","@type":"Person","name":"abashev"},"datePublished":"2026-11-13T21:41:39Z","url":"https://github.com/openjavaformat/open-java-format/issues/228"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:a91ad537-7356-d2ea-3529-793285eb9a10 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | D764:20CEDF:2C4E79:3ABCAB:6AFBB794 |
| html-safe-nonce | beaa11d17a6e7f807f9af453f1ebda3410ee5312536a5a6852c7ad5c9209a8d0 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJENzY0OjIwQ0VERjoyQzRFNzk6M0FCQ0FCOjZBRkJCNzk0IiwidmlzaXRvcl9pZCI6IjQ4NDE1OTcyMDQ0MjAwODE1NTYiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 44f911fa26ee5f7ca218d006bbc13af06d53982a14bc2ac5129cec9a16b65493 |
| hovercard-subject-tag | issue:6380471051 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/openjavaformat/open-java-format/228/issue_layout |
| twitter:image | https://opengraph.githubassets.com/425a4f20e10e4e0d22b85a2ac7e05ae695b9399346a6d628224cffeb1df59051/openjavaformat/open-java-format/issues/228 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/425a4f20e10e4e0d22b85a2ac7e05ae695b9399346a6d628224cffeb1df59051/openjavaformat/open-java-format/issues/228 |
| og:image:alt | What happens Main.formatFiles reads every file on the main thread, as new String(Files.readAllBytes(path), UTF_8), keeps the text in a map and submits it to the pool. It then takes the results on t... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | abashev |
| hostname | github.com |
| expected-hostname | github.com |
| None | 2b2ef9c1578c18d8ed27027940544c958743213cf4461659f14d3ccdf51e162c |
| turbo-cache-control | no-preview |
| go-import | github.com/openjavaformat/open-java-format git https://github.com/openjavaformat/open-java-format.git |
| octolytics-dimension-user_id | 325177626 |
| octolytics-dimension-user_login | openjavaformat |
| octolytics-dimension-repository_id | 1288055459 |
| octolytics-dimension-repository_nwo | openjavaformat/open-java-format |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | true |
| octolytics-dimension-repository_parent_id | 214273098 |
| octolytics-dimension-repository_parent_nwo | palantir/palantir-java-format |
| octolytics-dimension-repository_network_root_id | 214273098 |
| octolytics-dimension-repository_network_root_nwo | palantir/palantir-java-format |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | e4cce0e99cf61481a318ff12c2e8580153a6c7c6 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width