René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"darwin tarball binaries are not notarized at release time","articleBody":"Thanks @sxa for bringing this to my attention. It has been a while since I last touched the macOS notarization, so I went back to it with fresh eyes.\n\n> **Update 2026-10-31:** the \"no notarization ticket\" findings below were produced with `codesign --verify -R=notarized`, which only reads the local ticket cache. Asking Apple's ticket service directly shows that **15 of the 18 most recent tarballs do have a ticket**, created days after release by third parties that bundle our binary. Only v26.13.0 (both arches) and v26.12.0 x64 lack one. The root cause and the fix are unchanged; the gap is the first days after every release. Details in [this comment](https://github.com/nodejs/node/issues/67140#issuecomment-6528884555). Struck text below is kept for context.\n\n### Context\n\nWe have had reports about badly signed or non notarized macOS binaries for a long time:\n\n- nodejs/build#3538 (2023): Gatekeeper refuses to start `node` from the tarball, \"Notary Ticket Missing\".\n- nodejs/node#52610 (2024): my first attempt to notarize the tarballs, stalled on the Jenkins side and closed as stale.\n- nodejs/build#4527 (2026-10-19): pkg notarization ticket not visible right after `Accepted`, fixed with a retry loop in nodejs/node#66855.\n- nodejs/build#4545 (2026-10-29): v26.13.0 tarball binaries have no ticket, `syspolicyd` slows every spawn.\n- nodejs/help#5173, #5176, #5178 (October 2026): \"invalid signature\" reports on Node 24 artifacts that collaborators cannot reproduce.\n\nHistorically this was hard to reproduce locally, which is why #52610 never got the testing it needed.\n\n### [Solved] local reproduction at least to debug\n\nI wrote a script that downloads the darwin tarball and the pkg for a version, verifies signatures, asks Apple whether a notarization ticket exists, and compares the tarball binary with the one inside the pkg. It needs no credentials and executes nothing it downloads, so anyone can run it on any _modern_ macOS version and architecture: https://gist.github.com/UlisesGascon/ddce84a3f80521324e1c77bbfba12dba.\n\n```sh\nsh macos-validate.sh v26.12.1 v26.13.0 v24.22.1\nDIST=https://nodejs.org/download/nightly ARCH=arm64 sh macos-validate.sh v28.0.0-nightly202610304e48b9b06b\n```\n\nRelevant output from my execution, same result for all three releases and for the nightly:\n\n```text\n$ codesign --verify --strict --verbose=2 node-v26.13.0-darwin-x64/bin/node\nnode-v26.13.0-darwin-x64/bin/node: valid on disk\nnode-v26.13.0-darwin-x64/bin/node: satisfies its Designated Requirement\n  exit=0\n$ codesign --verify --verbose=2 -R=notarized node-v26.13.0-darwin-x64/bin/node\ntest-requirement: code failed to satisfy specified code requirement(s)\n  exit=3\n\n$ pkgutil --check-signature node-v26.13.0.pkg\n   Status: signed by a developer certificate issued by Apple for distribution\n   Notarization: trusted by the Apple notary service\n  exit=0\n$ xcrun stapler validate node-v26.13.0.pkg\nThe validate action worked!\n  exit=0\n\n  CDHash DIFFER: tarball=f2a61f63d9bcc07d74e1f67c443c85cff7a5cdd5 pkg=940e23498f45871c88f11bdf04d07760a4e243a7\n```\n\n~~So every tarball `bin/node` is correctly signed with our Developer ID but has no notarization ticket, on x64 for 24.22.1, 26.12.1 and 26.13.0, and on arm64 for today's nightly.~~ **Correction:** the signatures are fine, but the \"no ticket\" results came from `codesign -R=notarized`, which only reads the local cache. Apple's ticket service shows tickets for 24.22.1 and 26.12.1 on both arches (created 2026-10-21 to 10-23, by third parties) and none for 26.13.0 and the nightly. Every pkg is notarized and stapled. The pkg's `bin/node` is a different build from the tarball's, so the pkg ticket does not cover it. The \"invalid signature\" errors in the help issues do not reproduce on 15.0.1, 26.4.1 or 27.0.1.\n\n### What is wrong here?\n\n1. The [`binary` target](https://github.com/nodejs/node/blob/97aed16adb90a720308d7ebea417c9a9db81ea26/Makefile#L1435-L1447) in the Makefile only runs [`tools/osx-codesign.sh`](https://github.com/nodejs/node/blob/97aed16adb90a720308d7ebea417c9a9db81ea26/tools/osx-codesign.sh#L16-L21). [`tools/osx-notarize.sh`](https://github.com/nodejs/node/blob/97aed16adb90a720308d7ebea417c9a9db81ea26/tools/osx-notarize.sh) is only called by the [`pkg` target](https://github.com/nodejs/node/blob/97aed16adb90a720308d7ebea417c9a9db81ea26/Makefile#L1250), takes a version id and [hardcodes `node-$pkgid.pkg`](https://github.com/nodejs/node/blob/97aed16adb90a720308d7ebea417c9a9db81ea26/tools/osx-notarize.sh#L39-L42) as the thing to submit. Tarball binaries have never been submitted to Apple.\n2. On the Jenkins side the `NOTARIZATION_*` variables that [gate the script](https://github.com/nodejs/node/blob/97aed16adb90a720308d7ebea417c9a9db81ea26/tools/osx-notarize.sh#L12-L25) are only exported on the pkg axis, so even a Makefile change would be a no-op on the tar nodes until the job is adjusted.\n3. There is no verification step for tarballs. The pkg gets [`spctl --assess` and a stapler check](https://github.com/nodejs/node/blob/97aed16adb90a720308d7ebea417c9a9db81ea26/tools/osx-notarize.sh#L67-L78); the tar job has nothing, so an unticketed binary ships silently.\n4. ~~The one tarball known to carry a ticket (26.12.1 arm64, per nodejs/build#4545) is the exception, probably from a manual submission during the #4527 debugging, not evidence of a working path.~~ **Correction:** most tarball binaries do end up with a ticket, one to seven days after release, because third parties notarize apps that bundle our signed `bin/node` and Apple issues a ticket for every nested binary. That is accidental and late, not a working path; 26.13.0 shows what the first days after a release look like. The pkg's `bin/node` cannot cover it: the pkg target configures with the default prefix and the tarball target with [`--prefix=/` and `PORTABLE=1`](https://github.com/nodejs/node/blob/97aed16adb90a720308d7ebea417c9a9db81ea26/Makefile#L1421-L1426), so the two binaries always differ.\n\n### Proposal\n\nTwo PRs plus one Jenkins change:\n\n1. **nodejs/node**: make `tools/osx-notarize.sh` take a path, keep the pkg flow as is, and add a binary flow: zip `bin/node` with `ditto`, submit with notarytool, verify that Apple's ticket service returns a ticket for the binary's cdhash, using the existing retry helper, fail the build if no ticket appears. Call it from the `binary` target after codesign. The binary is not modified, so the published checksums do not change. Backport to 22, 24 and 26.\n2. **nodejs/build**: a `tools/macos-tarball-notarization.sh` that checks any published version for a ticket against Apple's ticket service (`codesign -R=notarized` is not a reliable check, see the update) and, with `--submit` and the release keychain profile, backfills the ones that lack it. Plus docs in `release-overview.md` describing the signing and notarization flow and how to enable it on a job axis.\n3. **ci-release `iojs+release`**: export `NOTARIZATION_ID` and `NOTARIZATION_TEAM_ID` on the macOS tarball axes. `NOTARIZATION_PASSWORD` already reaches every axis through the job-level credentials binding `apple-notarization-password`, and the release image already has the cert and the `NODE_RELEASE_PROFILE` keychain profile on every macOS release VM, so nothing changes in ansible or orka. The job has one conditional step per axis type; the pkg one injects the variables, the tar one does not. The change is two lines in the tar step's \"Inject environment variables\" block:\n\n```diff\n \u003corg.jenkinsci.plugins.conditionalbuildstep.ConditionalBuilder>\n   \u003crunCondition class=\"org.jenkins_ci.plugins.run_condition.core.ExpressionCondition\">\n     \u003cexpression>osx1(\\d|\\d\\d\\d)-release-tar|macos1(\\d|\\d\\d\\d)-release-tar\u003c/expression>\n     \u003clabel>${ENV, var=\"nodes\"}\u003c/label>\n   \u003c/runCondition>\n   \u003cconditionalbuilders>\n     \u003cEnvInjectBuilder>\n       \u003cinfo>\n-        \u003cpropertiesContent>CODESIGN_CERT=Developer ID Application: Node.js Foundation\u003c/propertiesContent>\n+        \u003cpropertiesContent>CODESIGN_CERT=Developer ID Application: Node.js Foundation\n+NOTARIZATION_ID=REDACTED\n+NOTARIZATION_TEAM_ID=REDACTED\u003c/propertiesContent>\n       \u003c/info>\n     \u003c/EnvInjectBuilder>\n     \u003chudson.tasks.Shell>\n       \u003ccommand>... security unlock-keychain -p ${SSH_RELEASE_PASSWORD} ... make -j $JOBS binary-upload ...\u003c/command>\n     \u003c/hudson.tasks.Shell>\n   \u003c/conditionalbuilders>\n \u003c/org.jenkinsci.plugins.conditionalbuildstep.ConditionalBuilder>\n```\n\n\n### Backlog\n\nWe can split into 2 goals (notarize on release and backfill) as they are independent and the priorities differ too.\n\n**Notarize on release**\n- [x] @UlisesGascon create the Node PR: #67143\n\t- [ ] land the PR\n- [ ] @UlisesGascon fork the existing `iojs+release` job into a new one and execute it\n- [ ] @UlisesGascon validate the results and decide if we apply the changes on `iojs+release` and remove the temporal job\n- [ ] @UlisesGascon check the nighthly buils now are properly notarize\n\n**Backfill**\n- [x] @UlisesGascon create the Build PR: https://github.com/nodejs/build/pull/4549\n\t- [ ] land the PR\n- [ ] @nodejs/build to decide if we want a pipeline job just to backfill\n\t- [ ] @UlisesGascon create the job to check and backfill recent releases in Jenkins release\n- [ ] @UlisesGascon backfill the missing binaries without a ticket via ssh on a release VM\n\t- [x] @UlisesGascon to check the total affected packages: see https://github.com/nodejs/build/pull/4549#issuecomment-6530505036.  In non-EOL 136 already have a notarization ticket and 42 do not, (29 x64 and 13 arm64).\n","author":{"url":"https://github.com/UlisesGascon","@type":"Person","name":"UlisesGascon"},"datePublished":"2026-10-30T13:29:15Z","url":"https://github.com/nodejs/node/issues/67140"}