Title: fix: make OAuth2 refresh token redemption single-use under concurrency by BobbyHo · Pull Request #28752 · coder/coder · GitHub
Open Graph Title: fix: make OAuth2 refresh token redemption single-use under concurrency by BobbyHo · Pull Request #28752 · coder/coder
X Title: fix: make OAuth2 refresh token redemption single-use under concurrency by BobbyHo · Pull Request #28752 · coder/coder
Description: TL;DR Second and last in PLAT-481, which closes PLAT-470. Two concurrent refreshes of one refresh token both minted a replacement. #28744 fixed the same bug on the authorization code; this applies it to api_keys, so the review is a comparison against a merged precedent. It also absorbs what was #28753: the revoked-token refresh is pinned as invalid_grant, and a stored scope that names nothing stops being echoed back. PR What it does #28744 Code redemption is single-use under concurrency. #28751 A refresh may name a narrower scope. this Refresh token redemption is single-use under concurrency. A revoked token's refresh is pinned as invalid_grant. A stored scope that names nothing stops being echoed back. No migration. No new sentinel, no new dispatch case, no docs: the error class is not documented behavior. Implementation Details The fix A refresh deletes the API key the presented token hangs off and mints a replacement. That delete was a blind :exec with no affected-rows check, so the request that lost the race deleted nothing and minted anyway. DeleteAPIKeyByIDReturningRow is the same DELETE ... RETURNING * shape as its code-side sibling: a delete that removed nothing surfaces sql.ErrNoRows. RETURNING * because fetchAndQuery needs an rbac.Objecter. refreshTokenGrant maps that to errBadToken, the invalid_grant it already returns for a token it cannot find (RFC 6749 §5.2). RFC 6749 §10.4 is why the presented token is invalidated at all: a second use of it must be detectable, which holds only if exactly one refresh succeeds. Both grants now log that refusal at warn with the app and the row involved; the client still receives the generic invalid_grant. refreshTokenGrant no longer reads the previous api_keys row before deleting it. The token row has carried user_id since migration 346, so the read supplied nothing, and it sat in the one window where a refresh that lost the race still answered 500: the winner's cascade removed the key between the token read and this lookup. The returning-row delete is now the first statement to touch the key, so a lost race can only surface where it already maps to invalid_grant. Both grant transactions now name READ COMMITTED instead of inheriting default_transaction_isolation. The zero-row delete the race relies on is READ COMMITTED behavior; under REPEATABLE READ or above the same delete raises a serialization failure that nothing maps and InTx does not retry, so a raised server default would have turned every lost race into a 500. The seven other DeleteAPIKeyByID call sites are untouched, including authorizationCodeGrant's previous-key delete, where the code delete already arbitrates single use and a returning-row delete would imply otherwise. The revoked-token error class Refreshing a revoked token returned HTTP 500, because the removed GetAPIKeyByID read returned its sql.ErrNoRows raw and it fell past the sentinel dispatch to the generic handler. Removing the read is the fix; the delete already answers invalid_grant when it finds nothing. The two revocation paths a client can reach both cascade the oauth2_provider_app_tokens row away, so the prefix lookup answers errBadToken first. Deleting the API key and deleting the app secret are pinned anyway: the property a client depends on is the response, not which statement notices it, and the cascades that make them pass are schema this function does not control. The case that answered 500 is a token row whose api_key_id names no key. The FK cascade makes that unreachable through any API, so TestOAuth2RefreshKeyMissing disables the constraints to seed it, and takes a database of its own because disabling them applies to every table. It now reaches the delete and answers 400. A third path, deleting the app, never reaches the grant: the client_id no longer resolves, so ExtractOAuth2ProviderAppWithOAuth2Errors answers 401 invalid_client first. AppDeleted pins that, and the spec is corrected. The scope that names nothing CHECK (scope <> '') admits a whitespace-only scope, and scopeStringToAPIKeyScopes echoed it into error_description as an empty pair of quotes. There is no name to report, so the rejection carries a fixed message instead. StoredScopeOutsideEnumRejectedOnRefresh covers an unmintable stored scope reached through a refresh rather than through an authorization code, which is the likelier way a name dropped from the enum surfaces: a grant outlives the code that issued it. Tests TestOAuth2RefreshSingleUse races two refreshes on one barrier and requires exactly one 200 and one 400 invalid_grant. A sequential pair passes pre-fix, so the race is the test; the barrier shape is shared with TestOAuth2TokenExchangeSingleUse as requireExactlyOneAccepted. Both tests check the accepted token authenticates and that exactly two requests reached the barriered read; the refresh test also checks the presented refresh token's row is gone. TestSingleUseDeleteNotFound in dbauthz pins that a fetch miss in the fetch-then-query wrapper still matches sql.ErrNoRows and never reaches the delete. That wrapping is what makes a refused single-use delete answer invalid_grant rather than 500. An APIKey subtest in the existing TestSingleUseDelete pins the second delete returning sql.ErrNoRows. MethodTestSuite fails on any untested database.Store method, so DeleteAPIKeyByIDReturningRow gets a case alongside DeleteAPIKeyByID. Stack: #28237, #28740, #28744, #28751, this PR. Manual Tests Scenario summary, all 32 # Scenario Result 1 Exchange for an allowlisted app states the granted scope (coder:workspaces.access) Pass 2 Exchange for an app with no allowlist grants coder:all Pass 3 Minted tokens are bounded differently at the API (composite cannot create a key, coder:all can) Pass 4 The same code exchanged twice sequentially is refused invalid_grant Pass 5 Two concurrent exchanges of one code mint exactly one token Pass 6 The delete arbiter fires under a barrier client, logged code already used Pass 7 The losing exchange writes nothing (one key, one token row) Pass 8 A PKCE-failure revokes the code, so a correct retry is refused Pass 9 A refresh narrows the access token; the grant row is unchanged Pass 10 A narrowing does not bind later refreshes (sibling scope, then whole grant) Pass 11 A refresh cannot widen, and a refusal mints nothing and keeps the token usable Pass 12 An unknown and an internal-only scope both answer invalid_scope, not 500 Pass 13 A coder:all grant narrows to workspace:read (coverage, not membership) Pass 14 The code exchange narrows too Pass 15 The code exchange cannot widen, and a scope refusal does not spend the code Pass 16 The narrowed key is enforced at the API Pass 17 Refusals log phase=refresh/redeem; the field is ceiling, not allowlist Pass 18 A hostile scope name comes back NQSCHAR-clean and still names the reason Pass 19 A 16 KiB scope name comes back capped at 2060 bytes, marked (truncated) Pass 20 A fixed message (RFC 7636 section 4.1) survives the sanitizer intact Pass 21 The same refresh token used twice sequentially is refused invalid_grant Pass 22 Two concurrent refreshes of one token mint exactly one replacement Pass 23 The refresh delete arbiter fires under a barrier client, logged refresh token already used Pass 24 A successful refresh kills the previous access token; the loser writes nothing Pass 25 Refreshing after the API key is deleted answers invalid_grant Pass 26 Refreshing after the app secret is deleted answers invalid_grant Pass 27 Refreshing after the app is deleted answers invalid_client (401) Pass 28 Refreshing an orphaned token row answers invalid_grant, not the old 500 Pass 29 A refresh with a wrong or absent client_secret is accepted Pass 30 A whitespace-only stored scope answers invalid_grant with a fixed message, no '' Pass 31 Both single-use races hold under default_transaction_isolation = repeatable read, no 500 Pass 32 Cleanup drains the child tables; DCR disabled and registration refused Pass Shell helpers used throughout Run beside the server (workspace localhost:3000), fixtures registered through DCR. BASE_URL=http://localhost:3000 AUTH_HEADER="Coder-Session-Token: $(cat .coderv2/session)" pgc() { PGPASSWORD=$(cat .coderv2/postgres/password) psql -h localhost -p "$(cat .coderv2/postgres/port)" -U coder -d coder -tA -c "$1"; } urlenc() { jq -rn --arg v "$1" '$v|@uri'; } new_pkce() { VERIFIER=$(openssl rand 32 | base64 | tr -d '\n=' | tr '+/' '-_') CHALLENGE=$(printf '%s' "$VERIFIER" | openssl dgst -sha256 -binary | base64 | tr -d '=' | tr '+/' '-_') STATE=$(openssl rand -hex 16) } # authorize CLIENT_ID [SCOPE] -> sets CODE authorize() { new_pkce local url="$BASE_URL/oauth2/authorize?client_id=$1&response_type=code&redirect_uri=$(urlenc http://localhost:9876/callback)&state=$STATE&code_challenge=$CHALLENGE&code_challenge_method=S256" [ -n "${2:-}" ] && url="$url&scope=$(urlenc "$2")" local loc; loc=$(curl -s -o /dev/null -X POST "$url" -H "$AUTH_HEADER" -w '%{redirect_url}') CODE=$(python3 -c 'import sys,urllib.parse as u;print(dict(u.parse_qsl(u.urlparse(sys.argv[1]).query)).get("code",""))' "$loc") } exchange_form() { printf 'grant_type=authorization_code&client_id=%s&client_secret=%s&code=%s&code_verifier=%s&redirect_uri=%s' "$1" "$2" "$CODE" "$VERIFIER" "$(urlenc http://localhost:9876/callback)"; [ -n "${3:-}" ] && printf '&scope=%s' "$(urlenc "$3")"; } refresh_form() { printf 'grant_type=refresh_token&client_id=%s&client_secret=%s&refresh_token=%s' "$1" "$2" "$3"; [ -n "${4:-}" ] && printf '&scope=%s' "$(urlenc "$4")"; } # tok FORM: prints HTTP status + body, sets AT/RT on 200 tok() { local s; s=$(curl -s -o /tmp/tok.body -w '%{http_code}' -X POST "$BASE_URL/oauth2/tokens" -H 'Content-Type: application/x-www-form-urlencoded' --data-binary "$1"); echo "HTTP $s"; jq . /tmp/tok.body; [ "$s" = 200 ] && { AT=$(jq -r .access_token /tmp/tok.body); RT=$(jq -r .refresh_token /tmp/tok.body); }; } latest_token() { pgc "SELECT t.api_key_id, t.scope, k.scopes FROM oauth2_provider_app_tokens t JOIN api_keys k ON k.id=t.api_key_id JOIN oauth2_provider_apps a ON a.id=t.app_id WHERE a.name='$1' ORDER BY t.created_at DESC LIMIT 1;"; } A barrier-synchronized racer (racer.py) fires N POSTs at once over warmed sockets, the black-box analogue of the suite's barrierStore: import sys, threading, http.client, urllib.parse n=int(sys.argv[1]); form=sys.argv[2]; barrier=threading.Barrier(n); res=[None]*n def worker(i): c=http.client.HTTPConnection("localhost",3000); c.connect(); barrier.wait() c.request("POST","/oauth2/tokens",body=form,headers={"Content-Type":"application/x-www-form-urlencoded"}) r=c.getresponse(); r.read(); res[i]=r.status; c.close() ts=[threading.Thread(target=worker,args=(i,)) for i in range(n)] [t.start() for t in ts]; [t.join() for t in ts] from collections import Counter; print(dict(Counter(res))) Fixtures, both registered through DCR: plat481-access coder:workspaces.access (allowlist; the composite that makes narrowing observable) plat481-plain (no allowlist -> coder:all) 1. Baseline: exchange succeeds and states the grant authorize "$ACCESS_ID"; tok "$(exchange_form "$ACCESS_ID" "$ACCESS_SECRET")"; latest_token plat481-access authorize "$PLAIN_ID"; tok "$(exchange_form "$PLAIN_ID" "$PLAIN_SECRET")"; latest_token plat481-plain probe "$ACCESS_AT"; probe "$PLAIN_AT" plat481-access: HTTP 200 scope=coder:workspaces.access row: coder:workspaces.access | {coder:workspaces.access} plat481-plain : HTTP 200 scope=coder:all row: coder:all | {coder:all} composite token: GET /workspaces 200, create-token 404 (denied) coder:all token: GET /workspaces 200, create-token 201 (created) RFC 6749 §5.1 states the scope even though none was requested (§3.3 default: the whole allowlist, or coder:all). The two tokens are genuinely bounded differently. The composite's create denial is 404, not 403, because Coder hides existence on that endpoint; pre-existing. 2. Authorization code redemption is single-use (#28744) # sequential authorize "$ACCESS_ID"; F=$(exchange_form "$ACCESS_ID" "$ACCESS_SECRET"); tok "$F"; tok "$F" # concurrent (barrier racer), and the log python3 racer.py 8 "$(exchange_form ...)" # per fresh code grep -c 'code already used' coderd-dev.log sequential: HTTP 200, then HTTP 400 invalid_grant "The authorization code is invalid or expired" plain curl race (2-8 way): exactly one 200, rest 400; 'code already used' warns: 0 barrier racer (8 sockets, 8 rounds): {200:1, 400:7} each; warns: 55 [warn] code already used request_id=c4838d63... code_id=97ac3f63... (same code_id, distinct request_ids) after: tokens=1 keys=1 codes=0; deployment oauth2 keys=2 PKCE failure then correct retry: 400 (PKCE), then 400 (code revoked), no 500 Exactly one token in every configuration across ~70 contested redemptions. Note: the plain curl race is decided at the prefix read on fast local Postgres; only the barrier client reaches the DELETE ... RETURNING * arbiter (the 55 warns, same code_id under distinct request_ids), which is what the suite's barrierStore forces deterministically. 3. A refresh may narrow, and may not widen (#28751) authorize "$ACCESS_ID"; tok "$(exchange_form "$ACCESS_ID" "$ACCESS_SECRET")" tok "$(refresh_form "$ACCESS_ID" "$ACCESS_SECRET" "$RT" workspace:read)"; latest_token plat481-access # ... chained: workspace:ssh, then no scope; template:update; banana:read; debug_info:read # coder:all grant narrowed; exchange with scope; exchange over-ask refresh workspace:read -> 200 scope=workspace:read row grant=coder:workspaces.access (unchanged), key={workspace:read} chain read -> ssh -> (no scope): 200 ws:read, 200 ws:ssh, 200 coder:workspaces.access (ceiling never moved) refresh template:update -> 400 invalid_scope "beyond the scope originally granted"; nothing minted; token still redeems refresh banana:read / debug_info:read -> 400 invalid_scope "unknown or unsupported scope" (not 500) coder:all grant, refresh workspace:read -> 200 key={workspace:read}, grant coder:all exchange with scope=workspace:read -> 200 key={workspace:read}, grant coder:workspaces.access exchange with scope=template:update -> 400 invalid_scope; code NOT spent (codes=1, retries) narrowed key at API: GET /workspaces 200, create-token 404 log: phase=refresh / phase=redeem; allowlist= absent (renamed ceiling=) The narrowing lands on the access token, never the grant row (OAuth 2.1 §4.3.3); the ceiling is the grant, not the previous request (§6); coverage lets a coder:all grant narrow, which membership could not. On the exchange path the over-ask message still says "a refresh cannot widen a grant". 4. Token-endpoint error descriptions are sanitized and capped (#28751) # exact bytes via a Python client: BEL ESC[31m " \ é § # 16 KiB scope; and code_verifier=short hostile scope -> 400 invalid_scope; decoded error_description has 0 bytes outside NQSCHAR (the ", \, controls and non-ASCII removed; reason "unknown or unsupported scope" kept) 16 KiB scope -> 400 invalid_scope; error_description length 2060 (2048 cap + " (truncated)") code_verifier=short -> 400 invalid_request; message ends "(RFC 7636 section 4.1)", no § , 0 offending bytes writeTokenError runs sanitizeErrorDescription then capErrorDescription at the write, so the RFC 6749 §5.2 character set holds on the decoded value (JSON escaping alone would not) and the cap matches the authorize path. 5. Refresh token redemption is single-use (#28752) authorize "$ACCESS_ID"; tok "$(exchange_form ...)"; F=$(refresh_form "$ACCESS_ID" "$ACCESS_SECRET" "$RT"); tok "$F"; tok "$F" python3 racer.py 6 "$(refresh_form ...)" # per fresh token sequential: 200, then 400 invalid_grant "The refresh token is invalid or expired" plain curl race: one 200, one 400; 'refresh token already used' warns: 0 barrier racer (6 sockets, 5 rounds): {200:1, 400:5} each; warns: 24 (same api_key_id, distinct request_ids) after a refresh: old access token -> 401 (its key deleted), new -> 200; tokens=1 keys=1; deployment keys=2 The refresh-path twin of section 2, on api_keys via DeleteAPIKeyByIDReturningRow. The old access token dying (section 2 has no analogue) confirms the delete is real, not just an arbitration signal. 6. Revoked-token paths answer 4xx, never 500 (#28752) # key deleted / secret deleted / app deleted / FK-bypassed orphan pgc "SET session_replication_role = replica; DELETE FROM api_keys WHERE id='$KEY_ID'; SET session_replication_role = DEFAULT;" # control: refresh with a wrong / absent client_secret API key deleted -> refresh 400 invalid_grant app secret deleted-> refresh 400 invalid_grant (matches AppSecretDeleted; not 401) app deleted -> refresh 401 invalid_client (client_id no longer resolves) orphaned token row-> refresh 400 invalid_grant (was HTTP 500 before #28752) control: wrong client_secret -> 200 ; no client_secret -> 200 ; (code exchange wrong secret -> 401 invalid_client) The removed pre-delete GetAPIKeyByID read is what turns the orphan case from 500 into 400. Note: the refresh grant never compares client_secret, so a confidential client's refresh token is redeemed without authenticating (both wrong and absent secret return 200), while the exchange path authenticates correctly. RFC 6749 §§3.2.1 and 6 require authentication; pre-existing, and wider than #28751's stolen-refresh-token framing. 7 and 8. Whitespace scope, and the isolation level (#28752) pgc "UPDATE oauth2_provider_app_tokens SET scope=' ' WHERE api_key_id='$KEY_ID';"; tok "$(refresh_form ...)" pgc "ALTER DATABASE coder SET default_transaction_isolation='repeatable read';" # + drop pooled conns, rerun both barrier races whitespace-only stored scope -> 400 invalid_grant "the grant names no scope: ..." (no echoed '') under repeatable read: code + refresh barrier races -> {200:1, rest 400} each; 500s: 0; serialization errors: 0 arbiter warns still fired (80 code + 50 refresh), so the deletes did contend The CHECK (scope <> '') gap for a whitespace value now reports a fixed message. The explicit sql.LevelReadCommitted in singleUseTxOptions overrides a raised server default: the deletes contended (warns fired) yet no 40001 serialization failure surfaced as a 500. 9. Cleanup plat481 apps deleted (204); child tables drained: apps 0, codes 0, tokens 0, oauth2 keys 0 DCR disabled; register after disable -> 403; default_transaction_isolation -> read committed The cascade from app deletion took every code, token, and session key, and the run left the deployment in the clean state it started from.
Open Graph Description: TL;DR Second and last in PLAT-481, which closes PLAT-470. Two concurrent refreshes of one refresh token both minted a replacement. #28744 fixed the same bug on the authorization code; this applies ...
X Description: TL;DR Second and last in PLAT-481, which closes PLAT-470. Two concurrent refreshes of one refresh token both minted a replacement. #28744 fixed the same bug on the authorization code; this applies ...
Opengraph URL: https://github.com/coder/coder/pull/28752
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:1c42d45d-8ba3-f539-1fd0-fe78f17ce1e3 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | E912:2C4BED:23CF4EE:2D798E3:6B0183F7 |
| html-safe-nonce | 270df9b6bb229b67e4b2f0f73313156d754d583e82cd5306e81ee2b70e389956 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFOTEyOjJDNEJFRDoyM0NGNEVFOjJENzk4RTM6NkIwMTgzRjciLCJ2aXNpdG9yX2lkIjoiMTYyOTE0MjExMTQ3MzQ2ODQwNyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 61b40112af097d71c0c026e26087d0ea724413e9dbbd9de2d3cb39e0f2b8fdba |
| hovercard-subject-tag | pull_request:4389963170 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/coder/coder/pull/28752/files |
| twitter:image | https://avatars.githubusercontent.com/u/23092290?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/23092290?s=400&v=4 |
| og:image:alt | TL;DR Second and last in PLAT-481, which closes PLAT-470. Two concurrent refreshes of one refresh token both minted a replacement. #28744 fixed the same bug on the authorization code; this applies ... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | cc5dfdd32f62ebd3223cab3af3aa2b268d3605218237326a48a99d1efd89495c |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/coder/coder git https://github.com/coder/coder.git |
| octolytics-dimension-user_id | 95932066 |
| octolytics-dimension-user_login | coder |
| octolytics-dimension-repository_id | 440752086 |
| octolytics-dimension-repository_nwo | coder/coder |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 440752086 |
| octolytics-dimension-repository_network_root_nwo | coder/coder |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 8c77c9e8439031ad54c75a250694842892f693e1 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width