Title: Bug: Agent loses authorization before `run_on_stop` script completes · Issue #19467 · coder/coder · GitHub
Open Graph Title: Bug: Agent loses authorization before `run_on_stop` script completes · Issue #19467 · coder/coder
X Title: Bug: Agent loses authorization before `run_on_stop` script completes · Issue #19467 · coder/coder
Description: Summary The Coder agent is losing authorization and API connectivity before run_on_stop scripts complete execution, violating the documented behavior and preventing proper workspace shutdown operations. Expected Behavior According to the...
Open Graph Description: Summary The Coder agent is losing authorization and API connectivity before run_on_stop scripts complete execution, violating the documented behavior and preventing proper workspace shutdown operat...
X Description: Summary The Coder agent is losing authorization and API connectivity before run_on_stop scripts complete execution, violating the documented behavior and preventing proper workspace shutdown operat...
Opengraph URL: https://github.com/coder/coder/issues/19467
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Bug: Agent loses authorization before `run_on_stop` script completes","articleBody":"\n## Summary\n\nThe Coder agent is losing authorization and API connectivity before `run_on_stop` scripts complete execution, violating the documented behavior and preventing proper workspace shutdown operations.\n\n## Expected Behavior\n\nAccording to the [Terraform provider documentation](https://registry.terraform.io/providers/coder/coder/latest/docs/resources/script):\n\n> **run_on_stop** (Boolean) This option defines whether or not the script should run when the agent stops. **The script should exit when it is done to signal that the workspace can be stopped.**\n\nThe agent would be expected to maintain full functionality (including external auth access and API connectivity) until the `run_on_stop` script completes and signals that the workspace can be stopped.\n\nThe alternative is to change the wording in the terraform article to 'should run after the agent stops'.\n\n## Actual Behavior\n\nThe agent loses authorization and API connectivity during `run_on_stop` script execution, causing:\n\n1. `coder external-auth access-token` commands to fail with \"Workspace agent not authorized\"\n2. API connectivity tests to fail\n3. Git operations requiring authentication to fail\n\nThis reduces the ability of a shutdown script to perform actions.\n\n## Evidence\n\nThe example is from a shutdown script which is created and should perform GIT commands on shutdown.\n\n### Logs from `run_on_stop` script execution:\n\n```\n[2025-08-21 12:27:14] === On-stop demo script started ===\n[2025-08-21 12:27:14] System information:\n[2025-08-21 12:27:14] Hostname: coder-workspace-pod\n[2025-08-21 12:27:14] Uptime: 12:27:14 up 4:31, 0 user, load average: 0.19, 0.32, 0.33\n[2025-08-21 12:27:14] Disk usage: overlay 61G 42G 17G 72% /\n[2025-08-21 12:27:14] Workspace information:\n[2025-08-21 12:27:14] Workspace name: test-workspace\n[2025-08-21 12:27:14] Workspace owner: user\n[2025-08-21 12:27:14] Workspace ID: [workspace-id]\n[2025-08-21 12:27:14] Coder API connectivity: FAILED ← API already failing\n[2025-08-21 12:27:14] Looking for Git repositories in /home/coder/src/server...\n[2025-08-21 12:27:14] Found Git repository at: /home/coder/src/server/repo\n[2025-08-21 12:27:14] Git status:\n M TEST9.TXT\n?? TEST10.TXT\n[2025-08-21 12:27:14] Current branch: main\n[2025-08-21 12:27:14] All branches: * main\n remotes/origin/HEAD -> origin/main\n remotes/origin/main\n[2025-08-21 12:27:14] Remote branches: origin/HEAD -> origin/main\n origin/main\n[2025-08-21 12:27:14] Remote URL: https://github.com/user/repo\n[2025-08-21 12:27:14] Last commit: abc123 Manual commit\n[2025-08-21 12:27:14] Upstream branch: origin/main\n[2025-08-21 12:27:14] Repository has full history\n[2025-08-21 12:27:14] Created workspace status file\n[2025-08-21 12:27:14] Found uncommitted changes, creating commit...\n[main def456] Auto-save on workspace stop - Thu Aug 21 12:27:14 PM UTC 2025\n 3 files changed, 3 insertions(+), 1 deletion(-)\n create mode 100644 TEST10.TXT\n[2025-08-21 12:27:14] Attempting to get GitHub token directly...\n[2025-08-21 12:27:14] ✗ Failed to get GitHub token directly ← External auth failing\n[2025-08-21 12:27:14] This may indicate that GitHub external auth is not configured or the agent cannot access it\n[2025-08-21 12:27:14] Attempting push without authentication...\nEncountered an error running \"coder\", see \"coder --help\" for more information\nerror: Trace=[get git token: ]\nWorkspace agent not authorized. ← Agent unauthorized during script execution\nSuggestion: Try logging in using 'coder login'.\nerror: unable to read askpass response from '/tmp/coder.XXXXXX/coder'\nfatal: could not read Username for 'https://github.com': No such device or address\n[2025-08-21 12:27:15] Saved workspace metadata to /tmp/workspace-metadata.json\n[2025-08-21 12:27:15] Cleaning up temporary files...\n[2025-08-21 12:27:15] === On-stop demo script completed successfully ===\n[2025-08-21 12:27:15] Check /home/coder/shutdown.log for full execution details\n```\n\n### Key Issues Observed:\n\n1. **API Connectivity Fails**: `curl -s -H \"Coder-Session-Token: ${CODER_AGENT_TOKEN}\" \"${CODER_URL}/api/v2/workspaces\"` returns failure\n2. **External Auth Fails**: `coder external-auth access-token GH` returns \"Workspace agent not authorized\"\n3. **Script Continues**: The script continues to run and completes successfully, but without the expected agent capabilities\n\n## Impact\n\nThis bug prevents legitimate use cases such as:\n\n- Auto-committing and pushing code changes on workspace stop\n- Backing up workspace state to external services\n- Performing authenticated cleanup operations\n- Any operation requiring external auth or API access during shutdown\n\n## Similar Issues\n\n- https://github.com/coder/coder/issues/17108\n- https://github.com/coder/coder/issues/13387\n\n## Workaround\n\nWe've created a workaround that caches external auth tokens during workspace startup and uses them during shutdown:\n\n```hcl\nresource \"coder_script\" \"startup_auth\" {\n agent_id = coder_agent.main.id\n display_name = \"Cache Auth Tokens\"\n run_on_start = true\n \n script = \u003c\u003c-EOT\n #!/bin/bash\n mkdir -p ~/.cache/coder\n # Cache GitHub token for use in shutdown script\n if coder external-auth access-token GH > ~/.cache/coder/github_token 2>/dev/null; then\n chmod 600 ~/.cache/coder/github_token\n echo \"Successfully cached GitHub token\"\n fi\n EOT\n}\n\nresource \"coder_script\" \"shutdown_script\" {\n agent_id = coder_agent.main.id\n display_name = \"Auto-save on Stop\"\n run_on_stop = true\n \n script = \u003c\u003c-EOT\n #!/bin/bash\n # Use cached token instead of direct external-auth access\n if [[ -f ~/.cache/coder/github_token ]]; then\n GITHUB_TOKEN=$(cat ~/.cache/coder/github_token)\n # Use token for git operations...\n fi\n EOT\n}\n```\n\nHowever, this workaround has security implications:\n- Tokens are stored on disk in the workspace\n- Tokens may expire during long-running workspaces (24 hours)\n- It adds complexity and potential failure points\n\n## Environment\n\n- **Coder Version**: Dev based off 2.24.X-2.25.1\n- **Platform**: Kubernetes\n- **Template**: Using `coder_script` resource with `run_on_stop = true`\n- **External Auth**: GitHub configured and working during normal operation\n\n## Reproduction Steps\n\n1. Create a template with a `run_on_stop` script that uses `coder external-auth access-token`\n2. Create a workspace from the template\n3. Stop the workspace\n4. Observe that external auth fails during script execution\n\n## Expected Fix\n\nThe agent should maintain authorization and API connectivity until the `run_on_stop` script exits, as documented. The agent shutdown sequence should be:\n\n1. Trigger `run_on_stop` scripts\n2. Wait for all `run_on_stop` scripts to complete\n3. **Only then** revoke agent authorization and stop the agent\n4. Proceed with workspace shutdown\n\nThis would align with the documented behavior and enable proper workspace cleanup operations.","author":{"url":"https://github.com/bjornrobertsson","@type":"Person","name":"bjornrobertsson"},"datePublished":"2025-08-21T12:57:37Z","url":"https://github.com/coder/coder/issues/19467"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:c850bd6f-43cb-1bcf-68c2-15b3a4ca69f0 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | A662:2454CC:29B6FBC:34F1D1C:6B01A711 |
| html-safe-nonce | 3742ae55cc55f33961389c00db1dfd68f4271aedf893f01c7463e98780b9f114 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBNjYyOjI0NTRDQzoyOUI2RkJDOjM0RjFEMUM6NkIwMUE3MTEiLCJ2aXNpdG9yX2lkIjoiNjE3NzM5OTUxMTcwNzY1ODAwMSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 874d3f0437fdf9bdb7921a02d7c15c0bbf245c8690dcb56ab2cda9254c5bb929 |
| hovercard-subject-tag | issue:3341653560 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/coder/coder/19467/issue_layout |
| twitter:image | https://opengraph.githubassets.com/b654f9f6c75808393bee7294eddc5e749ae8b3625d9b11105cbdd57f6ae3888e/coder/coder/issues/19467 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/b654f9f6c75808393bee7294eddc5e749ae8b3625d9b11105cbdd57f6ae3888e/coder/coder/issues/19467 |
| og:image:alt | Summary The Coder agent is losing authorization and API connectivity before run_on_stop scripts complete execution, violating the documented behavior and preventing proper workspace shutdown operat... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | bjornrobertsson |
| hostname | github.com |
| expected-hostname | github.com |
| None | cc5dfdd32f62ebd3223cab3af3aa2b268d3605218237326a48a99d1efd89495c |
| turbo-cache-control | no-preview |
| go-import | github.com/coder/coder git https://github.com/coder/coder.git |
| octolytics-dimension-user_id | 95932066 |
| octolytics-dimension-user_login | coder |
| octolytics-dimension-repository_id | 440752086 |
| octolytics-dimension-repository_nwo | coder/coder |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 440752086 |
| octolytics-dimension-repository_network_root_nwo | coder/coder |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 8c77c9e8439031ad54c75a250694842892f693e1 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width