Title: Use postgres views to reduce database calls in api (allow `JOIN`s in pg queries) · Issue #2201 · coder/coder · GitHub
Open Graph Title: Use postgres views to reduce database calls in api (allow `JOIN`s in pg queries) · Issue #2201 · coder/coder
X Title: Use postgres views to reduce database calls in api (allow `JOIN`s in pg queries) · Issue #2201 · coder/coder
Description: This is just tracking a suggestion and making a place for discussion. Branch that implements this suggestion: stevenmasley/build_initiator_username PR that started conversation: #2137 Enable use of JOIN in sqlc queries Currently the UI r...
Open Graph Description: This is just tracking a suggestion and making a place for discussion. Branch that implements this suggestion: stevenmasley/build_initiator_username PR that started conversation: #2137 Enable use of...
X Description: This is just tracking a suggestion and making a place for discussion. Branch that implements this suggestion: stevenmasley/build_initiator_username PR that started conversation: #2137 Enable use of...
Opengraph URL: https://github.com/coder/coder/issues/2201
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Use postgres views to reduce database calls in api (allow `JOIN`s in pg queries)","articleBody":"_This is just tracking a suggestion and making a place for discussion._\r\n\r\nBranch that implements this suggestion: [`stevenmasley/build_initiator_username`](https://github.com/coder/coder/tree/stevenmasley/build_initiator_username)\r\n\r\nPR that started conversation: https://github.com/coder/coder/pull/2137\r\n\r\n---\r\n\r\n# Enable use of `JOIN` in sqlc queries\r\n\r\nCurrently the UI requires additional information regarding data structures returned by the database. An example is the `workspace_build` table does not include the following fields that the UI wants and uses [`convertWorkspaceBuild`][convertWorkspaceBuild] to add these fields.\r\n\r\n- workspace owner (id & name)\r\n- initiator name\r\n- workspace name\r\n- ~~provisioner job~~ (although included, going to not JOIN this data in right now)\r\n\r\nThis requires 2 extra database calls (fetch users, and workspace). Ideally we could fetch this information with a `JOIN`.\r\n\r\n```sql\r\n-- Make the SELECT a VIEW\r\nCREATE VIEW workspace_build_with_names AS\r\nSELECT\r\n -- coalesce is used because technically the joins do not guarantee a value.\r\n -- If we setup proper foreign keys, we can remove the coalesce.\r\n\tcoalesce(initiator_user.username, 'unknown') AS initiator_username,\r\n\tcoalesce(workspaces.owner_id, '00000000-00000000-00000000-00000000') AS owner_id,\r\n\tcoalesce(owner_user.username, 'unknown') AS owner_name,\r\n\tcoalesce(workspaces.name, 'unknown') AS workspace_name,\r\n -- This information also seems helpful for UI purposes and is used in some areas of the UI.\r\n\tcoalesce(templates.id, '00000000-00000000-00000000-00000000') AS template_id,\r\n\tcoalesce(templates.name, 'unknown') AS template_name,\r\n\tcoalesce(templates.active_version_id, '00000000-00000000-00000000-00000000') AS template_active_version,\r\n\tworkspace_builds.*\r\nFROM workspace_builds\r\n\t\t LEFT JOIN users AS initiator_user ON workspace_builds.initiator_id = initiator_user.id\r\n\t\t LEFT JOIN workspaces ON workspaces.id = workspace_builds.workspace_id\r\n\t\t LEFT JOIN users AS owner_user ON workspaces.owner_id = owner_user.id\r\n\t\t LEFT JOIN templates ON workspaces.template_id = templates.id;\r\n```\r\n\r\nThe solution to using joins is to make a migration that makes the above a VIEW in our database that can be reused. This simplifies all the queries around `workspace_builds` as we do not need to copy the `JOIN` more than once (DRY). \r\n\r\n# Downsides to views\r\n\r\n## Maintaining the view\r\n\r\nIf a column is added to `workspace_build`, that column is **not** automatically added to the view. So your migration will look like this:\r\n\r\n```sql\r\n-- We must drop the old view first, then recreate it. We cannot just update the old view\r\n-- as the new view must have the same columns.\r\nDROP VIEW workspace_build_with_initiator;\r\n\r\n-- This alter table could be adding a column OR removing a column. In both cases, we will \r\n-- need to drop the view first.\r\n-- You cannot delete a column that has a dependency on it, and a view is a dependency.\r\nALTER TABLE workspace_builds\r\n ADD extra_column int DEFAULT 1 NOT NULL;\r\n\r\nCREATE VIEW workspace_build_with_names AS\r\nSELECT -- ... Copy the Select query again ...\r\n```\r\n\r\nThe biggest downside is that is adds some complexity to our migrations. The developer writing the migration needs to know which views to also update. Postgres will fail if a view is a dependency and the underlying table is changed. They will likely have to copy the `CREATE VIEW ...` code block from migration to migration.\r\n\r\nThis is just the cost of the views.\r\n\r\n### Con: Defaulting to using joins\r\n\r\nOne downside is that all our queries now use joins. In the example above, the data being joined is only used by the UI. So internal usage doesn't require the join (at the moment). However, having the additional context could be nice for logging purposes, so it might not be a total waste.\r\n\r\nI don't think join performance will be so bad that this is really an issue, but it does cause over-fetching. \r\n\r\nIf we use the [sqlc.embed][embed] fork of sqlc, we can make db calls with the join, and some db calls without the join. Having this mix can prevent overfetching if we hit performance issues. Using the `embed` means we can nest data-structs and have the UI data-structure be something like:\r\n\r\n```golang\r\ntype WorkspaceBuildWithNames struct {\r\n // The backend can work exclusively with 'WorkspaceBuild' type. So calling the FE version of the call\r\n // is still usable for BE purposes.\r\n WorkspaceBuild\r\n OwnerName string\r\n // .. The rest of the fields\r\n}\r\n```\r\n\r\n## The upside \r\n\r\n- Less db calls as we can merge them into 1 with a join\r\n- Still 1 `database.Type` shared by all sqlc queries\r\n- Join queries are reused, and only typed once as a view.\r\n- Filter options can use names rather than ids. (owner_name vs owner_id from a user perspective)\r\n\r\n## Alternative (Status Quo)\r\n\r\nThe alternative is to do the status quo, where we make multiple db calls and merge the resulting structs. \r\n\r\n- Query for workspace build\r\n- Query for owner & initiator user's\r\n- Merge in a go `convertWorkspaceBuild` function.\r\n - Handling errors in convert is currently inconsistent. The SQL joins handle these more consistently.\r\n\r\n### Extra links\r\n\r\nSQLc implementations that could help. But gated since these features do not exist yet :(.\r\n\r\n- `sqlc.embed()` fork: https://github.com/kyleconroy/sqlc/pull/1615. \r\n- This was a design sqlc dropped, but [`template: workspace_build_initiator`](https://github.com/kyleconroy/sqlc/issues/112) support for putting the views in the queries.sql\r\n - Downside is that we have to copy the view's SELECT query with the join to every query. \r\n\r\n[embed]: https://github.com/kyleconroy/sqlc/pull/1615\r\n[convertWorkspaceBuild]: https://github.com/coder/coder/blob/fff59ef6ad4e1d9026b228e8d2164004a5c7ff60/coderd/workspacebuilds.go#L524-L524","author":{"url":"https://github.com/Emyrk","@type":"Person","name":"Emyrk"},"datePublished":"2022-06-09T14:04:10Z","url":"https://github.com/coder/coder/issues/2201"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:3a5cf5c0-7525-30bd-d773-8975fd7051fb |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | E31C:A71F9:174E24A:1D5CFDF:6B01756E |
| html-safe-nonce | 665ac2b4e4f0990893070185841826a67f42c64deb6ebd21445e96a3d66a99b4 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFMzFDOkE3MUY5OjE3NEUyNEE6MUQ1Q0ZERjo2QjAxNzU2RSIsInZpc2l0b3JfaWQiOiIzMzAwNjc2NzgxMDE4ODA1NjE0IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 59968970ed8d66a3e11d6f7477fb0972c970615616c53d76de7a05773037e520 |
| hovercard-subject-tag | issue:1266186795 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/coder/coder/2201/issue_layout |
| twitter:image | https://opengraph.githubassets.com/7ae9b79e1b8d0856bd203399742c136bcc4a82ca0d1d476d7522d16d19e1a0e2/coder/coder/issues/2201 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/7ae9b79e1b8d0856bd203399742c136bcc4a82ca0d1d476d7522d16d19e1a0e2/coder/coder/issues/2201 |
| og:image:alt | This is just tracking a suggestion and making a place for discussion. Branch that implements this suggestion: stevenmasley/build_initiator_username PR that started conversation: #2137 Enable use of... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | Emyrk |
| hostname | github.com |
| expected-hostname | github.com |
| None | cc5dfdd32f62ebd3223cab3af3aa2b268d3605218237326a48a99d1efd89495c |
| turbo-cache-control | no-preview |
| go-import | github.com/coder/coder git https://github.com/coder/coder.git |
| octolytics-dimension-user_id | 95932066 |
| octolytics-dimension-user_login | coder |
| octolytics-dimension-repository_id | 440752086 |
| octolytics-dimension-repository_nwo | coder/coder |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 440752086 |
| octolytics-dimension-repository_network_root_nwo | coder/coder |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 8c77c9e8439031ad54c75a250694842892f693e1 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width