René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"[RFC] Support for horizontal scaling Sourcebot","articleBody":"This discussion tracks adding support for horizontal scaling to Sourcebot.\n\n## TL;DR\n- I think the existing single, self-contained [docker container](https://github.com/sourcebot-dev/sourcebot/pkgs/container/sourcebot) is great since it's super easy to use and is [radically simple](https://www.radicalsimpli.city/).\n- But it's probably not practical in larger deployments, especially ones with thousands of repositories.\n- We've seen instances where when a machine is under heavy indexing load, the webapp can become unresponsive or even OOM errors can occur.\n- I think the first logical step here is to distribute our services as separate images (i.e., `ghcr.io/sourcebot-dev/(webapp|worker|zoekt)`)\n- Upstream dependencies like [Postgres](https://hub.docker.com/_/postgres), and [Redis](https://hub.docker.com/_/redis) would be separate.\n- More images = more complexity, so I think the best mechanism to manage this complexity is moving towards **declarative configs**. We could do the following:\n  - **For local deployments**, we can define a `docker-container.yml` at the root of the repository. Deploying will now involve 1) Cloning the repo, and 2) running `docker compose up`.\n  - **For production deployments**, we can use technologies such as [Helm](https://helm.sh/), [Terraform](https://developer.hashicorp.com/terraform), [Railway](https://railway.com/), and [NixOS](https://nixos.org/)\n- I think it will be important to have sensible defaults (s.t., it's very easy to deploy), while having deep configuration options that are **well documented** (s.t., it fits the user's deployment story).\n\n## Open questions / investigations\n- What will we need to do to make each one of our services (webapp, worker, zoekt) to support multiple replicas at the same time?\n\n## Is this Overkill?\nI'm trying not to go too overboard with the design here. DX should not be sacrificed with this change: ideally it should improve. Sourcebot should be **just as easy** to deploy as a vertically scaling service, both locally and in production. Deploying it as a horizontally scaling service in production should also be easy, while including configuration depth to support a wide range of deployment scenarios.\n\nLocally, using Docker Compose will streamline deployment since a) it's a single command, `docker compose up`, and b) it's declarative, so all of the options (w/ sensible defaults, or commented out) can be included inside the `docker-compose.yml`.\n\nIn production, using Helm, Terraform, NixOS, etc. will help get people setup quickly. Again, sensible defaults here are key.\n\nWe also should continue supporting the self-contained image since to avoid introducing a breaking change here. We can evaluate if it's still worth shipping whenever we do a v5.","author":{"url":"https://github.com/brendan-kellam","@type":"Person","name":"brendan-kellam"},"datePublished":"2025-08-07T05:59:42.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/439/sourcebot/issues/439"}