René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Support fetching very large extracts via CSV/download export (beyond query.maxLimit)","articleBody":"## Background\n\nFollow-up to #19. That issue/PR (#25) fixed the SDK's incorrect 50k cap so the query results endpoint can now return up to the instance's configured `query.maxLimit` (e.g. 100,000 on Cloud).\n\nHowever, the **interactive query results endpoint is itself capped at `query.maxLimit`**. Data-extraction / pipeline workflows that need *millions* of rows (the use case that motivated #19) can't be served by that endpoint — the Lightdash UI uses a separate **CSV/download export** path for those, governed by `query.csvCellsLimit` / `query.csvMaxLimit` (default ceiling 5,000,000 cells).\n\n## Proposed solution\n\nAdd an export-based fetch path to the SDK for result sets larger than `query.maxLimit`, e.g.:\n\n```python\nresult = query.to_df(via=\"export\")   # or query.export() / result.download()\n```\n\nThis would POST to the CSV/scheduler-style export endpoint, poll for the generated file, and stream it back (CSV → DataFrame), rather than paging the interactive results API.\n\n### Notes / open questions\n- Confirm the exact endpoint(s): the download/`/csv` or scheduler export API used by the UI's \"Export\" / \"Download results\".\n- Respect `query.csvCellsLimit` (rows × columns) and surface a clear error when exceeded, consistent with the no-silent-truncation behaviour added in #19.\n- Decide on the surface area: a flag on `to_df()`/`to_records()` vs. a dedicated `export()` method.\n\n### References\n- #19 (row-limit fix) and PR #25\n- Lightdash limits: `query.maxLimit`, `query.csvCellsLimit`, `query.csvMaxLimit` (see `docs/export-limits.md` in lightdash/lightdash)","author":{"url":"https://github.com/jpetey75","@type":"Person","name":"jpetey75"},"datePublished":"2026-06-17T02:25:21.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/26/python-sdk/issues/26"}