René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"feat: progress bar during query in `QueryJob.to_dataframe` and `QueryJob.to_arrow`","articleBody":" **Is your feature request related to a problem? Please describe.**\r\n\r\nWhen running a query via the `%%bigquery` magics or waiting for it to finish via [QueryJob.to_dataframe](https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.job.QueryJob.html#google.cloud.bigquery.job.QueryJob.to_dataframe) or [QueryJob.to_arrow](https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.job.QueryJob.html#google.cloud.bigquery.job.QueryJob.to_arrow), an argument `progress_bar_type` is accepted.\r\n\r\nCurrently, this only shows the progress of the query results download. It would be great if it would also give an indicator while the query is executing.\r\n\r\n\r\n **Describe the solution you'd like**\r\n\r\nWhen a value is passed to `progress_bar_type`, show some kind of progress bar. Ideally, it would work similarly to the UI. For example,\r\n\r\n* Show the job state in the progress bar description. For example, is it currently \"pending\" (queued, waiting for resources) or \"running\" (actually executing).\r\n* Show how many \"stages\" there are via length of the [query_plan](https://github.com/googleapis/python-bigquery/blob/d093cd4ee255bbcbe5a6a198e819b876b4aa51f9/google/cloud/bigquery/job.py#L2841).\r\n* Find the latest incomplete stage.\r\n* Use [parallel_inputs](https://github.com/googleapis/python-bigquery/blob/d093cd4ee255bbcbe5a6a198e819b876b4aa51f9/google/cloud/bigquery/job.py#L3514) as the total amount of work (per stage)\r\n* Use [completed_parallel_inputs](https://github.com/googleapis/python-bigquery/blob/d093cd4ee255bbcbe5a6a198e819b876b4aa51f9/google/cloud/bigquery/job.py#L3521) as the amount of work completed so far.\r\n\r\nTo populate this, instead of calling `result()` once:\r\n\r\n* Call `result(timeout=[a few seconds])` every few seconds.\r\n* Call `job.reload()` to fetch the latest job statistics.\r\n* Update the progress bar.\r\n* Repeat.\r\n\r\n **Describe alternatives you've considered**\r\n\r\n* Some kind of spinner that only shows that some time has elapsed. (This is actually harder than it sounds because `tqdm` doesn't actually support spinners, and if it ever does, it sounds like it'll be via a different API than the current progress bars. tqdm/tqdm#427 tqdm/tqdm#925\r\n\r\n **Additional context**\r\n\r\n* pandas-gbq feature request: https://github.com/pydata/pandas-gbq/issues/327\r\n* Example of BigQuery UI progress bar:\r\n\r\n  ![bq-progress-small](https://user-images.githubusercontent.com/247555/97185499-1e78b280-176e-11eb-86e6-44d35fa98b15.gif)","author":{"url":"https://github.com/tswast","@type":"Person","name":"tswast"},"datePublished":"2020-10-26T14:49:47.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/343/python-bigquery/issues/343"}