René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"improve test latency","articleBody":"recent runs of presubmit kokoro show we're taking something on the order of 37-48 minutes for typical runs, with a couple invocations significantly higher than that.  It's a significant impediment to actually getting things done.\r\n\r\nWe can definitely test much faster than this.\r\n\r\nSome things to look at:\r\n\r\n* Decompose the monolilthic kokoro job that runs all the nox sessions serially into parallel invocations, the same way we decompose the samples invocations.  Fan out to get better wall time.\r\n\r\n* Look for obvious slow tests and improve them, reducing the tail of the (to be) parallelized test runs.  Based on logging, system tests are unsuprisingly the longer session typically. \r\n\r\n* Look at coverage overlap between samples and library; we may be able to reduce redundancy in testing.\r\n\r\n\r\nHere's some quick info from running locally on 3.8:\r\n\r\n```\r\n$ nox -s system-3.8 -- --durations=0\r\n\r\n44.84s call     tests/system.py::TestBigQuery::test_dbapi_w_standard_sql_types\r\n29.28s call     tests/system.py::TestBigQuery::test_dbapi_w_query_parameters\r\n27.39s call     tests/system.py::TestBigQuery::test_query_many_columns\r\n15.91s call     tests/system.py::TestBigQuery::test_query_w_query_params\r\n15.04s call     tests/system.py::TestBigQuery::test_load_table_from_file_w_explicit_location\r\n13.98s call     tests/system.py::TestBigQuery::test_load_table_from_dataframe_w_automatic_schema\r\n13.79s call     tests/system.py::TestBigQuery::test_query_w_standard_sql_types\r\n13.19s call     tests/system.py::TestBigQuery::test_load_table_from_dataframe_w_nullable_int64_datatype\r\n11.90s call     tests/system.py::TestBigQuery::test_load_table_from_local_avro_file_then_dump_table\r\n11.19s call     tests/system.py::TestBigQuery::test_load_table_from_dataframe_w_required\r\n10.83s call     tests/system.py::TestBigQuery::test_nested_table_to_dataframe\r\n10.50s call     tests/system.py::TestBigQuery::test_load_table_from_dataframe_w_nulls\r\n10.30s call     tests/system.py::TestBigQuery::test_load_table_from_json_schema_autodetect\r\n10.11s call     tests/system.py::TestBigQuery::test_copy_table\r\n9.60s call     tests/system.py::TestBigQuery::test_load_table_from_dataframe_w_explicit_schema\r\n8.56s call     tests/system.py::TestBigQuery::test_dbapi_fetchall\r\n8.33s call     tests/system.py::TestBigQuery::test_extract_table\r\n7.61s call     tests/system.py::TestBigQuery::test_nested_table_to_arrow\r\n7.20s call     tests/system.py::TestBigQuery::test_load_table_from_dataframe_w_nullable_int64_datatype_automatic_schema\r\n7.09s call     tests/system.py::TestBigQuery::test_load_table_from_uri_then_dump_table\r\n7.03s call     tests/system.py::TestBigQuery::test_list_rows_page_size\r\n6.76s call     tests/system.py::TestBigQuery::test_query_w_legacy_sql_types\r\n6.67s call     tests/system.py::TestBigQuery::test_dbapi_connection_does_not_leak_sockets\r\n6.34s call     tests/system.py::TestBigQuery::test_query_w_dml\r\n6.23s call     tests/system.py::TestBigQuery::test_dbapi_w_dml\r\n5.71s call     tests/system.py::TestBigQuery::test_load_avro_from_uri_then_dump_table\r\n4.26s call     tests/system.py::TestBigQuery::test_load_table_from_json_basic_use\r\n3.32s call     tests/system.py::TestBigQuery::test_querying_data_w_timeout\r\n3.12s call     tests/system.py::test_bigquery_magic\r\n2.89s call     tests/system.py::TestBigQuery::test_list_tables\r\n2.87s call     tests/system.py::TestBigQuery::test_query_results_to_dataframe\r\n2.81s call     tests/system.py::TestBigQuery::test_list_datasets\r\n2.81s call     tests/system.py::TestBigQuery::test_job_cancel\r\n2.69s call     tests/system.py::TestBigQuery::test_update_table\r\n2.69s call     tests/system.py::TestBigQuery::test_insert_rows_from_dataframe\r\n2.50s call     tests/system.py::TestBigQuery::test_create_routine\r\n2.46s call     tests/system.py::TestBigQuery::test_query_w_page_size\r\n2.22s call     tests/system.py::TestBigQuery::test_query_results_to_dataframe_w_bqstorage_v1beta1\r\n2.19s call     tests/system.py::TestBigQuery::test_query_results_to_dataframe_w_bqstorage\r\n2.18s call     tests/system.py::TestBigQuery::test_large_query_w_public_data\r\n2.10s call     tests/system.py::TestBigQuery::test_create_table_rows_fetch_nested_schema\r\n2.00s call     tests/system.py::TestBigQuery::test_insert_rows_then_dump_table\r\n1.98s call     tests/system.py::TestBigQuery::test_dbapi_fetch_w_bqstorage_client_large_result_set\r\n1.90s call     tests/system.py::TestBigQuery::test_update_table_schema\r\n1.89s call     tests/system.py::TestBigQuery::test_create_table\r\n1.84s call     tests/system.py::TestBigQuery::test_dbapi_fetch_w_bqstorage_client_v1beta1_large_result_set\r\n1.60s call     tests/system.py::TestBigQuery::test_insert_rows_nested_nested_dictionary\r\n1.60s call     tests/system.py::TestBigQuery::test_create_table_w_time_partitioning_w_clustering_fields\r\n1.59s call     tests/system.py::TestBigQuery::test_insert_rows_nested_nested\r\n1.57s call     tests/system.py::TestBigQuery::test_query_statistics\r\n1.55s call     tests/system.py::TestBigQuery::test_update_dataset\r\n1.46s call     tests/system.py::TestBigQuery::test_query_w_timeout\r\n1.44s call     tests/system.py::TestBigQuery::test_list_rows_empty_table\r\n1.30s call     tests/system.py::TestBigQuery::test_get_dataset\r\n1.18s call     tests/system.py::TestBigQuery::test_delete_dataset_delete_contents_true\r\n1.11s call     tests/system.py::TestBigQuery::test_create_table_with_policy\r\n1.07s call     tests/system.py::TestBigQuery::test_delete_dataset_delete_contents_false\r\n1.00s call     tests/system.py::TestBigQuery::test_create_dataset\r\n0.99s call     tests/system.py::TestBigQuery::test_delete_dataset_with_string\r\n0.93s call     tests/system.py::TestBigQuery::test_query_w_wrong_config\r\n0.88s call     tests/system.py::TestBigQuery::test_list_datasets_w_project\r\n0.77s call     tests/system.py::TestBigQuery::test_close_releases_open_sockets\r\n0.75s call     tests/system.py::TestBigQuery::test_query_w_start_index\r\n0.75s call     tests/system.py::TestBigQuery::test_query_future\r\n0.73s call     tests/system.py::TestBigQuery::test_query_iter\r\n0.47s call     tests/system.py::TestBigQuery::test_list_partitions\r\n0.39s call     tests/system.py::TestBigQuery::test_dbapi_dry_run_query\r\n0.27s call     tests/system.py::TestBigQuery::test_get_failed_job\r\n0.26s call     tests/system.py::TestBigQuery::test_query_w_failed_query\r\n0.17s call     tests/system.py::TestBigQuery::test_get_table_w_public_dataset\r\n0.17s call     tests/system.py::TestBigQuery::test_list_rows_max_results_w_bqstorage\r\n0.16s call     tests/system.py::TestBigQuery::test_get_service_account_email\r\n0.06s setup    tests/system.py::test_bigquery_magic\r\n0.01s setup    tests/system.py::TestBigQuery::test_close_releases_open_sockets\r\n```\r\n\r\nHere's the slowest unit-3.8 timings (way more tests than included here):\r\n\r\n```\r\n1.73s call     tests/unit/test_job.py::TestQueryJob::test_result_invokes_begins\r\n1.45s call     tests/unit/test_client.py::TestClient::test_get_dataset\r\n1.00s call     tests/unit/test_client.py::TestClient::test_create_job_query_config_w_rateLimitExceeded_error\r\n1.00s call     tests/unit/test_client.py::TestClient::test_get_service_account_email_w_custom_retry\r\n0.44s call     tests/unit/test_table.py::TestRowIterator::test_to_dataframe_w_bqstorage_updates_progress_bar\r\n0.37s call     tests/unit/test_job.py::Test_AsyncJob::test_cancel_w_custom_retry\r\n0.31s call     tests/unit/test_table.py::TestRowIterator::test_to_dataframe_w_bqstorage_empty_streams\r\n0.28s call     tests/unit/test_table.py::TestRowIterator::test_to_dataframe_error_if_pandas_is_none\r\n0.27s call     tests/unit/test_dbapi_connection.py::TestConnection::test_does_not_keep_cursor_instances_alive\r\n0.26s call     tests/unit/test_table.py::TestRowIterator::test_to_dataframe_progress_bar_wo_pyarrow\r\n0.26s call     tests/unit/test_table.py::TestRowIterator::test_to_dataframe_progress_bar\r\n0.25s call     tests/unit/test_table.py::TestRowIterator::test_to_arrow_progress_bar\r\n0.25s call     tests/unit/test_dbapi_cursor.py::TestCursor::test_fetchmany_w_row\r\n0.25s call     tests/unit/test_dbapi_cursor.py::TestCursor::test_execute_w_query_dry_run\r\n0.25s call     tests/unit/test_table.py::TestRowIterator::test_to_dataframe_tqdm_error\r\n0.25s call     tests/unit/test_job.py::TestQueryJob::test_result_w_timeout\r\n0.24s call     tests/unit/test_table.py::TestRowIterator::test_to_dataframe_w_bqstorage_snapshot\r\n0.22s call     tests/unit/test_table.py::TestRowIterator::test_to_dataframe_tabledata_list_w_multiple_pages_return_unique_index\r\n0.21s call     tests/unit/test_table.py::TestRowIterator::test_to_arrow_max_results_w_create_bqstorage_warning\r\n0.21s call     tests/unit/test_magics.py::test_bigquery_magic_w_table_id_and_destination_var\r\n0.21s call     tests/unit/test_client.py::TestClient::test__call_api_applying_custom_retry_on_timeout\r\n0.20s call     tests/unit/test_dbapi_connection.py::TestConnection::test_connect_w_client\r\n```","author":{"url":"https://github.com/shollyman","@type":"Person","name":"shollyman"},"datePublished":"2020-07-24T22:16:24.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":9},"url":"https://github.com/191/python-bigquery/issues/191"}