René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"test failures on presubmit session","articleBody":"Many tests are failing in the presubmit session.\r\n\r\n```\r\nnox \u003e python -m pip freeze\r\naiohttp==3.9.1\r\naiosignal==1.3.1\r\nappdirs==1.4.4\r\nargcomplete==3.2.1\r\nasttokens==2.4.1\r\nasyncmock==0.4.2\r\natpublic==4.0\r\nattrs==23.2.0\r\nbidict==0.22.1\r\n# Editable Git install with no remote (bigframes==0.19.2)\r\n-e /tmpfs/src/github/python-bigquery-dataframes\r\nblack==23.12.1\r\nbleach==6.1.0\r\ncachetools==5.3.2\r\ncertifi==2023.11.17\r\ncffi==1.16.0\r\ncfgv==3.4.0\r\ncharset-normalizer==3.3.2\r\nclick==8.0.4\r\nclick-plugins==1.1.1\r\ncligj==0.7.2\r\ncloudpickle==3.0.0\r\ncolorlog==6.8.0\r\ncomm==0.2.1\r\ncoverage==7.4.0\r\ncryptography==41.0.7\r\ndb-dtypes==1.2.0\r\ndecorator==5.1.1\r\ndistlib==0.3.8\r\ndocstring-inheritance==2.1.2\r\ndocutils==0.20.1\r\net-xmlfile==1.1.0\r\nexceptiongroup==1.2.0\r\nexecnet==2.0.2\r\nexecuting==2.0.1\r\nfilelock==3.13.1\r\nfiona==1.9.5\r\nfrozenlist==1.4.1\r\nfsspec==2023.12.2\r\ngcp-docuploader==0.6.5\r\ngcp-releasetool==1.17.0\r\ngcsfs==2023.12.2.post1\r\ngeopandas==0.14.2\r\ngoogle-api-core==1.34.0\r\ngoogle-auth==2.26.2\r\ngoogle-auth-oauthlib==1.2.0\r\ngoogle-cloud-bigquery @ git+[https://github.com/googleapis/python-bigquery.git@1f96439b3dbd27f11be5e2af84f290ec6094d0a4](https://www.google.com/url?q=https://github.com/googleapis/python-bigquery.git@1f96439b3dbd27f11be5e2af84f290ec6094d0a4\u0026sa=D)\r\ngoogle-cloud-bigquery-connection==1.14.0\r\n-e git+[https://github.com/googleapis/python-bigquery-storage.git@fb0d6f07d4b7582cf01ec98fee3081ec34995603#egg=google_cloud_bigquery_storage](https://www.google.com/url?q=https://github.com/googleapis/python-bigquery-storage.git@fb0d6f07d4b7582cf01ec98fee3081ec34995603%23egg%3Dgoogle_cloud_bigquery_storage\u0026sa=D)\r\ngoogle-cloud-core==2.4.1\r\ngoogle-cloud-functions==1.14.0\r\ngoogle-cloud-iam==2.13.0\r\ngoogle-cloud-resource-manager==1.11.0\r\ngoogle-cloud-storage==2.14.0\r\ngoogle-cloud-testutils==1.4.0\r\ngoogle-crc32c==1.5.0\r\ngoogle-resumable-media==2.7.0\r\ngoogleapis-common-protos==1.62.0\r\ngreenlet==3.0.3\r\ngrpc-google-iam-v1==0.13.0\r\ngrpcio==1.60.0\r\ngrpcio-status==1.48.2\r\nhumanize==4.9.0\r\nibis-framework==7.1.0\r\nidentify==2.5.33\r\nidna==3.6\r\nimportlib-metadata==7.0.1\r\nimportlib-resources==6.1.1\r\niniconfig==2.0.0\r\nipython==8.20.0\r\nipywidgets==8.1.1\r\njaraco.classes==3.3.0\r\njedi==0.19.1\r\njeepney==0.8.0\r\nJinja2==3.1.3\r\njoblib==1.3.2\r\njupyterlab-widgets==3.0.9\r\nkeyring==24.3.0\r\nmarkdown-it-py==3.0.0\r\nMarkupSafe==2.1.4\r\nmatplotlib-inline==0.1.6\r\nmdurl==0.1.2\r\nmock==5.1.0\r\nmore-itertools==10.2.0\r\nmultidict==6.0.4\r\nmultipledispatch==1.0.0\r\nmypy-extensions==1.0.0\r\nnh3==0.2.15\r\nnodeenv==1.8.0\r\nnox==2023.4.22\r\nnumpy==1.26.3\r\noauthlib==3.2.2\r\nopenpyxl==3.1.2\r\npackaging==23.2\r\npandas==2.2.0\r\npandas-gbq==0.20.0\r\nparso==0.8.3\r\nparsy==2.1\r\npathspec==0.12.1\r\npexpect==4.9.0\r\npins==0.8.4\r\npkginfo==1.9.6\r\nplatformdirs==4.1.0\r\npluggy==1.3.0\r\npooch==1.8.0\r\npre-commit==3.6.0\r\nprompt-toolkit==3.0.43\r\nproto-plus==1.23.0\r\nprotobuf==3.20.3\r\nptyprocess==0.7.0\r\npure-eval==0.2.2\r\npyarrow==14.0.2\r\npyarrow-hotfix==0.6\r\npyasn1==0.5.1\r\npyasn1-modules==0.3.0\r\npycparser==2.21\r\npydata-google-auth==1.8.2\r\nPygments==2.17.2\r\nPyJWT==2.8.0\r\npyperclip==1.8.2\r\npyproj==3.6.1\r\npytest==7.4.4\r\npytest-asyncio==0.23.3\r\npytest-cov==4.1.0\r\npytest-mock==3.12.0\r\npytest-retry==1.6.1\r\npytest-timeout==2.2.0\r\npytest-xdist==3.5.0\r\npython-dateutil==2.8.2\r\npytz==2023.3.post1\r\nPyYAML==6.0.1\r\nreadme-renderer==42.0\r\nrequests==2.31.0\r\nrequests-oauthlib==1.3.1\r\nrequests-toolbelt==1.0.0\r\nrfc3986==2.0.0\r\nrich==13.7.0\r\nrsa==4.9\r\nscikit-learn==1.4.0\r\nscipy==1.12.0\r\nSecretStorage==3.3.3\r\nshapely==2.0.2\r\nsix==1.16.0\r\nSQLAlchemy==2.0.25\r\nsqlglot==19.9.0\r\nstack-data==0.6.3\r\ntabulate==0.9.0\r\nthreadpoolctl==3.2.0\r\ntomli==2.0.1\r\ntoolz==0.12.0\r\ntqdm==4.66.1\r\ntraitlets==5.14.1\r\ntwine==4.0.2\r\ntyping_extensions==4.9.0\r\ntzdata==2023.4\r\nurllib3==2.1.0\r\nvirtualenv==20.25.0\r\nwcwidth==0.2.13\r\nwebencodings==0.5.1\r\nwidgetsnbextension==4.0.9\r\nxarray==2024.1.0\r\nxxhash==3.4.1\r\nyarl==1.9.4\r\nzipp==3.17.0\r\n\r\n...\r\n\r\ndef test_read_gbq_after_session_start_must_comply_with_default_location(\r\n        scalars_pandas_df_index,\r\n        test_data_tables,\r\n        test_data_tables_tokyo,\r\n        dataset_id_permanent_tokyo,\r\n        read_method,\r\n        query_prefix,\r\n    ):\r\n        # Form query as a table name or a SQL depending on the test scenario\r\n        query_tokyo = test_data_tables_tokyo[\"scalars\"]\r\n        query = test_data_tables[\"scalars\"]\r\n        if query_prefix:\r\n            query_tokyo = f\"{query_prefix} {query_tokyo}\"\r\n            query = f\"{query_prefix} {query}\"\r\n    \r\n        # Initially there is no location set in the bigquery options\r\n        assert not bpd.options.bigquery.location\r\n    \r\n        # Starting user journey with anything other than read_gbq*, such as\r\n        # read_pandas would bind the session to default location US\r\n\u003e       df = bpd.read_pandas(scalars_pandas_df_index)\r\n\r\n[tests/system/small/test_pandas_options.py:143](https://cs.corp.google.com/piper///depot/google3/tests/system/small/test_pandas_options.py?l=143): \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n[bigframes/pandas/__init__.py:570](https://cs.corp.google.com/piper///depot/google3/bigframes/pandas/__init__.py?l=570): in read_pandas\r\n    return global_session.with_default_session(\r\n[bigframes/core/global_session.py:65](https://cs.corp.google.com/piper///depot/google3/bigframes/core/global_session.py?l=65): in with_default_session\r\n    return func(get_global_session(), *args, **kwargs)\r\n[bigframes/session/__init__.py:849](https://cs.corp.google.com/piper///depot/google3/bigframes/session/__init__.py?l=849): in read_pandas\r\n    return self._read_pandas(pandas_dataframe, \"read_pandas\")\r\n[bigframes/session/__init__.py:889](https://cs.corp.google.com/piper///depot/google3/bigframes/session/__init__.py?l=889): in _read_pandas\r\n    load_job = self.bqclient.load_table_from_dataframe(\r\n[.nox/system_prerelease/lib/python3.11/site-packages/google/cloud/bigquery/client.py:2713](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/google/cloud/bigquery/client.py?l=2713): in load_table_from_dataframe\r\n    dataframe.to_parquet(\r\n[.nox/system_prerelease/lib/python3.11/site-packages/pandas/util/_decorators.py:333](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pandas/util/_decorators.py?l=333): in wrapper\r\n    return func(*args, **kwargs)\r\n[.nox/system_prerelease/lib/python3.11/site-packages/pandas/core/frame.py:3101](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pandas/core/frame.py?l=3101): in to_parquet\r\n    return to_parquet(\r\n[.nox/system_prerelease/lib/python3.11/site-packages/pandas/io/parquet.py:480](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pandas/io/parquet.py?l=480): in to_parquet\r\n    impl.write(\r\n[.nox/system_prerelease/lib/python3.11/site-packages/pandas/io/parquet.py:190](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pandas/io/parquet.py?l=190): in write\r\n    table = self.api.Table.from_pandas(df, **from_pandas_kwargs)\r\n[pyarrow/table.pxi:3869](https://cs.corp.google.com/piper///depot/google3/pyarrow/table.pxi?l=3869): in pyarrow.lib.Table.from_pandas\r\n    ???\r\n[.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py:613](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py?l=613): in dataframe_to_arrays\r\n    arrays = [convert_column(c, f)\r\n[.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py:613](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py?l=613): in \u003clistcomp\u003e\r\n    arrays = [convert_column(c, f)\r\n[.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py:600](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py?l=600): in convert_column\r\n    raise e\r\n[.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py:594](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py?l=594): in convert_column\r\n    result = pa.array(col, type=type_, from_pandas=True, safe=safe)\r\n[pyarrow/array.pxi:340](https://cs.corp.google.com/piper///depot/google3/pyarrow/array.pxi?l=340): in pyarrow.lib.array\r\n    ???\r\n[pyarrow/array.pxi:82](https://cs.corp.google.com/piper///depot/google3/pyarrow/array.pxi?l=82): in pyarrow.lib._ndarray_to_array\r\n    ???\r\n[pyarrow/array.pxi:70](https://cs.corp.google.com/piper///depot/google3/pyarrow/array.pxi?l=70): in pyarrow.lib._ndarray_to_type\r\n    ???\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n\r\n\u003e   ???\r\nE   pyarrow.lib.ArrowTypeError: ('Did not pass numpy.dtype object', 'Conversion failed for column geography_col with type geometry')\r\n\r\n[pyarrow/error.pxi:91](https://cs.corp.google.com/piper///depot/google3/pyarrow/error.pxi?l=91): ArrowTypeError\r\n\r\n...\r\n\r\n=========================== short test summary info ============================\r\nFAILED tests/system/small/test_dataframe.py::test_dataframe_bool_aggregates[all_axis0]\r\nFAILED tests/system/small/test_dataframe.py::test_dataframe_bool_aggregates[any_axis0]\r\nFAILED tests/system/small/test_dataframe.py::test_df_pivot[values2-int64_too-columns2]\r\nFAILED tests/system/small/test_groupby.py::test_dataframe_groupby_analytic[cumprod]\r\nFAILED tests/system/small/test_pandas_options.py::test_read_gbq_after_session_start_must_comply_with_default_location[read_gbq-on-table-name]\r\nFAILED tests/system/small/test_pandas_options.py::test_read_gbq_after_session_start_must_comply_with_default_location[read_gbq-on-sql]\r\nFAILED tests/system/small/test_pandas_options.py::test_read_gbq_after_session_start_must_comply_with_default_location[read_gbq_table-on-table-name]\r\nFAILED tests/system/small/test_pandas_options.py::test_read_gbq_after_session_start_must_comply_with_default_location[read_gbq_query-on-sql]\r\nFAILED tests/system/small/test_series.py::test_groupby_window_ops[cumprod] - ...\r\nFAILED tests/system/small/test_dataframe.py::test_to_pandas_downsampling_option_override\r\nFAILED tests/system/small/test_series.py::test_series_add_prefix - AssertionE...\r\nFAILED tests/system/small/test_series.py::test_series_add_suffix - AssertionE...\r\nFAILED tests/system/small/test_session.py::test_read_pandas - pyarrow.lib.Arr...\r\nFAILED tests/system/small/test_session.py::test_read_pandas_multi_index - pya...\r\nFAILED tests/system/small/test_session.py::test_read_pandas_rowid_exists_adds_suffix\r\nFAILED tests/system/small/test_series.py::test_string_astype_int - AssertionE...\r\nFAILED tests/system/small/test_session.py::test_read_pandas_tokyo - pyarrow.l...\r\nFAILED tests/system/small/test_session.py::test_read_csv_local_default_engine[default_sep]\r\nFAILED tests/system/small/test_session.py::test_read_csv_gcs_default_engine\r\nFAILED tests/system/small/test_session.py::test_read_csv_local_default_engine[custom_sep]\r\n20 failed, 1599 passed, 21 skipped, 7 xfailed, 804 warnings in 468.88s (0:07:48)\r\nnox \u003e Command py.test --quiet -n=20 --timeout=600 --junitxml=small_prerelease_3.11_sponge_log.xml --cov=bigframes --cov=tests/system/small --cov-append --cov-config=.coveragerc --cov-report=term-missing --cov-fail-under=0 tests/system/small failed with exit code 1\r\nnox \u003e Session system_prerelease failed.\r\nnox \u003e Ran multiple sessions:\r\nnox \u003e * unit_prerelease: success\r\nnox \u003e * system_prerelease: failed\r\n```\r\n\r\nhttps://fusion2.corp.google.com/invocations/f5604d0d-929d-456f-ab59-b7e292081af2/targets/bigframes%2Fpresubmit%2Fe2e/log","author":{"url":"https://github.com/tswast","@type":"Person","name":"tswast"},"datePublished":"2024-01-22T19:06:18.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/337/python-bigquery-dataframes/issues/337"}