Title: BigQuery to_dataframe() ArrowNotImplementedError · Issue #63 · googleapis/python-bigquery · GitHub
Open Graph Title: BigQuery to_dataframe() ArrowNotImplementedError · Issue #63 · googleapis/python-bigquery
X Title: BigQuery to_dataframe() ArrowNotImplementedError · Issue #63 · googleapis/python-bigquery
Description: I'm working with the Google Vision API and trying to do some analysis in pandas. When I try to get the output into pandas there is an issue and I get the NotImplementedError Below. Environment details Specify the API at the beginning of ...
Open Graph Description: I'm working with the Google Vision API and trying to do some analysis in pandas. When I try to get the output into pandas there is an issue and I get the NotImplementedError Below. Environment deta...
X Description: I'm working with the Google Vision API and trying to do some analysis in pandas. When I try to get the output into pandas there is an issue and I get the NotImplementedError Below. Environment ...
Opengraph URL: https://github.com/googleapis/python-bigquery/issues/63
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"BigQuery to_dataframe() ArrowNotImplementedError","articleBody":"I'm working with the Google Vision API and trying to do some analysis in pandas. When I try to get the output into pandas there is an issue and I get the NotImplementedError Below.\r\n\r\n#### Environment details\r\n\r\n1. Specify the API at the beginning of the title (for example, \"BigQuery: ...\")\r\n General, Core, and Other are also allowed as types\r\n2. OS type and version: **GCP AI Notebook** Python framework\r\n3. Python version and virtual environment information: 3.7.6\r\n4. google-cloud-bigquery version: 1.24.0\r\n\r\n#### Steps to reproduce\r\n\r\n 1. Query Vision API output results in nested structs. Our output has **RECORD** datatypes of mode **REPEATED**\r\n\r\n#### Code example\r\n\r\n```python\r\n\r\ndef get_data(_project_id, _table_id):\r\n \r\n client = bq.Client(project=_project_id)\r\n try:\r\n df = client.query(sql_)\r\n except OperationalError as oe:\r\n print(oe.msg)\r\n print(\"Query Complete. Converting to Dataframe\")\r\n\r\n df = df.to_dataframe(progress_bar_type='tqdm') # Converts to dataframe\r\n return df\r\n\r\nsql_ = (\r\n '''\r\n SELECT\r\n *\r\n FROM `{}`\r\n '''.format(_table_id)\r\n )\r\n\r\ndf = get_data(project_id, table_id)\r\n```\r\n\r\n#### Stack trace\r\n```\r\nArrowNotImplementedError Traceback (most recent call last)\r\n\u003cipython-input-17-15263a59b273> in \u003cmodule>\r\n----> 1 df = get_data(project_id, table_id)\r\n\r\n\u003cipython-input-16-83704d37548f> in get_data(_project_id, _table_id, verbose)\r\n 18 print(\"Query Complete. Converting to Dataframe\")\r\n 19 \r\n---> 20 df = df.to_dataframe(progress_bar_type='tqdm') # Converts to dataframe\r\n 21 return df\r\n\r\n/opt/conda/lib/python3.7/site-packages/google/cloud/bigquery/job.py in to_dataframe(self, bqstorage_client, dtypes, progress_bar_type, create_bqstorage_client)\r\n 3372 dtypes=dtypes,\r\n 3373 progress_bar_type=progress_bar_type,\r\n-> 3374 create_bqstorage_client=create_bqstorage_client,\r\n 3375 )\r\n 3376 \r\n\r\n/opt/conda/lib/python3.7/site-packages/google/cloud/bigquery/table.py in to_dataframe(self, bqstorage_client, dtypes, progress_bar_type, create_bqstorage_client)\r\n 1729 create_bqstorage_client=create_bqstorage_client,\r\n 1730 )\r\n-> 1731 df = record_batch.to_pandas()\r\n 1732 for column in dtypes:\r\n 1733 df[column] = pandas.Series(df[column], dtype=dtypes[column])\r\n\r\n/opt/conda/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib._PandasConvertible.to_pandas()\r\n\r\n/opt/conda/lib/python3.7/site-packages/pyarrow/table.pxi in pyarrow.lib.Table._to_pandas()\r\n\r\n/opt/conda/lib/python3.7/site-packages/pyarrow/pandas_compat.py in table_to_blockmanager(options, table, categories, ignore_metadata, types_mapper)\r\n 764 _check_data_column_metadata_consistency(all_columns)\r\n 765 columns = _deserialize_column_index(table, all_columns, column_indexes)\r\n--> 766 blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)\r\n 767 \r\n 768 axes = [columns, index]\r\n\r\n/opt/conda/lib/python3.7/site-packages/pyarrow/pandas_compat.py in _table_to_blocks(options, block_table, categories, extension_columns)\r\n 1099 columns = block_table.column_names\r\n 1100 result = pa.lib.table_to_blocks(options, block_table, categories,\r\n-> 1101 list(extension_columns.keys()))\r\n 1102 return [_reconstruct_block(item, columns, extension_columns)\r\n 1103 for item in result]\r\n\r\n/opt/conda/lib/python3.7/site-packages/pyarrow/table.pxi in pyarrow.lib.table_to_blocks()\r\n\r\n/opt/conda/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()\r\n\r\nArrowNotImplementedError: Not implemented type for Arrow list to pandas: struct\u003cscore: double, description: string>\r\n```","author":{"url":"https://github.com/jvschoen","@type":"Person","name":"jvschoen"},"datePublished":"2020-03-17T20:06:59Z","url":"https://github.com/googleapis/python-bigquery/issues/63"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:43642e4d-e75b-0d22-48d7-cf466d28206f |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 8D8E:49756:9D1DD:D1F11:6AFE1F5F |
| html-safe-nonce | 089598c168aa0b430af9b4b5e1de2c210a1edb5385b87b96c34c3ece9a06f019 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4RDhFOjQ5NzU2OjlEMUREOkQxRjExOjZBRkUxRjVGIiwidmlzaXRvcl9pZCI6IjU3ODgwODE5MjYyODM1OTk3MTEiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 3061fea1f1cf4624aabfa886072f66c6ebe51497b9377f40679bf1472c84ec68 |
| hovercard-subject-tag | issue:583637057 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/googleapis/python-bigquery/63/issue_layout |
| twitter:image | https://opengraph.githubassets.com/0793417beb8238ff586aec59b9a375e8d8e6b1c9462556a8817dadce77b43308/googleapis/python-bigquery/issues/63 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/0793417beb8238ff586aec59b9a375e8d8e6b1c9462556a8817dadce77b43308/googleapis/python-bigquery/issues/63 |
| og:image:alt | I'm working with the Google Vision API and trying to do some analysis in pandas. When I try to get the output into pandas there is an issue and I get the NotImplementedError Below. Environment deta... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | jvschoen |
| hostname | github.com |
| expected-hostname | github.com |
| None | d0525ab1464526bf4532a5551aa348cf100dff75ef808a96fc67a08b1d946aa9 |
| turbo-cache-control | no-preview |
| go-import | github.com/googleapis/python-bigquery git https://github.com/googleapis/python-bigquery.git |
| octolytics-dimension-user_id | 16785467 |
| octolytics-dimension-user_login | googleapis |
| octolytics-dimension-repository_id | 226992475 |
| octolytics-dimension-repository_nwo | googleapis/python-bigquery |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 226992475 |
| octolytics-dimension-repository_network_root_nwo | googleapis/python-bigquery |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | fef3993a71fa20f7be8e167132fbf9619bc66e12 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width