Title: BigQuery: Use BigQuery schema (from LoadJobConfig) if available when converting to Parquet in load_table_from_dataframe · Issue #7370 · googleapis/google-cloud-python · GitHub
Open Graph Title: BigQuery: Use BigQuery schema (from LoadJobConfig) if available when converting to Parquet in load_table_from_dataframe · Issue #7370 · googleapis/google-cloud-python
X Title: BigQuery: Use BigQuery schema (from LoadJobConfig) if available when converting to Parquet in load_table_from_dataframe · Issue #7370 · googleapis/google-cloud-python
Description: Environment details OS = Windows 7 Python = 3.7.2 google-cloud-bigquery 1.8.1 Steps to reproduce Try to upload a DataFrame with only pd.np.nan in a column into a table that normally uses STRING Code example import pandas as pd from googl...
Open Graph Description: Environment details OS = Windows 7 Python = 3.7.2 google-cloud-bigquery 1.8.1 Steps to reproduce Try to upload a DataFrame with only pd.np.nan in a column into a table that normally uses STRING Cod...
X Description: Environment details OS = Windows 7 Python = 3.7.2 google-cloud-bigquery 1.8.1 Steps to reproduce Try to upload a DataFrame with only pd.np.nan in a column into a table that normally uses STRING Cod...
Opengraph URL: https://github.com/googleapis/google-cloud-python/issues/7370
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"BigQuery: Use BigQuery schema (from LoadJobConfig) if available when converting to Parquet in load_table_from_dataframe","articleBody":"#### Environment details\r\n\r\nOS = Windows 7\r\nPython = 3.7.2\r\ngoogle-cloud-bigquery 1.8.1\r\n\r\n#### Steps to reproduce\r\n\r\n1. Try to upload a DataFrame with only pd.np.nan in a column into a table that normally uses STRING\r\n\r\n#### Code example\r\n\r\n```python\r\nimport pandas as pd\r\nfrom google.cloud import bigquery\r\nfrom oauth2client.client import GoogleCredentials\r\nfrom oauth2client.service_account import ServiceAccountCredentials\r\n\r\nJSON = # insert json access here\r\ndataset_name = # insert dataset name here\r\nclient = bigquery.Client.from_service_account_json(JSON)\r\n\r\ntest1 = pd.DataFrame({'column_1':[pd.np.nan, pd.np.nan, 'b'], \r\n\t\t\t\t\t 'column_2':['1', '1', '1']})\r\n\r\ntest2 = test1.copy()\r\ntest2 = test2[:2]\r\n\r\ndef upload(data, name):\r\n\tdataset = client.dataset(dataset_name)\r\n\ttable = dataset.table(name)\r\n\treturn client.load_table_from_dataframe(data, table).result()\r\n\r\nprint('uploading 1: nan with string in column')\r\nresult = upload(test1, 'test1')\r\n\r\nprint('uploading 2: nan without string in column (fails)')\r\nresult2 = upload(test2, 'test1')\r\n```\r\n\r\n#### Error\r\n```\r\ngoogle.api_core.exceptions.BadRequest: 400 Provided Schema does not match Table. Field column_1 has changed type from STRING to INTEGER\r\n```\r\n\r\nSo I've been in contact with enterprise support, but they were unable to help me. Apparently there's a bug in google.bigquery.Client in the line dataframe.to_parquet(buffer) that causes columns with all NaN values to be interpreted as FLOAT or INTEGER instead of STRING. This prevents the dataframe from being uploaded and there is no other way to introduce NULLs into the table in BigQuery. The issue does not occur in pandas-gbq. Support (ticket 18371705) advised me to use that as a workaround instead until this is fixed and report the issue here. If you have any questions or need more information, feel free to ask.","author":{"url":"https://github.com/Unprocessable","@type":"Person","name":"Unprocessable"},"datePublished":"2019-02-15T13:31:23Z","url":"https://github.com/googleapis/google-cloud-python/issues/7370"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:2af2a256-4a3c-e7d9-0170-6884bfdf078d |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | B53E:A9DCA:926FCB:CEA297:6AFE1CCA |
| html-safe-nonce | 062653e57a2d43625ebd4133773c987e630818c60150917f5493a0bf9c5e21b3 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCNTNFOkE5RENBOjkyNkZDQjpDRUEyOTc6NkFGRTFDQ0EiLCJ2aXNpdG9yX2lkIjoiNjMxMzc2MjYzNjAwMTMyMDEzOCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 08f63429e05fcf0b425772b33244eaed673250e32556dadbcf2e581e21e556b8 |
| hovercard-subject-tag | issue:410770771 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/googleapis/google-cloud-python/7370/issue_layout |
| twitter:image | https://opengraph.githubassets.com/272d04474bd29b086cea3a0fbc69facbb4471b6eb09bdf467f3c1a10cf543d7e/googleapis/google-cloud-python/issues/7370 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/272d04474bd29b086cea3a0fbc69facbb4471b6eb09bdf467f3c1a10cf543d7e/googleapis/google-cloud-python/issues/7370 |
| og:image:alt | Environment details OS = Windows 7 Python = 3.7.2 google-cloud-bigquery 1.8.1 Steps to reproduce Try to upload a DataFrame with only pd.np.nan in a column into a table that normally uses STRING Cod... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | Unprocessable |
| hostname | github.com |
| expected-hostname | github.com |
| None | d0525ab1464526bf4532a5551aa348cf100dff75ef808a96fc67a08b1d946aa9 |
| turbo-cache-control | no-preview |
| go-import | github.com/googleapis/google-cloud-python git https://github.com/googleapis/google-cloud-python.git |
| octolytics-dimension-user_id | 16785467 |
| octolytics-dimension-user_login | googleapis |
| octolytics-dimension-repository_id | 16316451 |
| octolytics-dimension-repository_nwo | googleapis/google-cloud-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 16316451 |
| octolytics-dimension-repository_network_root_nwo | googleapis/google-cloud-python |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | fef3993a71fa20f7be8e167132fbf9619bc66e12 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width