René's URL Explorer Experiment


Title: BigQuery: Use BigQuery schema (from LoadJobConfig) if available when converting to Parquet in load_table_from_dataframe · Issue #7370 · googleapis/google-cloud-python · GitHub

Open Graph Title: BigQuery: Use BigQuery schema (from LoadJobConfig) if available when converting to Parquet in load_table_from_dataframe · Issue #7370 · googleapis/google-cloud-python

X Title: BigQuery: Use BigQuery schema (from LoadJobConfig) if available when converting to Parquet in load_table_from_dataframe · Issue #7370 · googleapis/google-cloud-python

Description: Environment details OS = Windows 7 Python = 3.7.2 google-cloud-bigquery 1.8.1 Steps to reproduce Try to upload a DataFrame with only pd.np.nan in a column into a table that normally uses STRING Code example import pandas as pd from googl...

Open Graph Description: Environment details OS = Windows 7 Python = 3.7.2 google-cloud-bigquery 1.8.1 Steps to reproduce Try to upload a DataFrame with only pd.np.nan in a column into a table that normally uses STRING Cod...

X Description: Environment details OS = Windows 7 Python = 3.7.2 google-cloud-bigquery 1.8.1 Steps to reproduce Try to upload a DataFrame with only pd.np.nan in a column into a table that normally uses STRING Cod...

Opengraph URL: https://github.com/googleapis/google-cloud-python/issues/7370

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"BigQuery: Use BigQuery schema (from LoadJobConfig) if available when converting to Parquet in load_table_from_dataframe","articleBody":"#### Environment details\r\n\r\nOS = Windows 7\r\nPython = 3.7.2\r\ngoogle-cloud-bigquery 1.8.1\r\n\r\n#### Steps to reproduce\r\n\r\n1. Try to upload a DataFrame with only pd.np.nan in a column into a table that normally uses STRING\r\n\r\n#### Code example\r\n\r\n```python\r\nimport pandas as pd\r\nfrom google.cloud import bigquery\r\nfrom oauth2client.client import GoogleCredentials\r\nfrom oauth2client.service_account import ServiceAccountCredentials\r\n\r\nJSON = # insert json access here\r\ndataset_name = # insert dataset name here\r\nclient = bigquery.Client.from_service_account_json(JSON)\r\n\r\ntest1 = pd.DataFrame({'column_1':[pd.np.nan, pd.np.nan, 'b'], \r\n\t\t\t\t\t  'column_2':['1', '1', '1']})\r\n\r\ntest2 = test1.copy()\r\ntest2 = test2[:2]\r\n\r\ndef upload(data, name):\r\n\tdataset = client.dataset(dataset_name)\r\n\ttable = dataset.table(name)\r\n\treturn client.load_table_from_dataframe(data, table).result()\r\n\r\nprint('uploading 1: nan with string in column')\r\nresult = upload(test1, 'test1')\r\n\r\nprint('uploading 2: nan without string in column (fails)')\r\nresult2 = upload(test2, 'test1')\r\n```\r\n\r\n#### Error\r\n```\r\ngoogle.api_core.exceptions.BadRequest: 400 Provided Schema does not match Table. Field column_1 has changed type from STRING to INTEGER\r\n```\r\n\r\nSo I've been in contact with enterprise support, but they were unable to help me. Apparently there's a bug in google.bigquery.Client in the line dataframe.to_parquet(buffer) that causes columns with all NaN values to be interpreted as FLOAT or INTEGER instead of STRING. This prevents the dataframe from being uploaded and there is no other way to introduce NULLs into the table in BigQuery. The issue does not occur in pandas-gbq. Support (ticket 18371705) advised me to use that as a workaround instead until this is fixed and report the issue here. If you have any questions or need more information, feel free to ask.","author":{"url":"https://github.com/Unprocessable","@type":"Person","name":"Unprocessable"},"datePublished":"2019-02-15T13:31:23Z","url":"https://github.com/googleapis/google-cloud-python/issues/7370"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:2af2a256-4a3c-e7d9-0170-6884bfdf078d
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idB53E:A9DCA:926FCB:CEA297:6AFE1CCA
html-safe-nonce062653e57a2d43625ebd4133773c987e630818c60150917f5493a0bf9c5e21b3
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCNTNFOkE5RENBOjkyNkZDQjpDRUEyOTc6NkFGRTFDQ0EiLCJ2aXNpdG9yX2lkIjoiNjMxMzc2MjYzNjAwMTMyMDEzOCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac08f63429e05fcf0b425772b33244eaed673250e32556dadbcf2e581e21e556b8
hovercard-subject-tagissue:410770771
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/googleapis/google-cloud-python/7370/issue_layout
twitter:imagehttps://opengraph.githubassets.com/272d04474bd29b086cea3a0fbc69facbb4471b6eb09bdf467f3c1a10cf543d7e/googleapis/google-cloud-python/issues/7370
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/272d04474bd29b086cea3a0fbc69facbb4471b6eb09bdf467f3c1a10cf543d7e/googleapis/google-cloud-python/issues/7370
og:image:altEnvironment details OS = Windows 7 Python = 3.7.2 google-cloud-bigquery 1.8.1 Steps to reproduce Try to upload a DataFrame with only pd.np.nan in a column into a table that normally uses STRING Cod...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameUnprocessable
hostnamegithub.com
expected-hostnamegithub.com
Noned0525ab1464526bf4532a5551aa348cf100dff75ef808a96fc67a08b1d946aa9
turbo-cache-controlno-preview
go-importgithub.com/googleapis/google-cloud-python git https://github.com/googleapis/google-cloud-python.git
octolytics-dimension-user_id16785467
octolytics-dimension-user_logingoogleapis
octolytics-dimension-repository_id16316451
octolytics-dimension-repository_nwogoogleapis/google-cloud-python
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id16316451
octolytics-dimension-repository_network_root_nwogoogleapis/google-cloud-python
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releasefef3993a71fa20f7be8e167132fbf9619bc66e12
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/googleapis/google-cloud-python/issues/7370#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fgoogleapis%2Fgoogle-cloud-python%2Fissues%2F7370
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fgoogleapis%2Fgoogle-cloud-python%2Fissues%2F7370
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=googleapis%2Fgoogle-cloud-python
Reloadhttps://github.com/googleapis/google-cloud-python/issues/7370
Reloadhttps://github.com/googleapis/google-cloud-python/issues/7370
Reloadhttps://github.com/googleapis/google-cloud-python/issues/7370
googleapishttps://github.com/googleapis
google-cloud-pythonhttps://github.com/googleapis/google-cloud-python
Codehttps://github.com/googleapis/google-cloud-python
Issues449 (449)https://github.com/googleapis/google-cloud-python/issues
Pull requests199 (199)https://github.com/googleapis/google-cloud-python/pulls
Discussionshttps://github.com/googleapis/google-cloud-python/discussions
Actionshttps://github.com/googleapis/google-cloud-python/actions
Projectshttps://github.com/googleapis/google-cloud-python/projects
Security and qualityhttps://github.com/googleapis/google-cloud-python/security
Insightshttps://github.com/googleapis/google-cloud-python/insights
https://github.com/googleapis/google-cloud-python
New issuehttps://github.com/login?return_to=https://github.com/googleapis/google-cloud-python/issues/7370
New issuehttps://github.com/login?return_to=https://github.com/googleapis/google-cloud-python/issues/7370
#8105https://github.com/googleapis/google-cloud-python/pull/8105
BigQuery: Use BigQuery schema (from LoadJobConfig) if available when converting to Parquet in load_table_from_dataframehttps://github.com/googleapis/google-cloud-python/issues/7370#top
#8105https://github.com/googleapis/google-cloud-python/pull/8105
https://github.com/tswast
api: bigqueryIssues related to the BigQuery API.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22api%3A%20bigquery%22
type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22type%3A%20feature%20request%22
https://github.com/Unprocessable
Unprocessablehttps://github.com/Unprocessable
on Feb 15, 2019https://github.com/googleapis/google-cloud-python/issues/7370#issue-410770771
Unprocessablehttps://github.com/Unprocessable
on Feb 15, 2019https://github.com/googleapis/google-cloud-python/issues/7370#event-2142481067
tseaverhttps://github.com/tseaver
type: questionRequest for information or clarification. Not an issue.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22type%3A%20question%22
api: bigqueryIssues related to the BigQuery API.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22api%3A%20bigquery%22
on Feb 15, 2019https://github.com/googleapis/google-cloud-python/issues/7370#event-2143601571
tseaverhttps://github.com/tseaver
tswasthttps://github.com/tswast
on Feb 15, 2019https://github.com/googleapis/google-cloud-python/issues/7370#event-2143601721
https://github.com/tseaver
tseaverhttps://github.com/tseaver
on Feb 15, 2019https://github.com/googleapis/google-cloud-python/issues/7370#issuecomment-464208398
@tswasthttps://github.com/tswast
https://github.com/tswast
tswasthttps://github.com/tswast
on Feb 16, 2019https://github.com/googleapis/google-cloud-python/issues/7370#issuecomment-464255553
https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#values-considered-missinghttps://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#values-considered-missing
to_parquethttps://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html
pyarrow.parquet.write_tablehttps://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html
followhttps://sourcegraph.com/github.com/pandas-dev/pandas@83fe6ca422ecc35f311897abedadcc492b2a0777/-/blob/pandas/io/parquet.py#L23
thehttps://sourcegraph.com/github.com/pandas-dev/pandas@83fe6ca422ecc35f311897abedadcc492b2a0777/-/blob/pandas/io/parquet.py#L113
breadcrumbshttps://github.com/apache/arrow/blob/524a9dc79556a15d1f56c094913f0e0682358844/python/pyarrow/table.pxi#L840
pyarrow's schema inferencehttps://github.com/apache/arrow/blob/524a9dc79556a15d1f56c094913f0e0682358844/python/pyarrow/table.pxi#L587
pyarrow Schemahttps://arrow.apache.org/docs/python/generated/pyarrow.Schema.html
https://github.com/tswast
tswasthttps://github.com/tswast
on Feb 16, 2019https://github.com/googleapis/google-cloud-python/issues/7370#issuecomment-464256097
https://github.com/tswast
tswasthttps://github.com/tswast
on Feb 16, 2019https://github.com/googleapis/google-cloud-python/issues/7370#issuecomment-464256754
https://issues.apache.org/jira/browse/ARROW-2298https://issues.apache.org/jira/browse/ARROW-2298
https://issues.apache.org/jira/browse/ARROW-2135https://issues.apache.org/jira/browse/ARROW-2135
tswasthttps://github.com/tswast
type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22type%3A%20feature%20request%22
type: questionRequest for information or clarification. Not an issue.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22type%3A%20question%22
on Feb 16, 2019https://github.com/googleapis/google-cloud-python/issues/7370#event-2143861329
tswasthttps://github.com/tswast
on Feb 16, 2019https://github.com/googleapis/google-cloud-python/issues/7370#event-2143862019
https://github.com/max-sixty
max-sixtyhttps://github.com/max-sixty
on Feb 16, 2019https://github.com/googleapis/google-cloud-python/issues/7370#issuecomment-464283460
apache/arrow#3479https://github.com/apache/arrow/issues/3479
https://github.com/Unprocessable
Unprocessablehttps://github.com/Unprocessable
on Feb 18, 2019https://github.com/googleapis/google-cloud-python/issues/7370#issuecomment-464777762
apache/arrow#3479https://github.com/apache/arrow/issues/3479
https://issues.apache.org/jira/browse/ARROW-2298https://issues.apache.org/jira/browse/ARROW-2298
https://issues.apache.org/jira/browse/ARROW-2135https://issues.apache.org/jira/browse/ARROW-2135
pyarrow Schemahttps://arrow.apache.org/docs/python/generated/pyarrow.Schema.html
https://github.com/max-sixty
max-sixtyhttps://github.com/max-sixty
on Feb 26, 2019https://github.com/googleapis/google-cloud-python/issues/7370#issuecomment-467611390
https://github.com/tswast
tswasthttps://github.com/tswast
on May 17, 2019https://github.com/googleapis/google-cloud-python/issues/7370#issuecomment-493351444
https://github.com/tswast
tswasthttps://github.com/tswast
on May 17, 2019https://github.com/googleapis/google-cloud-python/issues/7370#issuecomment-493618102
https://arrow.apache.org/docs/python/pandas.html#pandas-arrow-conversionhttps://arrow.apache.org/docs/python/pandas.html#pandas-arrow-conversion
https://arrow.apache.org/docs/python/parquet.html#data-type-handlinghttps://arrow.apache.org/docs/python/parquet.html#data-type-handling
https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-parquet#parquet_conversionshttps://cloud.google.com/bigquery/docs/loading-data-cloud-storage-parquet#parquet_conversions
https://cloud.google.com/bigquery/data-types#legacy_sql_data_typeshttps://cloud.google.com/bigquery/data-types#legacy_sql_data_types
tswasthttps://github.com/tswast
completedhttps://github.com/googleapis/google-cloud-python/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
#8105https://github.com/googleapis/google-cloud-python/pull/8105
on May 31, 2019https://github.com/googleapis/google-cloud-python/issues/7370#event-2379635660
Sign up for freehttps://github.com/signup?return_to=https://github.com/googleapis/google-cloud-python/issues/7370
Sign in to commenthttps://github.com/login?return_to=https://github.com/googleapis/google-cloud-python/issues/7370
tswasthttps://github.com/tswast
api: bigqueryIssues related to the BigQuery API.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22api%3A%20bigquery%22
type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22type%3A%20feature%20request%22
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.