René's URL Explorer Experiment


Title: BigQuery: Upload pandas DataFrame containing arrays · Issue #19 · googleapis/python-bigquery · GitHub

Open Graph Title: BigQuery: Upload pandas DataFrame containing arrays · Issue #19 · googleapis/python-bigquery

X Title: BigQuery: Upload pandas DataFrame containing arrays · Issue #19 · googleapis/python-bigquery

Description: The support for python Bigquery API indicates that arrays are possible, however, when passing from a pandas dataframe to bigquery there is a pyarrow struct issue. The only way round it seems its to drop columns then use JSON Normalise fo...

Open Graph Description: The support for python Bigquery API indicates that arrays are possible, however, when passing from a pandas dataframe to bigquery there is a pyarrow struct issue. The only way round it seems its to...

X Description: The support for python Bigquery API indicates that arrays are possible, however, when passing from a pandas dataframe to bigquery there is a pyarrow struct issue. The only way round it seems its to...

Opengraph URL: https://github.com/googleapis/python-bigquery/issues/19

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"BigQuery: Upload pandas DataFrame containing arrays","articleBody":"The support for python Bigquery API indicates that arrays are possible, however, when passing from a pandas dataframe to bigquery there is a pyarrow struct issue.\r\n\r\nThe only way round it seems its to drop columns then use JSON Normalise for a separate table.\r\n\r\n```python\r\nfrom google.cloud import bigquery\r\n\r\nproject = 'lake'\r\nclient = bigquery.Client(credentials=credentials, project=project)\r\ndataset_ref = client.dataset('XXX')\r\ntable_ref = dataset_ref.table('RAW_XXX')\r\njob_config = bigquery.LoadJobConfig()\r\njob_config.autodetect = True\r\njob_config.write_disposition = 'WRITE_TRUNCATE'\r\n\r\nclient.load_table_from_dataframe(appended_data, table_ref,job_config=job_config).result()\r\n```\r\n\r\nThis is the error recieved. NotImplementedError: struct\r\n\r\nThe reason I wanted to use this API as it indicates Nested Array support, which is perfect for our data lake in BQ but I assume this doesn't work?","author":{"url":"https://github.com/AETDDraper","@type":"Person","name":"AETDDraper"},"datePublished":"2019-07-01T12:15:39Z","url":"https://github.com/googleapis/python-bigquery/issues/19"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:767ebf06-edd7-d45a-5d43-7db30c354499
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-id88F6:E40BB:914D06:C50547:6AFE7534
html-safe-nonce6b0c1ec93667b512d0f5e230670aef33edc8be9e338544f81e31a692d47fac32
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4OEY2OkU0MEJCOjkxNEQwNjpDNTA1NDc6NkFGRTc1MzQiLCJ2aXNpdG9yX2lkIjoiNTMyNzU4OTA5NjMxMzIyMjQ1MiIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmacd840fb27b14b48eb0267c2d5ccaa1eb054459ff6157e0849eb7394afaca10c5b
hovercard-subject-tagissue:559707414
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/googleapis/python-bigquery/19/issue_layout
twitter:imagehttps://opengraph.githubassets.com/d078ce0e9cce29c4179ce22dab2e4d7af664e21fc1d2abddd94077c6c889b462/googleapis/python-bigquery/issues/19
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/d078ce0e9cce29c4179ce22dab2e4d7af664e21fc1d2abddd94077c6c889b462/googleapis/python-bigquery/issues/19
og:image:altThe support for python Bigquery API indicates that arrays are possible, however, when passing from a pandas dataframe to bigquery there is a pyarrow struct issue. The only way round it seems its to...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameAETDDraper
hostnamegithub.com
expected-hostnamegithub.com
Noneb3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86
turbo-cache-controlno-preview
go-importgithub.com/googleapis/python-bigquery git https://github.com/googleapis/python-bigquery.git
octolytics-dimension-user_id16785467
octolytics-dimension-user_logingoogleapis
octolytics-dimension-repository_id226992475
octolytics-dimension-repository_nwogoogleapis/python-bigquery
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id226992475
octolytics-dimension-repository_network_root_nwogoogleapis/python-bigquery
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release4a10310af51a1bfffbedc4fd81cb582acf0cb0ff
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/googleapis/google-cloud-python/issues/8544#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fgoogleapis%2Fpython-bigquery%2Fissues%2F19
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fgoogleapis%2Fpython-bigquery%2Fissues%2F19
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=googleapis%2Fpython-bigquery
Reloadhttps://github.com/googleapis/google-cloud-python/issues/8544
Reloadhttps://github.com/googleapis/google-cloud-python/issues/8544
Reloadhttps://github.com/googleapis/google-cloud-python/issues/8544
googleapishttps://github.com/googleapis
python-bigqueryhttps://github.com/googleapis/python-bigquery
Codehttps://github.com/googleapis/python-bigquery
Issueshttps://github.com/googleapis/python-bigquery/issues
Pull requestshttps://github.com/googleapis/python-bigquery/pulls
Actionshttps://github.com/googleapis/python-bigquery/actions
Projectshttps://github.com/googleapis/python-bigquery/projects
Security and qualityhttps://github.com/googleapis/python-bigquery/security
Insightshttps://github.com/googleapis/python-bigquery/insights
https://github.com/googleapis/python-bigquery
#980https://github.com/googleapis/python-bigquery/pull/980
BigQuery: Upload pandas DataFrame containing arrayshttps://github.com/googleapis/python-bigquery/issues/19#top
#980https://github.com/googleapis/python-bigquery/pull/980
https://github.com/HemangChothani
api: bigqueryIssues related to the googleapis/python-bigquery API.https://github.com/googleapis/python-bigquery/issues?q=state%3Aopen%20label%3A%22api%3A%20bigquery%22
type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.https://github.com/googleapis/python-bigquery/issues?q=state%3Aopen%20label%3A%22type%3A%20feature%20request%22
https://github.com/AETDDraper
AETDDraperhttps://github.com/AETDDraper
on Jul 1, 2019https://github.com/googleapis/python-bigquery/issues/19#issue-559707414
https://github.com/tswast
tswasthttps://github.com/tswast
on Jul 1, 2019https://github.com/googleapis/python-bigquery/issues/19#issuecomment-507367634
https://github.com/googleapis/google-cloud-python/issues/8191https://github.com/googleapis/google-cloud-python/issues/8191
tswasthttps://github.com/tswast
on Jul 1, 2019https://github.com/googleapis/python-bigquery/issues/19#event-2452031078
https://github.com/sumit-ql
sumit-qlhttps://github.com/sumit-ql
on Sep 23, 2019https://github.com/googleapis/python-bigquery/issues/19#issuecomment-534099980
@AETDDraperhttps://github.com/AETDDraper
plamuthttps://github.com/plamut
googleapis/google-cloud-pythonhttps://github.com/googleapis/google-cloud-python
on Feb 4, 2020https://github.com/googleapis/python-bigquery/issues/19#event-3006003474
product-auto-labelhttps://github.com/apps/product-auto-label
api: bigqueryIssues related to the googleapis/python-bigquery API.https://github.com/googleapis/python-bigquery/issues?q=state%3Aopen%20label%3A%22api%3A%20bigquery%22
on Feb 4, 2020https://github.com/googleapis/python-bigquery/issues/19#event-3006003623
plamuthttps://github.com/plamut
type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.https://github.com/googleapis/python-bigquery/issues?q=state%3Aopen%20label%3A%22type%3A%20feature%20request%22
on Feb 4, 2020https://github.com/googleapis/python-bigquery/issues/19#event-3006004042
https://github.com/JonathanBonnaud
JonathanBonnaudhttps://github.com/JonathanBonnaud
on Mar 3, 2020https://github.com/googleapis/python-bigquery/issues/19#issuecomment-594032301
https://github.com/plamut
plamuthttps://github.com/plamut
on Apr 2, 2020https://github.com/googleapis/python-bigquery/issues/19#issuecomment-607798973
triedhttps://github.com/googleapis/python-bigquery/issues/21#issuecomment-606549404
https://github.com/aaaaahaaaaa
aaaaahaaaaahttps://github.com/aaaaahaaaaa
on May 20, 2020https://github.com/googleapis/python-bigquery/issues/19#issuecomment-631385953
https://github.com/HemangChothani
HemangChothanihttps://github.com/HemangChothani
on Jun 25, 2020https://github.com/googleapis/python-bigquery/issues/19#issuecomment-649502134
@aaaaahaaaaahttps://github.com/aaaaahaaaaa
#17https://github.com/googleapis/python-bigquery/issues/17
https://github.com/abuckenheimer
abuckenheimerhttps://github.com/abuckenheimer
on Aug 7, 2020https://github.com/googleapis/python-bigquery/issues/19#issuecomment-670712758
af2c987https://github.com/googleapis/python-bigquery/commit/af2c987fedca61d2b6d0ccdcef64455002ad684f
https://github.com/tswast
tswasthttps://github.com/tswast
on Nov 2, 2020https://github.com/googleapis/python-bigquery/issues/19#issuecomment-720492479
https://github.com/HemangChothani
HemangChothanihttps://github.com/HemangChothani
on Nov 3, 2020https://github.com/googleapis/python-bigquery/issues/19#issuecomment-721123310
@tswasthttps://github.com/tswast
https://github.com/tswast
tswasthttps://github.com/tswast
on Nov 3, 2020https://github.com/googleapis/python-bigquery/issues/19#issuecomment-721401033
https://cloud.google.com/bigquery/docs/bigquery-storage-python-pandashttps://cloud.google.com/bigquery/docs/bigquery-storage-python-pandas
https://cloud.google.com/bigquery/docs/pandas-gbq-migrationhttps://cloud.google.com/bigquery/docs/pandas-gbq-migration
HemangChothanihttps://github.com/HemangChothani
on Nov 4, 2020https://github.com/googleapis/python-bigquery/issues/19#event-3955984840
https://github.com/tswast
tswasthttps://github.com/tswast
on Nov 4, 2020https://github.com/googleapis/python-bigquery/issues/19#issuecomment-721880407
#365https://github.com/googleapis/python-bigquery/pull/365
https://github.com/emkornfield
emkornfieldhttps://github.com/emkornfield
on Apr 29, 2021https://github.com/googleapis/python-bigquery/issues/19#issuecomment-828942639
#365https://github.com/googleapis/python-bigquery/pull/365
--parquet_enable_list_inferencehttps://cloud.google.com/bigquery/docs/reference/bq-cli-reference#parquet_enable_list_inference_flag
https://github.com/judahrand
judahrandhttps://github.com/judahrand
on Sep 20, 2021https://github.com/googleapis/python-bigquery/issues/19#issuecomment-923099223
#365https://github.com/googleapis/python-bigquery/pull/365
--parquet_enable_list_inferencehttps://cloud.google.com/bigquery/docs/reference/bq-cli-reference#parquet_enable_list_inference_flag
https://github.com/bnaul
bnaulhttps://github.com/bnaul
on Sep 20, 2021https://github.com/googleapis/python-bigquery/issues/19#issuecomment-923144053
https://github.com/emkornfield
emkornfieldhttps://github.com/emkornfield
on Sep 20, 2021https://github.com/googleapis/python-bigquery/issues/19#issuecomment-923170663
https://github.com/judahrand
judahrandhttps://github.com/judahrand
on Sep 20, 2021https://github.com/googleapis/python-bigquery/issues/19#issuecomment-923188352
https://github.com/emkornfield
emkornfieldhttps://github.com/emkornfield
on Sep 21, 2021https://github.com/googleapis/python-bigquery/issues/19#issuecomment-924201084
@judahrandhttps://github.com/judahrand
https://github.com/judahrand
judahrandhttps://github.com/judahrand
on Sep 21, 2021https://github.com/googleapis/python-bigquery/issues/19#issuecomment-924252237
@judahrandhttps://github.com/judahrand
https://github.com/bnaul
bnaulhttps://github.com/bnaul
on Sep 21, 2021https://github.com/googleapis/python-bigquery/issues/19#issuecomment-924262035
@judahrandhttps://github.com/judahrand
python-bigquery/google/cloud/bigquery/_pandas_helpers.pyhttps://github.com/googleapis/python-bigquery/blob/ba02f248ba9c449c34859579a4011f4bfd2f4a93/google/cloud/bigquery/_pandas_helpers.py#L521-L549
ba02f24https://github.com/googleapis/python-bigquery/commit/ba02f248ba9c449c34859579a4011f4bfd2f4a93
https://github.com/judahrand
judahrandhttps://github.com/judahrand
on Sep 21, 2021https://github.com/googleapis/python-bigquery/issues/19#issuecomment-924297218
https://github.com/judahrand
judahrandhttps://github.com/judahrand
on Sep 21, 2021https://github.com/googleapis/python-bigquery/issues/19#issuecomment-924297498
gcf-merge-on-greenhttps://github.com/apps/gcf-merge-on-green
completedhttps://github.com/googleapis/python-bigquery/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
#980https://github.com/googleapis/python-bigquery/pull/980
on Oct 7, 2021https://github.com/googleapis/python-bigquery/issues/19#event-5429370519
gcf-merge-on-greenhttps://github.com/apps/gcf-merge-on-green
on Oct 7, 2021https://github.com/googleapis/python-bigquery/issues/19#event-5429370686
fix: support ARRAY data type when loading from DataFrame with Parquet (…https://github.com/googleapis/python-bigquery/commit/1e5908302d36e15442013af6f46b1c20af28255e
…https://github.com/googleapis/python-bigquery/pull/980
https://docs.github.com/articles/closing-issues-via-commit-messages
1e59083https://github.com/googleapis/python-bigquery/commit/1e5908302d36e15442013af6f46b1c20af28255e
abdelmegahedgooglehttps://github.com/abdelmegahedgoogle
on Apr 17, 2023https://github.com/googleapis/python-bigquery/issues/19#event-9028595693
fix: support ARRAY data type when loading from DataFrame with Parquet (…https://github.com/abdelmegahedgoogle/python-bigquery/commit/18b07c8c2b436f179f060395489b4c43597258e2
…https://github.com/googleapis/python-bigquery/pull/980
https://docs.github.com/articles/closing-issues-via-commit-messages
18b07c8https://github.com/abdelmegahedgoogle/python-bigquery/commit/18b07c8c2b436f179f060395489b4c43597258e2
Sign up for freehttps://github.com/signup?return_to=https://github.com/googleapis/python-bigquery/issues/19
Sign in to commenthttps://github.com/login?return_to=https://github.com/googleapis/python-bigquery/issues/19
HemangChothanihttps://github.com/HemangChothani
api: bigqueryIssues related to the googleapis/python-bigquery API.https://github.com/googleapis/python-bigquery/issues?q=state%3Aopen%20label%3A%22api%3A%20bigquery%22
type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.https://github.com/googleapis/python-bigquery/issues?q=state%3Aopen%20label%3A%22type%3A%20feature%20request%22
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.