René's URL Explorer Experiment


Title: Dataloader in multi-threaded environments · Issue #71 · graphql-python/graphql-core · GitHub

Open Graph Title: Dataloader in multi-threaded environments · Issue #71 · graphql-python/graphql-core

X Title: Dataloader in multi-threaded environments · Issue #71 · graphql-python/graphql-core

Description: Hi, I've been thinking a bit about how we could implement the Dataloader pattern in v3 while still running in multi-threaded mode. Since v3 does not support Syrus's Promise library, we need to come up with a story for batching in async m...

Open Graph Description: Hi, I've been thinking a bit about how we could implement the Dataloader pattern in v3 while still running in multi-threaded mode. Since v3 does not support Syrus's Promise library, we need to come...

X Description: Hi, I've been thinking a bit about how we could implement the Dataloader pattern in v3 while still running in multi-threaded mode. Since v3 does not support Syrus's Promise library, we need...

Opengraph URL: https://github.com/graphql-python/graphql-core/issues/71

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Dataloader in multi-threaded environments","articleBody":"Hi, \r\n\r\nI've been thinking a bit about how we could implement the Dataloader pattern in v3 while still running in multi-threaded mode. Since v3  does not support Syrus's [Promise library](https://github.com/syrusakbary/promise), we need to come up with a story for batching in async mode, as well as in multi-threaded environments. There are many libraries that do not support `asyncio` and there [are many cases](https://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/) where it does not make sense to go fully async. \r\n\r\nAs far as I understand, the only way to batch resolver calls from a [single frame of execution](https://github.com/graphql/dataloader/blob/master/src/index.js#L220-L222) would be to use [`loop.call_soon`](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.call_soon). But since `asyncio` is [not threadsafe](https://docs.python.org/3/library/asyncio-dev.html#concurrency-and-multithreading), that means we would need to run a separate event loop in each worker thread. We would need to wrap the `graphql` call with something like this:\r\n\r\n```py\r\ndef run_batched_query(...):\r\n    loop = asyncio.new_event_loop()\r\n    execution_future = graphql(...)\r\n    loop.run_until_complete(result_future)\r\n    return execution_future.result()\r\n```\r\n\r\nIs that completely crazy? If yes, do you see a less hacky way? I'm not very familiar with `asyncio` so I would love to get feedback.\r\n\r\nCheers\r\n","author":{"url":"https://github.com/jnak","@type":"Person","name":"jnak"},"datePublished":"2019-12-09T23:33:55Z","url":"https://github.com/graphql-python/graphql-core/issues/71"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:521f6c30-7b66-fe8a-bdbb-d7afb72cf6a0
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-id9386:2DB972:192998E:2058703:6AFFB855
html-safe-nonce8eb0ea41ff6ef94f10f1025bc44fb1dcbf987737233bd46c4277eef1dff1e0de
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5Mzg2OjJEQjk3MjoxOTI5OThFOjIwNTg3MDM6NkFGRkI4NTUiLCJ2aXNpdG9yX2lkIjoiODUyNTgxNjMwNjkzMzQxMzk3MyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac6dca97e847de35acf816ce39bf6c8b46508f13601a16f812af038c4012eb511f
hovercard-subject-tagissue:535400876
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/graphql-python/graphql-core/71/issue_layout
twitter:imagehttps://opengraph.githubassets.com/d1b2a95ae130f8b445ac01eab6c99f9bdfa79bcd2f8a498a1e43f9881ef0fed3/graphql-python/graphql-core/issues/71
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/d1b2a95ae130f8b445ac01eab6c99f9bdfa79bcd2f8a498a1e43f9881ef0fed3/graphql-python/graphql-core/issues/71
og:image:altHi, I've been thinking a bit about how we could implement the Dataloader pattern in v3 while still running in multi-threaded mode. Since v3 does not support Syrus's Promise library, we need to come...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernamejnak
hostnamegithub.com
expected-hostnamegithub.com
None7e461843db1c75e40d788c5cfd440d0e4ae1271cf19e06a798d0e7feb2d708f3
turbo-cache-controlno-preview
go-importgithub.com/graphql-python/graphql-core git https://github.com/graphql-python/graphql-core.git
octolytics-dimension-user_id15002022
octolytics-dimension-user_logingraphql-python
octolytics-dimension-repository_id143207933
octolytics-dimension-repository_nwographql-python/graphql-core
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id143207933
octolytics-dimension-repository_network_root_nwographql-python/graphql-core
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
released29a6a99861646939301302fab465f9f123b4773
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/graphql-python/graphql-core/issues/71#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fgraphql-python%2Fgraphql-core%2Fissues%2F71
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fgraphql-python%2Fgraphql-core%2Fissues%2F71
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=graphql-python%2Fgraphql-core
Reloadhttps://github.com/graphql-python/graphql-core/issues/71
Reloadhttps://github.com/graphql-python/graphql-core/issues/71
Reloadhttps://github.com/graphql-python/graphql-core/issues/71
graphql-pythonhttps://github.com/graphql-python
graphql-corehttps://github.com/graphql-python/graphql-core
Codehttps://github.com/graphql-python/graphql-core
Issues24 (24)https://github.com/graphql-python/graphql-core/issues
Pull requests3 (3)https://github.com/graphql-python/graphql-core/pulls
Actionshttps://github.com/graphql-python/graphql-core/actions
Projectshttps://github.com/graphql-python/graphql-core/projects
Security and quality3 (3)https://github.com/graphql-python/graphql-core/security
Insightshttps://github.com/graphql-python/graphql-core/insights
https://github.com/graphql-python/graphql-core
New issuehttps://github.com/login?return_to=https://github.com/graphql-python/graphql-core/issues/71
New issuehttps://github.com/login?return_to=https://github.com/graphql-python/graphql-core/issues/71
Dataloader in multi-threaded environmentshttps://github.com/graphql-python/graphql-core/issues/71#top
https://github.com/Cito
discussionNeeds more discussionhttps://github.com/graphql-python/graphql-core/issues?q=state%3Aopen%20label%3A%22discussion%22
help wantedExtra attention is neededhttps://github.com/graphql-python/graphql-core/issues?q=state%3Aopen%20label%3A%22help%20wanted%22
investigateNeeds investigaton or experimentationhttps://github.com/graphql-python/graphql-core/issues?q=state%3Aopen%20label%3A%22investigate%22
https://github.com/jnak
jnakhttps://github.com/jnak
on Dec 9, 2019https://github.com/graphql-python/graphql-core/issues/71#issue-535400876
Promise libraryhttps://github.com/syrusakbary/promise
are many caseshttps://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/
single frame of executionhttps://github.com/graphql/dataloader/blob/master/src/index.js#L220-L222
loop.call_soonhttps://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.call_soon
not threadsafehttps://docs.python.org/3/library/asyncio-dev.html#concurrency-and-multithreading
https://github.com/Cito
Citohttps://github.com/Cito
on Dec 10, 2019https://github.com/graphql-python/graphql-core/issues/71#issuecomment-564217564
https://github.com/graphql/dataloaderhttps://github.com/graphql/dataloader
https://github.com/syrusakbary/aiodataloaderhttps://github.com/syrusakbary/aiodataloader
Citohttps://github.com/Cito
on Dec 10, 2019https://github.com/graphql-python/graphql-core/issues/71#event-2871872687
Citohttps://github.com/Cito
investigateNeeds investigaton or experimentationhttps://github.com/graphql-python/graphql-core/issues?q=state%3Aopen%20label%3A%22investigate%22
on Dec 10, 2019https://github.com/graphql-python/graphql-core/issues/71#event-2871873257
https://github.com/jnak
jnakhttps://github.com/jnak
on Dec 10, 2019https://github.com/graphql-python/graphql-core/issues/71#issuecomment-564288876
loop.call_soonhttps://github.com/syrusakbary/aiodataloader/blob/master/aiodataloader.py#L160
Dataloader v2https://github.com/graphql/dataloader/releases/tag/v2.0.0
same underlying mechanismhttps://github.com/graphql/dataloader/blob/master/src/index.js#L206-L245
Dataloader queuehttps://github.com/syrusakbary/aiodataloader/blob/master/aiodataloader.py#L47
threading.localhttps://docs.python.org/3.8/library/threading.html#thread-local-data
a single global event loophttps://github.com/syrusakbary/aiodataloader/blob/master/aiodataloader.py#L21
https://github.com/Cito
Citohttps://github.com/Cito
on Dec 12, 2019https://github.com/graphql-python/graphql-core/issues/71#issuecomment-564973061
graphql/dataloader#30https://github.com/graphql/dataloader/issues/30
https://github.com/jnak
jnakhttps://github.com/jnak
on Dec 16, 2019https://github.com/graphql-python/graphql-core/issues/71#issuecomment-566244809
syrusakbary/promise#81https://github.com/syrusakbary/promise/pull/81
https://github.com/silas
silashttps://github.com/silas
on Jan 18, 2020https://github.com/graphql-python/graphql-core/issues/71#issuecomment-575915595
sync_to_asynchttps://github.com/django/asgiref/blob/e47947e1add08e7103ef514abb63aa9e1e7bc790/asgiref/sync.py#L165-L300
https://github.com/jnak
jnakhttps://github.com/jnak
on Jan 22, 2020https://github.com/graphql-python/graphql-core/issues/71#issuecomment-577404489
@silashttps://github.com/silas
syrusakbary/promise#81https://github.com/syrusakbary/promise/pull/81
https://github.com/silas
silashttps://github.com/silas
on Jan 22, 2020https://github.com/graphql-python/graphql-core/issues/71#issuecomment-577417395
@jhgghttps://github.com/jhgg
https://github.com/miracle2k
miracle2khttps://github.com/miracle2k
on Apr 27, 2020https://github.com/graphql-python/graphql-core/issues/71#issuecomment-620079719
https://github.com/Cito
Citohttps://github.com/Cito
on Apr 27, 2020https://github.com/graphql-python/graphql-core/issues/71#issuecomment-620084294
@miracle2khttps://github.com/miracle2k
https://github.com/miracle2k
miracle2khttps://github.com/miracle2k
on Apr 27, 2020https://github.com/graphql-python/graphql-core/issues/71#issuecomment-620106364
@Citohttps://github.com/Cito
Citohttps://github.com/Cito
discussionNeeds more discussionhttps://github.com/graphql-python/graphql-core/issues?q=state%3Aopen%20label%3A%22discussion%22
help wantedExtra attention is neededhttps://github.com/graphql-python/graphql-core/issues?q=state%3Aopen%20label%3A%22help%20wanted%22
on Dec 28, 2021https://github.com/graphql-python/graphql-core/issues/71#event-5822922211
pylipphttps://github.com/pylipp
on Aug 20, 2022https://github.com/graphql-python/graphql-core/issues/71#event-7226509129
Integrate async data loaders for box product, size, and tagshttps://github.com/boxwise/boxtribute/commit/8833b21561ab38c6625f7647f0fb6540d1f2d967
8833b21https://github.com/boxwise/boxtribute/commit/8833b21561ab38c6625f7647f0fb6540d1f2d967
Integrate async data loaders for box product, size, and tagshttps://github.com/boxwise/boxtribute/commit/a77a30472ca872729277a84d606bf07e85839ae1
a77a304https://github.com/boxwise/boxtribute/commit/a77a30472ca872729277a84d606bf07e85839ae1
Integrate async data loaders for box product, size, and tagshttps://github.com/boxwise/boxtribute/commit/4b85da438077bce6063e605aea7aa2b752455722
4b85da4https://github.com/boxwise/boxtribute/commit/4b85da438077bce6063e605aea7aa2b752455722
Sign up for freehttps://github.com/signup?return_to=https://github.com/graphql-python/graphql-core/issues/71
Sign in to commenthttps://github.com/login?return_to=https://github.com/graphql-python/graphql-core/issues/71
Citohttps://github.com/Cito
discussionNeeds more discussionhttps://github.com/graphql-python/graphql-core/issues?q=state%3Aopen%20label%3A%22discussion%22
help wantedExtra attention is neededhttps://github.com/graphql-python/graphql-core/issues?q=state%3Aopen%20label%3A%22help%20wanted%22
investigateNeeds investigaton or experimentationhttps://github.com/graphql-python/graphql-core/issues?q=state%3Aopen%20label%3A%22investigate%22
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.