René's URL Explorer Experiment


Title: GitHub - Ascend/TransferQueue: An asynchronous streaming data management module for efficient post-training. · GitHub

Open Graph Title: GitHub - Ascend/TransferQueue: An asynchronous streaming data management module for efficient post-training.

X Title: GitHub - Ascend/TransferQueue: An asynchronous streaming data management module for efficient post-training.

Description: An asynchronous streaming data management module for efficient post-training. - Ascend/TransferQueue

Open Graph Description: An asynchronous streaming data management module for efficient post-training. - Ascend/TransferQueue

X Description: An asynchronous streaming data management module for efficient post-training. - Ascend/TransferQueue

Opengraph URL: https://github.com/Ascend/TransferQueue

X: @github

direct link

Domain: github.com

IPv4 address: 140.82.113.3

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository
route-controllerfiles
route-actiondisambiguate
fetch-noncev2:6d6d7a46-f10b-15b7-9774-d63c34c5f0a2
current-catalog-service-hashf3abb0cc802f3d7b95fc8762b94bdcb13bf39634c40c357301c4aa1d67a256fb
request-idC81A:1E74B0:AD6500:DBF4AE:6B036ACE
html-safe-nonce153b4604490a33d61a52cb6410c4c463d8e02062048b2b193b775f37393429d9
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDODFBOjFFNzRCMDpBRDY1MDA6REJGNEFFOjZCMDM2QUNFIiwidmlzaXRvcl9pZCI6Ijc5OTQ2ODI1NTIxMTgxMTA5MjYiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac0f2f8baf711a54389ebfed972da0292696c80bdfbf06dfcc1911bf98ac7f74a4
hovercard-subject-tagrepository:1130891249
github-keyboard-shortcutsrepository,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location//
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/Ascend/TransferQueue
twitter:imagehttps://opengraph.githubassets.com/e6cd1983a87c132e02b3c9a590e3ad160125c3da985804fefedb8bc94e5ad0db/Ascend/TransferQueue
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/e6cd1983a87c132e02b3c9a590e3ad160125c3da985804fefedb8bc94e5ad0db/Ascend/TransferQueue
og:image:altAn asynchronous streaming data management module for efficient post-training. - Ascend/TransferQueue
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456
turbo-cache-controlno-cache
go-importgithub.com/Ascend/TransferQueue git https://github.com/Ascend/TransferQueue.git
octolytics-dimension-user_id53141342
octolytics-dimension-user_loginAscend
octolytics-dimension-repository_id1130891249
octolytics-dimension-repository_nwoAscend/TransferQueue
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id1130891249
octolytics-dimension-repository_network_root_nwoAscend/TransferQueue
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release423dbb54c5e7bfecc838eb387c1435eb3d0e760d
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/Ascend/TransferQueue#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FAscend%2FTransferQueue
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FAscend%2FTransferQueue
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=Ascend%2FTransferQueue
Reloadhttps://github.com/Ascend/TransferQueue
Reloadhttps://github.com/Ascend/TransferQueue
Reloadhttps://github.com/Ascend/TransferQueue
Ascendhttps://github.com/Ascend
TransferQueuehttps://github.com/Ascend/TransferQueue
Codehttps://github.com/Ascend/TransferQueue
Issues13 (13)https://github.com/Ascend/TransferQueue/issues
Pull requests12 (12)https://github.com/Ascend/TransferQueue/pulls
Actionshttps://github.com/Ascend/TransferQueue/actions
Projectshttps://github.com/Ascend/TransferQueue/projects
Security and qualityhttps://github.com/Ascend/TransferQueue/security
Insightshttps://github.com/Ascend/TransferQueue/insights
https://github.com/Ascend/TransferQueue
TransferQueuehttps://github.com/Ascend/TransferQueue
Notificationshttps://github.com/login?return_to=%2FAscend%2FTransferQueue
Fork58 (58)https://github.com/login?return_to=%2FAscend%2FTransferQueue
Star177 (177)https://github.com/login?return_to=%2FAscend%2FTransferQueue
Readmehttps://github.com/Ascend/TransferQueue#readme-ov-file
Apache-2.0 licensehttps://github.com/Ascend/TransferQueue#Apache-2.0-1-ov-file
Activityhttps://github.com/Ascend/TransferQueue/activity
Custom propertieshttps://github.com/Ascend/TransferQueue/custom-properties
58 forkshttps://github.com/Ascend/TransferQueue/forks
Report repositoryhttps://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2FAscend%2FTransferQueue&report=Ascend+%28user%29
Brancheshttps://github.com/Ascend/TransferQueue/branches
Tagshttps://github.com/Ascend/TransferQueue/tags
https://github.com/Ascend/TransferQueue/branches
https://github.com/Ascend/TransferQueue/tags
181 Commitshttps://github.com/Ascend/TransferQueue/commits/main/
https://github.com/Ascend/TransferQueue/commits/main/
.agent/skills/simplicity-firsthttps://github.com/Ascend/TransferQueue/tree/main/.agent/skills/simplicity-first
.agent/skills/simplicity-firsthttps://github.com/Ascend/TransferQueue/tree/main/.agent/skills/simplicity-first
.claude/skills/simplicity-firsthttps://github.com/Ascend/TransferQueue/tree/main/.claude/skills/simplicity-first
.claude/skills/simplicity-firsthttps://github.com/Ascend/TransferQueue/tree/main/.claude/skills/simplicity-first
.codex/skills/simplicity-firsthttps://github.com/Ascend/TransferQueue/tree/main/.codex/skills/simplicity-first
.codex/skills/simplicity-firsthttps://github.com/Ascend/TransferQueue/tree/main/.codex/skills/simplicity-first
.github/workflowshttps://github.com/Ascend/TransferQueue/tree/main/.github/workflows
.github/workflowshttps://github.com/Ascend/TransferQueue/tree/main/.github/workflows
docshttps://github.com/Ascend/TransferQueue/tree/main/docs
docshttps://github.com/Ascend/TransferQueue/tree/main/docs
recipe/simple_use_casehttps://github.com/Ascend/TransferQueue/tree/main/recipe/simple_use_case
recipe/simple_use_casehttps://github.com/Ascend/TransferQueue/tree/main/recipe/simple_use_case
scriptshttps://github.com/Ascend/TransferQueue/tree/main/scripts
scriptshttps://github.com/Ascend/TransferQueue/tree/main/scripts
testshttps://github.com/Ascend/TransferQueue/tree/main/tests
testshttps://github.com/Ascend/TransferQueue/tree/main/tests
transfer_queuehttps://github.com/Ascend/TransferQueue/tree/main/transfer_queue
transfer_queuehttps://github.com/Ascend/TransferQueue/tree/main/transfer_queue
tutorialhttps://github.com/Ascend/TransferQueue/tree/main/tutorial
tutorialhttps://github.com/Ascend/TransferQueue/tree/main/tutorial
.gitignorehttps://github.com/Ascend/TransferQueue/blob/main/.gitignore
.gitignorehttps://github.com/Ascend/TransferQueue/blob/main/.gitignore
.pre-commit-config.yamlhttps://github.com/Ascend/TransferQueue/blob/main/.pre-commit-config.yaml
.pre-commit-config.yamlhttps://github.com/Ascend/TransferQueue/blob/main/.pre-commit-config.yaml
AGENTS.mdhttps://github.com/Ascend/TransferQueue/blob/main/AGENTS.md
AGENTS.mdhttps://github.com/Ascend/TransferQueue/blob/main/AGENTS.md
CLAUDE.mdhttps://github.com/Ascend/TransferQueue/blob/main/CLAUDE.md
CLAUDE.mdhttps://github.com/Ascend/TransferQueue/blob/main/CLAUDE.md
LICENSEhttps://github.com/Ascend/TransferQueue/blob/main/LICENSE
LICENSEhttps://github.com/Ascend/TransferQueue/blob/main/LICENSE
README.mdhttps://github.com/Ascend/TransferQueue/blob/main/README.md
README.mdhttps://github.com/Ascend/TransferQueue/blob/main/README.md
pyproject.tomlhttps://github.com/Ascend/TransferQueue/blob/main/pyproject.toml
pyproject.tomlhttps://github.com/Ascend/TransferQueue/blob/main/pyproject.toml
requirements.txthttps://github.com/Ascend/TransferQueue/blob/main/requirements.txt
requirements.txthttps://github.com/Ascend/TransferQueue/blob/main/requirements.txt
READMEhttps://github.com/Ascend/TransferQueue
Apache-2.0 licensehttps://github.com/Ascend/TransferQueue
https://github.com/Ascend/TransferQueue#------transferqueue-an-asynchronous-streaming-data-management-module-for-efficient-post-training--
Paperhttps://arxiv.org/abs/2507.01663
Zhihuhttps://zhuanlan.zhihu.com/p/1930244241625449814
WeChathttps://github.com/TransferQueue/community_doc/blob/main/other_assets/WeChatGroup.png?raw=true
https://deepwiki.com/Ascend/TransferQueue
https://github.com/Ascend/TransferQueue/stargazers/
https://github.com/Ascend/TransferQueue/graphs/commit-activity
https://github.com/Ascend/TransferQueue#-overview
https://github.com/TransferQueue/community_doc/blob/main/docs/tq_arch.png?raw=true
https://github.com/TransferQueue/community_doc/blob/main/docs/main_func.png?raw=true
https://github.com/Ascend/TransferQueue#-updates
Meshyhttps://github.com/OpenBMB/Meshy
Dressagehttps://github.com/Accio-Lab/Dressage
reporthttps://github.com/Accio-Lab/Dressage/blob/main/docs/trajectory-storage-refactor-en.md
ROLLhttps://github.com/alibaba/ROLL/pull/463
RemoteBatchhttps://github.com/alibaba/ROLL/blob/main/docs_roll/docs/User%20Guides/Advanced%20Features/remote_batch_transfer.md
UniRLhttps://github.com/Tencent-Hunyuan/UniRL
Relaxhttps://github.com/redai-infra/Relax
verlhttps://github.com/verl-project/verl/pull/5401
our bloghttps://www.yuque.com/haomingzi-lfse7/lhp4el/gm8mkpfu83luuhxg?singleDoc
PR#26https://github.com/Ascend/TransferQueue/pull/26
PR#28https://github.com/Ascend/TransferQueue/pull/28
tutorials/06_streaming_dataloader.pyhttps://github.com/Ascend/TransferQueue/blob/main/tutorial/06_streaming_dataloader.py
tutorialhttps://github.com/Ascend/TransferQueue/tree/main/tutorial
PR#101https://github.com/TransferQueue/TransferQueue/pull/101
PR#96https://github.com/TransferQueue/TransferQueue/pull/96
openYuanronghttps://gitcode.com/openeuler/yuanrong-datasystem
PR#98https://github.com/TransferQueue/TransferQueue/pull/98
PR#66https://github.com/TransferQueue/TransferQueue/pull/66
verl/pull/3649https://github.com/volcengine/verl/pull/3649
Zhihu 1https://zhuanlan.zhihu.com/p/1930244241625449814
2https://zhuanlan.zhihu.com/p/1933259599953232589
verl/RFC#2662https://github.com/volcengine/verl/discussions/2662
AsyncFlowhttps://arxiv.org/abs/2507.01663
https://github.com/Ascend/TransferQueue#-components
https://github.com/Ascend/TransferQueue#control-plane-panoramic-data-management
https://github.com/TransferQueue/community_doc/blob/main/docs/control_plane.png?raw=true
Customizehttps://github.com/Ascend/TransferQueue#customize
#PR70https://github.com/Ascend/TransferQueue/pull/70
https://github.com/Ascend/TransferQueue#data-plane-distributed-data-storage
Customizehttps://github.com/Ascend/TransferQueue#customize
local SSD offloadhttps://github.com/Ascend/TransferQueue/blob/main/docs/ssd_offload.md
Yuanronghttps://gitee.com/openeuler/yuanrong-datasystem
usage guidehttps://github.com/Ascend/TransferQueue/blob/main/docs/storage_backends/openyuanrong_datasystem.md
#PR107https://github.com/TransferQueue/TransferQueue/pull/107
#PR96https://github.com/TransferQueue/TransferQueue/pull/96
MooncakeStorehttps://github.com/kvcache-ai/Mooncake
#PR162https://github.com/TransferQueue/TransferQueue/pull/162
RayRDThttps://docs.ray.io/en/master/ray-core/direct-transport.html
#PR167https://github.com/TransferQueue/TransferQueue/pull/167
https://github.com/TransferQueue/community_doc/blob/main/docs/data_plane.png?raw=true
https://github.com/Ascend/TransferQueue#user-interface-high-level--low-level-apis
https://github.com/Ascend/TransferQueue#key-value-api
the interactive notebookhttps://github.com/Ascend/TransferQueue/blob/main/tutorial/basic.ipynb
Tutorial 02https://github.com/Ascend/TransferQueue/blob/main/tutorial/02_kv_interface.py
Tutorial 02https://github.com/Ascend/TransferQueue/blob/main/tutorial/02_kv_interface.py
Tutorial 07https://github.com/Ascend/TransferQueue/blob/main/tutorial/07_shared_fields.py
Tutorial 08https://github.com/Ascend/TransferQueue/blob/main/tutorial/08_data_parser.py
the notebookhttps://github.com/Ascend/TransferQueue/blob/main/tutorial/basic.ipynb
Tutorial 09https://github.com/Ascend/TransferQueue/blob/main/tutorial/09_data_dump.py
https://github.com/Ascend/TransferQueue#streamingdataloader-api
tutorial/06_streaming_dataloader.pyhttps://github.com/Ascend/TransferQueue/blob/main/tutorial/06_streaming_dataloader.py
https://github.com/Ascend/TransferQueue#low-level-native-api
tutorial/03_metadata_concepts.pyhttps://github.com/Ascend/TransferQueue/blob/main/tutorial/03_metadata_concepts.py
tutorial/04_understanding_controller.pyhttps://github.com/Ascend/TransferQueue/blob/main/tutorial/04_understanding_controller.py
tutorial/05_custom_sampler.pyhttps://github.com/Ascend/TransferQueue/blob/main/tutorial/05_custom_sampler.py
https://github.com/Ascend/TransferQueue#tutorials
the basic notebookhttps://github.com/Ascend/TransferQueue/blob/main/tutorial/basic.ipynb
01: Core componentshttps://github.com/Ascend/TransferQueue/blob/main/tutorial/01_core_components.py
02: KV interfacehttps://github.com/Ascend/TransferQueue/blob/main/tutorial/02_kv_interface.py
03: Metadatahttps://github.com/Ascend/TransferQueue/blob/main/tutorial/03_metadata_concepts.py
04: Controllerhttps://github.com/Ascend/TransferQueue/blob/main/tutorial/04_understanding_controller.py
05: Custom samplershttps://github.com/Ascend/TransferQueue/blob/main/tutorial/05_custom_sampler.py
06: StreamingDataLoaderhttps://github.com/Ascend/TransferQueue/blob/main/tutorial/06_streaming_dataloader.py
07: Shared fieldshttps://github.com/Ascend/TransferQueue/blob/main/tutorial/07_shared_fields.py
08: Data parsershttps://github.com/Ascend/TransferQueue/blob/main/tutorial/08_data_parser.py
09: Selective recoveryhttps://github.com/Ascend/TransferQueue/blob/main/tutorial/09_data_dump.py
https://github.com/Ascend/TransferQueue#-showcases
https://github.com/Ascend/TransferQueue#collocated-example
https://github.com/Ascend/TransferQueue#verl
https://raw.githubusercontent.com/wuxibin89/verl/refs/heads/wuxibin/doc_images/docs/_static/transfer_queue.png
verl/pull/5401https://github.com/verl-project/verl/pull/5401
[RFC] PPOTrainer with TransferQueue Integrationhttps://github.com/verl-project/verl/issues/5400
recipehttps://github.com/Ascend/TransferQueue/blob/main/recipe/simple_use_case/single_controller_demo.py
https://github.com/Ascend/TransferQueue#disaggregated-example
tutorial/06_streaming_dataloader.pyhttps://github.com/Ascend/TransferQueue/blob/main/tutorial/06_streaming_dataloader.py
https://github.com/TransferQueue/community_doc/blob/main/docs/tq_streaming_dataloader.png?raw=true
https://github.com/Ascend/TransferQueue#-quick-start
https://github.com/Ascend/TransferQueue#use-python-package
https://github.com/Ascend/TransferQueue#install-from-source-code
https://github.com/Ascend/TransferQueue#build-wheel-package-from-source-code
https://github.com/Ascend/TransferQueue#-performance
https://github.com/Ascend/TransferQueue#simple-case-regular-tensor
https://github.com/TransferQueue/community_doc/blob/main/docs/performance_simple_0.1.8.png?raw=true
https://github.com/Ascend/TransferQueue#complex-case-regular-tensor--nestedtensor--nontensor
https://github.com/TransferQueue/community_doc/blob/main/docs/performance_complex_0.1.8.png?raw=true
the full benchmark reporthttps://www.yuque.com/haomingzi-lfse7/lhp4el/mywsxovevynra42u?singleDoc
https://github.com/Ascend/TransferQueue#stress-test
stress test reporthttps://www.yuque.com/haomingzi-lfse7/lhp4el/mt0vedqy7c337pgg?singleDoc
https://github.com/Ascend/TransferQueue#-️-customize-transferqueue
https://github.com/Ascend/TransferQueue#define-your-own-data-retrieval-logic
tutorial/05_custom_sampler.pyhttps://github.com/Ascend/TransferQueue/blob/main/tutorial/05_custom_sampler.py
https://github.com/Ascend/TransferQueue#how-to-integrate-a-new-storage-backend
https://github.com/Ascend/TransferQueue#-️-contribution-guide
https://github.com/Ascend/TransferQueue#-citation
Readmehttps://github.com/Ascend/TransferQueue#readme-ov-file
Apache-2.0 licensehttps://github.com/Ascend/TransferQueue#Apache-2.0-1-ov-file
Activityhttps://github.com/Ascend/TransferQueue/activity
Custom propertieshttps://github.com/Ascend/TransferQueue/custom-properties
58 forkshttps://github.com/Ascend/TransferQueue/forks
Report repositoryhttps://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2FAscend%2FTransferQueue&report=Ascend+%28user%29
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Response headers:
Servergithub.com
Content-Typetext/html; charset=utf-8
Cache-Controlmax-age=0, private, must-revalidate

Viewport: width=device-width


URLs of crawlers that visited me.