René's URL Explorer Experiment


Title: Ascend/TransferQueue - AtomGit

Open Graph Title: Ascend/TransferQueue - AtomGit

X Title: Ascend/TransferQueue - AtomGit

Description: 可用于优化训练后工作流中的高效数据流,提供全景数据可见性和流式调度能力。具备细粒度数据管理、存储后端可插拔及多种API接口,能解耦计算任务数据依赖,简化算法控制器设计。

Open Graph Description: 可用于优化训练后工作流中的高效数据流,提供全景数据可见性和流式调度能力。具备细粒度数据管理、存储后端可插拔及多种API接口,能解耦计算任务数据依赖,简化算法控制器设计。

X Description: 可用于优化训练后工作流中的高效数据流,提供全景数据可见性和流式调度能力。具备细粒度数据管理、存储后端可插拔及多种API接口,能解耦计算任务数据依赖,简化算法控制器设计。

Keywords:

Opengraph URL: https://gitcode.com/Ascend/TransferQueue

direct link

Domain: gitcode.com

IPv4 address: 116.205.6.180


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"SoftwareSourceCode","title":"TransferQueue:基于分布式存储的高性能数据流转管理项目 - AtomGit","description":"可用于优化训练后工作流中的高效数据流,提供全景数据可见性和流式调度能力。具备细粒度数据管理、存储后端可插拔及多种API接口,能解耦计算任务数据依赖,简化算法控制器设计。","programmingLanguage":"Python","codeRepository":"https://gitcode.com/Ascend/TransferQueue","author":{"@type":"Person","name":"ascend-robot"},"dateModified":"2026-11-23T02:04:22.021+08:00","dateCreated":"2026-01-04T09:47:14.546+08:00","license":"Apache-2.0","keywords":"分布式训练框架,Apache-2.0,Linux,库,pytorch,人工智能,Python,数据工程,ai_repos,存储,Ascend,频道-存储接口"}

applicable-devicepc,mobile
theme-color#171717
mobile-web-app-capableyes
og:typewebsite
twitter:cardsummary

Links:

9https://gitcode.com/Ascend/TransferQueue/forks
CodeIntroductionhttps://gitcode.com/Ascend/TransferQueue
Codehttps://gitcode.com/Ascend/TransferQueue/tree/main
Issueshttps://gitcode.com/Ascend/TransferQueue/issues
Pull Requests3https://gitcode.com/Ascend/TransferQueue/pulls
Pipelinehttps://gitcode.com/Ascend/TransferQueue/pipeline
Actionshttps://gitcode.com/Ascend/TransferQueue/actions
Discussionhttps://gitcode.com/Ascend/TransferQueue/discussions
Wikihttps://gitcode.com/Ascend/TransferQueue/wiki
Members68https://gitcode.com/Ascend/TransferQueue/member
Analysishttps://gitcode.com/Ascend/TransferQueue/analysis
Settingshttps://gitcode.com/Ascend/TransferQueue/setting
9https://gitcode.com/Ascend/TransferQueue/forks
Branch10https://gitcode.com/Ascend/TransferQueue/branches
Tags7https://gitcode.com/Ascend/TransferQueue/tags
https://gitcode.com/Ascend/TransferQueue#transferqueue-an-asynchronous-streaming-data-management-module-for-efficient-post-training
Paperhttps://arxiv.org/abs/2507.01663
Zhihuhttps://zhuanlan.zhihu.com/p/1930244241625449814
WeChathttps://github.com/TransferQueue/community_doc/blob/main/other_assets/WeChatGroup.png?raw=true
https://deepwiki.com/Ascend/TransferQueue
https://github.com/Ascend/TransferQueue/stargazers/
https://github.com/Ascend/TransferQueue/graphs/commit-activity
https://gitcode.com/Ascend/TransferQueue#overview
https://gitcode.com/Ascend/TransferQueue#updates
Meshyhttps://github.com/OpenBMB/Meshy
Dressagehttps://github.com/Accio-Lab/Dressage
reporthttps://github.com/Accio-Lab/Dressage/blob/main/docs/trajectory-storage-refactor-en.md
ROLLhttps://github.com/alibaba/ROLL/pull/463
RemoteBatchhttps://github.com/alibaba/ROLL/blob/main/docs_roll/docs/User%20Guides/Advanced%20Features/remote_batch_transfer.md
UniRLhttps://github.com/Tencent-Hunyuan/UniRL
Relaxhttps://github.com/redai-infra/Relax
verlhttps://github.com/verl-project/verl/pull/5401
our bloghttps://www.yuque.com/haomingzi-lfse7/lhp4el/gm8mkpfu83luuhxg?singleDoc
PR#26https://github.com/Ascend/TransferQueue/pull/26
PR#28https://github.com/Ascend/TransferQueue/pull/28
tutorials/06_streaming_dataloader.pyhttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/06_streaming_dataloader.py
tutorialhttps://gitcode.com/Ascend/TransferQueue/tree/main/tutorial
PR#101https://github.com/TransferQueue/TransferQueue/pull/101
PR#96https://github.com/TransferQueue/TransferQueue/pull/96
openYuanronghttps://gitcode.com/openeuler/yuanrong-datasystem
PR#98https://github.com/TransferQueue/TransferQueue/pull/98
PR#66https://github.com/TransferQueue/TransferQueue/pull/66
verl/pull/3649https://github.com/volcengine/verl/pull/3649
Zhihu 1https://zhuanlan.zhihu.com/p/1930244241625449814
2https://zhuanlan.zhihu.com/p/1933259599953232589
verl/RFC#2662https://github.com/volcengine/verl/discussions/2662
AsyncFlowhttps://arxiv.org/abs/2507.01663
https://gitcode.com/Ascend/TransferQueue#components
https://gitcode.com/Ascend/TransferQueue#control-plane-panoramic-data-management
Customizehttps://gitcode.com/Ascend/TransferQueue#customize
#PR70https://github.com/Ascend/TransferQueue/pull/70
https://gitcode.com/Ascend/TransferQueue#data-plane-distributed-data-storage
Customizehttps://gitcode.com/Ascend/TransferQueue#customize
local SSD offloadhttps://gitcode.com/Ascend/TransferQueue/blob/main/docs/ssd_offload.md
Yuanronghttps://gitee.com/openeuler/yuanrong-datasystem
usage guidehttps://gitcode.com/Ascend/TransferQueue/blob/main/docs/storage_backends/openyuanrong_datasystem.md
#PR107https://github.com/TransferQueue/TransferQueue/pull/107
#PR96https://github.com/TransferQueue/TransferQueue/pull/96
MooncakeStorehttps://github.com/kvcache-ai/Mooncake
#PR162https://github.com/TransferQueue/TransferQueue/pull/162
RayRDThttps://docs.ray.io/en/master/ray-core/direct-transport.html
#PR167https://github.com/TransferQueue/TransferQueue/pull/167
https://gitcode.com/Ascend/TransferQueue#user-interface-high-level--low-level-apis
https://gitcode.com/Ascend/TransferQueue#key-value-api
the interactive notebookhttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/basic.ipynb
Tutorial 02https://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/02_kv_interface.py
Tutorial 02https://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/02_kv_interface.py
Tutorial 07https://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/07_shared_fields.py
Tutorial 08https://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/08_data_parser.py
the notebookhttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/basic.ipynb
Tutorial 09https://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/09_data_dump.py
https://gitcode.com/Ascend/TransferQueue#streamingdataloader-api
tutorial/06_streaming_dataloader.pyhttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/06_streaming_dataloader.py
https://gitcode.com/Ascend/TransferQueue#low-level-native-api
tutorial/03_metadata_concepts.pyhttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/03_metadata_concepts.py
tutorial/04_understanding_controller.pyhttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/04_understanding_controller.py
tutorial/05_custom_sampler.pyhttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/05_custom_sampler.py
https://gitcode.com/Ascend/TransferQueue#tutorials
the basic notebookhttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/basic.ipynb
01: Core componentshttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/01_core_components.py
02: KV interfacehttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/02_kv_interface.py
03: Metadatahttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/03_metadata_concepts.py
04: Controllerhttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/04_understanding_controller.py
05: Custom samplershttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/05_custom_sampler.py
06: StreamingDataLoaderhttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/06_streaming_dataloader.py
07: Shared fieldshttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/07_shared_fields.py
08: Data parsershttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/08_data_parser.py
09: Selective recoveryhttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/09_data_dump.py
https://gitcode.com/Ascend/TransferQueue#show-cases
https://gitcode.com/Ascend/TransferQueue#collocated-example
https://gitcode.com/Ascend/TransferQueue#verl
verl/pull/5401https://github.com/verl-project/verl/pull/5401
[RFC] PPOTrainer with TransferQueue Integrationhttps://github.com/verl-project/verl/issues/5400
recipehttps://gitcode.com/Ascend/TransferQueue/blob/main/recipe/simple_use_case/single_controller_demo.py
https://gitcode.com/Ascend/TransferQueue#disaggregated-example
tutorial/06_streaming_dataloader.pyhttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/06_streaming_dataloader.py
https://gitcode.com/Ascend/TransferQueue#quick-start
https://gitcode.com/Ascend/TransferQueue#use-python-package
https://gitcode.com/Ascend/TransferQueue#install-from-source-code
https://gitcode.com/Ascend/TransferQueue#build-wheel-package-from-source-code
https://gitcode.com/Ascend/TransferQueue#performance
https://gitcode.com/Ascend/TransferQueue#simple-case-regular-tensor
https://gitcode.com/Ascend/TransferQueue#complex-case-regular-tensor--nestedtensor--nontensor
the full benchmark reporthttps://www.yuque.com/haomingzi-lfse7/lhp4el/mywsxovevynra42u?singleDoc
https://gitcode.com/Ascend/TransferQueue#stress-test
stress test reporthttps://www.yuque.com/haomingzi-lfse7/lhp4el/mt0vedqy7c337pgg?singleDoc
https://gitcode.com/Ascend/TransferQueue#customize
https://gitcode.com/Ascend/TransferQueue#define-your-own-data-retrieval-logic
tutorial/05_custom_sampler.pyhttps://gitcode.com/Ascend/TransferQueue/blob/main/tutorial/05_custom_sampler.py
https://gitcode.com/Ascend/TransferQueue#how-to-integrate-a-new-storage-backend
https://gitcode.com/Ascend/TransferQueue#contribution
https://gitcode.com/Ascend/TransferQueue#citation
Apache-2.0https://gitcode.com/Ascend/TransferQueue?tab=license
Pythonhttps://gitcode.com/Ascend/TransferQueue/tree/main
181Commitshttps://gitcode.com/Ascend/TransferQueue/commits/main
pytorchhttps://gitcode.com/search?q=pytorch&type=tag
Customize your domainhttps://gitcode.com/Ascend/TransferQueue
README https://gitcode.com/Ascend/TransferQueue?tab=md#markdown-card-anchor
Rulesetshttps://gitcode.com/Ascend/TransferQueue/rules

Response headers:
Serverelb
Content-Typetext/html;charset=utf-8
X-Powered-ByNuxt
Cache-Controlprivate, no-cache, no-store

Viewport: width=device-width, initial-scale=1.0

Robots: index,follow


URLs of crawlers that visited me.