René's URL Explorer Experiment

    {
       "@context": "https://schema.org",
       "@type": "Article",
       "name": "\ubd84\uc0b0 \ub370\uc774\ud130 \ubcd1\ub82c \ucc98\ub9ac \uc2dc\uc791\ud558\uae30",
       "headline": "\ubd84\uc0b0 \ub370\uc774\ud130 \ubcd1\ub82c \ucc98\ub9ac \uc2dc\uc791\ud558\uae30",
       "description": "PyTorch Documentation. Explore PyTorch, an open-source machine learning library that accelerates the path from research prototyping to production deployment. Discover tutorials, API references, and guides to help you build and deploy deep learning models efficiently.",
       "url": "/intermediate/ddp_tutorial.html",
       "articleBody": "\ubd84\uc0b0 \ub370\uc774\ud130 \ubcd1\ub82c \ucc98\ub9ac \uc2dc\uc791\ud558\uae30# \uc800\uc790: Shen Li \uac10\uc218: Joe Zhu, Chirag Pandya \ubc88\uc5ed: \uc870\ubcd1\uadfc \ucc38\uace0 \uc774 \ud29c\ud1a0\ub9ac\uc5bc\uc758 \uc18c\uc2a4 \ucf54\ub4dc\ub294 GitHub \uc5d0\uc11c \ud655\uc778\ud558\uace0 \ubcc0\uacbd\ud574 \ubcfc \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc120\uc218\uacfc\ubaa9(Prerequisites): PyTorch \ubd84\uc0b0 \ucc98\ub9ac \uac1c\uc694 \ubd84\uc0b0 \ub370\uc774\ud130 \ubcd1\ub82c \ucc98\ub9ac API \ubb38\uc11c \ubd84\uc0b0 \ub370\uc774\ud130 \ubcd1\ub82c \ucc98\ub9ac \ubb38\uc11c DistributedDataParallel (DDP) is a powerful module in PyTorch that allows you to parallelize your model across multiple machines, making it perfect for large-scale deep learning applications. To use DDP, you\u2019ll need to spawn multiple processes and create a single instance of DDP per process. But how does it work? DDP uses collective communications from the torch.distributed package to synchronize gradients and buffers across all processes. This means that each process will have its own copy of the model, but they\u2019ll all work together to train the model as if it were on a single machine. To make this happen, DDP registers an autograd hook for each parameter in the model. When the backward pass is run, this hook fires and triggers gradient synchronization across all processes. This ensures that each process has the same gradients, which are then used to update the model. For more information on how DDP works and how to use it effectively, be sure to check out the DDP design note. With DDP, you can train your models faster and more efficiently than ever before! The recommended way to use DDP is to spawn one process for each model replica. The model replica can span multiple devices. DDP processes can be placed on the same machine or across machines. Note that GPU devices cannot be shared across DDP processes (i.e. one GPU for one DDP process). In this tutorial, we\u2019ll start with a basic DDP use case and then demonstrate more advanced use cases, including checkpointing models and combining DDP with model parallel. \ucc38\uace0 \uc774 \ud29c\ud1a0\ub9ac\uc5bc\uc758 \ucf54\ub4dc\ub294 8-GPU \uc11c\ubc84\uc5d0\uc11c \uc2e4\ud589\ub418\uc9c0\ub9cc \ub2e4\ub978 \ud658\uacbd\uc5d0\uc11c\ub3c4 \uc27d\uac8c \uc801\uc6a9\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. DataParallel\uacfc DistributedDataParallel \uac04\uc758 \ube44\uad50 \u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014-=== \ub0b4\uc6a9\uc5d0 \ub4e4\uc5b4\uac00\uae30\uc5d0 \uc55e\uc11c \ubcf5\uc7a1\uc131\uc774 \uc99d\uac00\ud588\uc74c\uc5d0\ub3c4 \ubd88\uad6c\ud558\uace0 DataParallel\uc5d0 DistributedDataParallel \uc0ac\uc6a9\uc744 \uace0\ub824\ud558\ub294 \uc774\uc720\ub97c \uc0dd\uac01\ud574\ubd05\uc2dc\ub2e4. \uccab\uc9f8, DataParallel\uc740 \ub2e8\uc77c \uc791\uc5c5, \uba40\ud2f0\uc4f0\ub808\ub4dc\uc774\uc9c0\ub9cc \ub2e8\uc77c \uae30\uae30\uc5d0\uc11c\ub9cc \uc791\ub3d9\ud558\ub294 \ubc18\uba74, DistributedDataParallel\uc740 \ub2e4\uc911 \uc791\uc5c5\uc774\uba70 \ub2e8\uc77c \ubc0f \ub2e4\uc911 \uae30\uae30 \ud559\uc2b5\uc744 \ubaa8\ub450 \uc9c0\uc6d0\ud569\ub2c8\ub2e4. \uc4f0\ub808\ub4dc\uac04 GIL \uacbd\ud569, \ubcf5\uc81c \ubaa8\ub378\uc758 \ubc18\ubcf5 \ub2f9 \uc0dd\uc131, \uc0b0\ub780 \uc785\ub825 \ubc0f \uc218\uc9d1 \ucd9c\ub825\uc73c\ub85c \uc778\ud55c \ucd94\uac00\uc801\uc778 \uc624\ubc84\ud5e4\ub4dc\ub85c \uc778\ud574, \ub2e8\uc77c \uae30\uae30\uc5d0\uc11c\uc870\ucc28 DataParallel\uc740 \uc77c\ubc18\uc801\uc73c\ub85c DistributedDataParallel\ubcf4\ub2e4 \ub290\ub9bd\ub2c8\ub2e4. \ubaa8\ub378\uc774 \ub108\ubb34 \ucee4\uc11c \ub2e8\uc77c GPU\uc5d0 \ub9de\uc9c0 \uc54a\uc744 \uacbd\uc6b0 model parallel\uc744 \uc0ac\uc6a9\ud558\uc5ec \uc5ec\ub7ec GPU\ub85c \ubd84\ud560\ud574\uc57c \ud55c\ub2e4\ub294 prior tutorial\uc744 \ub5a0\uc62c\ub824 \ubcf4\uc138\uc694. DistributedDataParallel\uc740 model parallel\uc5d0\uc11c \uc2e4\ud589\ub418\uc9c0\ub9cc DataParallel\uc740 \uc774\ub54c \uc2e4\ud589\ub418\uc9c0 \uc54a\uc2b5\ub2c8\ub2e4. DDP\ub97c \ubaa8\ub378 \ubcd1\ub82c \ucc98\ub9ac\uc640 \uacb0\ud569\ud558\uba74 \uac01 DDP \uc791\uc5c5\uc740 \ubaa8\ub378 \ubcd1\ub82c \ucc98\ub9ac\ub97c \uc0ac\uc6a9\ud558\uba70 \ubaa8\ub4e0 \uc791\uc5c5\uc740 \ub370\uc774\ud130 \ubcd1\ub82c \ucc98\ub9ac\ub97c \uc0ac\uc6a9\ud569\ub2c8\ub2e4. \uae30\ubcf8\uc801\uc778 \uc0ac\uc6a9\ubc95# DDP \ubaa8\ub4c8\uc744 \uc0dd\uc131\ud558\uae30 \uc804\uc5d0 \ubc18\ub4dc\uc2dc \uc6b0\uc120 \uc791\uc5c5 \uadf8\ub8f9\uc744 \uc62c\ubc14\ub974\uac8c \uc124\uc815\ud574\uc57c \ud569\ub2c8\ub2e4. \uc790\uc138\ud55c \ub0b4\uc6a9\uc740 PYTORCH\ub85c \ubd84\uc0b0 \uc5b4\ud50c\ub9ac\ucf00\uc774\uc158 \uac1c\ubc1c\ud558\uae30\uc5d0\uc11c \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. import os import sys import tempfile import torch import torch.distributed as dist import torch.nn as nn import torch.optim as optim import torch.multiprocessing as mp from torch.nn.parallel import DistributedDataParallel as DDP # \uc708\ub3c4\uc6b0 \ud50c\ub7ab\ud3fc\uc5d0\uc11c torch.distributed \ud328\ud0a4\uc9c0\ub294 # Gloo backend, FileStore \ubc0f TcpStore \ub9cc\uc744 \uc9c0\uc6d0\ud569\ub2c8\ub2e4. # FileStore\uc758 \uacbd\uc6b0, init_process_group \uc5d0\uc11c # init_method \ub9e4\uac1c\ubcc0\uc218\ub97c \ub85c\uceec \ud30c\uc77c\ub85c \uc124\uc815\ud569\ub2c8\ub2e4. # \ub2e4\uc74c \uc608\uc2dc: # init_method=\"file:///f:/libtmp/some_file\" # dist.init_process_group( # \"gloo\", # rank=rank, # init_method=init_method, # world_size=world_size) # TcpStore\uc758 \uacbd\uc6b0 \ub9ac\ub205\uc2a4\uc640 \ub3d9\uc77c\ud55c \ubc29\uc2dd\uc785\ub2c8\ub2e4. def setup(rank, world_size): os.environ[\u0027MASTER_ADDR\u0027] = \u0027localhost\u0027 os.environ[\u0027MASTER_PORT\u0027] = \u002712355\u0027 # \uc791\uc5c5 \uadf8\ub8f9 \ucd08\uae30\ud654 dist.init_process_group(\"gloo\", rank=rank, world_size=world_size) def cleanup(): dist.destroy_process_group() \uc774\uc81c DDP\ub85c \uac10\uc2f8\uc5ec\uc9c4 Toy \ubaa8\ub4c8\uc744 \uc0dd\uc131\ud558\uace0 \ub354\ubbf8 \uc785\ub825 \ub370\uc774\ud130\ub97c \uc785\ub825\ud574 \ubcf4\uaca0\uc2b5\ub2c8\ub2e4. \uc6b0\uc120 DDP\ub294 0\uc21c\uc704 \uc791\uc5c5\uc5d0\uc11c\ubd80\ud130 DDP \uc0dd\uc131\uc790\uc758 \ub2e4\ub978 \ubaa8\ub4e0 \uc791\uc5c5\ub4e4\uc5d0\uac8c \ubaa8\ub378\uc758 \uc0c1\ud0dc\ub97c \uc804\ub2ec\ud558\ubbc0\ub85c, \ub2e4\ub978 \ubaa8\ub378\uc758 \ub9e4\uac1c \ubcc0\uc218 \ucd08\uae30\uac12\ub4e4\uc5d0\uc11c \uc2dc\uc791\ud558\ub294 \ub2e4\ub978 DDP \uc791\uc5c5\ub4e4\uc5d0 \ub300\ud558\uc5ec \uac71\uc815\ud560 \ud544\uc694\uac00 \uc5c6\uc2b5\ub2c8\ub2e4. class ToyModel(nn.Module): def __init__(self): super(ToyModel, self).__init__() self.net1 = nn.Linear(10, 10) self.relu = nn.ReLU() self.net2 = nn.Linear(10, 5) def forward(self, x): return self.net2(self.relu(self.net1(x))) def demo_basic(rank, world_size): print(f\"Running basic DDP example on rank {rank}.\") setup(rank, world_size) # \ubaa8\ub378\uc744 \uc0dd\uc131\ud558\uace0 \uc21c\uc704 \uc544\uc774\ub514\uac00 \uc788\ub294 GPU\ub85c \uc804\ub2ec model = ToyModel().to(rank) ddp_model = DDP(model, device_ids=[rank]) loss_fn = nn.MSELoss() optimizer = optim.SGD(ddp_model.parameters(), lr=0.001) optimizer.zero_grad() outputs = ddp_model(torch.randn(20, 10)) labels = torch.randn(20, 5).to(rank) loss_fn(outputs, labels).backward() optimizer.step() cleanup() print(f\"Finished running basic DDP example on rank {rank}.\") def run_demo(demo_fn, world_size): mp.spawn(demo_fn, args=(world_size,), nprocs=world_size, join=True) \ubcf4\uc5ec\uc9c0\ub294 \ubc14\uc640 \uac19\uc774 DDP\ub294 \ud558\uc704 \uc218\uc900\uc758 \ubd84\uc0b0 \ucee4\ubba4\ub2c8\ucf00\uc774\uc158 \uc138\ubd80 \uc0ac\ud56d\uc744 \ud3ec\ud568\ud558\uace0 \ub85c\uceec \ubaa8\ub378\ucc98\ub7fc \uae54\ub054\ud55c API\ub97c \uc81c\uacf5\ud569\ub2c8\ub2e4. \ubcc0\ud654\ub3c4 \ub3d9\uae30\ud654 \ud1b5\uc2e0(gradient synchronization communications)\uc740 \uc5ed\uc804\ud30c \uc804\ub2ec(backward pass)\uac04 \uc218\ud589\ub418\uba70 \uc5ed\uc804\ud30c \uacc4\uc0b0(backward computation)\uacfc \uacb9\uce58\uac8c \ub429\ub2c8\ub2e4. backword()\uac00 \ubc18\ud658\ub418\uba74 param.grad\uc5d0\ub294 \ub3d9\uae30\ud654\ub41c \ubcc0\ud654\ub3c4 \ud150\uc11c(synchronized gradient tensor)\uac00 \ud3ec\ud568\ub418\uc5b4 \uc788\uc2b5\ub2c8\ub2e4. \uae30\ubcf8\uc801\uc73c\ub85c DDP\ub294 \uc791\uc5c5 \uadf8\ub8f9\uc744 \uc124\uc815\ud558\ub294\ub370 \uba87 \uc904\uc758 \ucf54\ub4dc\ub4e4\uc774 \ub354 \ud544\uc694\ud558\uc9c0\ub9cc, \ubcf4\ub2e4 \ub2e4\uc591\ud558\uac8c \uc0ac\uc6a9\ud558\ub294 \uacbd\uc6b0 \uc8fc\uc758\uac00 \ud544\uc694\ud569\ub2c8\ub2e4. \ube44\ub300\uce6d \uc791\uc5c5 \uc18d\ub3c4# DDP\uc5d0\uc11c\ub294 \uc0dd\uc131\uc790, \uc21c\uc804\ud30c(forward pass) \ubc0f \uc5ed\uc804\ud30c \uc804\ub2ec \ud638\ucd9c \uc9c0\uc810\uc774 \ubd84\uc0b0 \ub3d9\uae30\ud654 \uc9c0\uc810(distribute synchronization point)\uc785\ub2c8\ub2e4. \uc11c\ub85c \ub2e4\ub978 \uc791\uc5c5\uc774 \ub3d9\uc77c\ud55c \uc218\uc758 \ub3d9\uae30\ud654\ub97c \uc2dc\uc791\ud558\uace0 \ub3d9\uc77c\ud55c \uc21c\uc11c\ub85c \uc774\ub7ec\ud55c \ub3d9\uae30\ud654 \uc9c0\uc810\uc5d0 \ub3c4\ub2ec\ud558\uc5ec \uac01 \ub3d9\uae30\ud654 \uc9c0\uc810\uc744 \uac70\uc758 \ub3d9\uc2dc\uc5d0 \uc9c4\uc785\uc744 \uc694\uad6c\ud569\ub2c8\ub2e4. \uadf8\ub807\uc9c0 \uc54a\uc73c\uba74 \ube60\ub978 \uc791\uc5c5\uc774 \uc77c\ucc0d \ub3c4\ucc29\ud558\uace0 \ub2e4\ub978 \uc791\uc5c5\uc758 \ub300\uae30 \uc2dc\uac04\uc774 \ucd08\uacfc\ub420 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \ub530\ub77c\uc11c \uc0ac\uc6a9\uc790\ub294 \uc791\uc5c5 \uac04\uc758 \uc791\uc5c5\ub7c9\uc744 \uade0\ud615 \uc788\uac8c \ubd84\ubc30\ud560 \ud544\uc694\uac00 \uc788\uc2b5\ub2c8\ub2e4. \ub54c\ub54c\ub85c \ube44\ub300\uce6d \uc791\uc5c5(skewed processing) \uc18d\ub3c4\ub294 \ub2e4\uc74c\uacfc \uac19\uc740 \uc774\uc720\ub85c \uc778\ud558\uc5ec \ubd88\uac00\ud53c\ud558\uac8c \ubc1c\uc0dd\ud569\ub2c8\ub2e4. \uc608\ub97c \ub4e4\uc5b4, \ub124\ud2b8\uc6cc\ud06c \uc9c0\uc5f0, \ub9ac\uc18c\uc2a4 \uacbd\uc7c1(resource contentions), \uc608\uce21\ud558\uc9c0 \ubabb\ud55c \uc791\uc5c5\ub7c9 \uae09\uc99d \ub4f1\uc785\ub2c8\ub2e4. \uc774\ub7ec\ud55c \uc0c1\ud669\uc5d0\uc11c \uc2dc\uac04 \ucd08\uacfc\ub97c \ubc29\uc9c0\ud558\ub824\uba74, init_process_group\ub97c \ud638\ucd9c\ud560 \ub54c \ucda9\ubd84\ud55c timeout\uac12\uc744 \uc804\ub2ec\ud574\uc57c \ud569\ub2c8\ub2e4. \uccb4\ud06c\ud3ec\uc778\ud2b8\ub97c \uc800\uc7a5\ud558\uace0 \uc77d\uc5b4\uc624\uae30# \ud559\uc2b5 \uc911\uc5d0 torch.save\uc640 torch.load \ub85c \ubaa8\ub4c8\uc758 \uccb4\ud06c\ud3ec\uc778\ud2b8\ub97c \ub9cc\ub4e4\uace0 \uadf8 \uccb4\ud06c\ud3ec\uc778\ud2b8\ub85c\ubd80\ud130 \ubcf5\uad6c\ud558\ub294 \uac83\uc774 \uc77c\ubc18\uc801\uc785\ub2c8\ub2e4. \ub354 \uc790\uc138\ud55c \ub0b4\uc6a9\uc740 SAVING AND LOADING MODELS\ub97c \ucc38\uace0\ud558\uc138\uc694. DDP\ub97c \uc0ac\uc6a9\ud560 \ub54c, \ucd5c\uc801\uc758 \ubc29\ubc95\uc740 \ubaa8\ub378\uc744 \ud55c \uc791\uc5c5\uc5d0\ub9cc \uc800\uc7a5\ud558\uace0 \uadf8 \ubaa8\ub378\uc744 \ubaa8\ub4e0 \uc791\uc5c5\uc5d0 \uc4f0\uae30 \uacfc\ubd80\ud558(write overhead)\ub97c \uc904\uc774\uba70 \uc77d\uc5b4\uc624\ub294 \uac83\uc785\ub2c8\ub2e4. \uc774\ub294 \ubaa8\ub4e0 \uc791\uc5c5\uc774 \uac19\uc740 \ub9e4\uac1c\ubcc0\uc218\ub85c\ubd80\ud130 \uc2dc\uc791\ub418\uace0 \ubcc0\ud654\ub3c4\ub294 \uc5ed\uc804\ud30c \uc804\ub2ec\ub85c \ub3d9\uae30\ud654\ub418\ubbc0\ub85c \uc635\ud2f0\ub9c8\uc774\uc800(optimizer)\ub294 \ub9e4\uac1c\ubcc0\uc218\ub97c \ub3d9\uc77c\ud55c \uac12\uc73c\ub85c \uacc4\uc18d \uc124\uc815\ud574\uc57c \ud558\uae30 \ub54c\ubb38\uc5d0 \uc815\ud655\ud569\ub2c8\ub2e4. (\ud558\ub098\uc758 \ud504\ub85c\uc138\uc2a4\uc5d0\uc11c \uc800\uc7a5\ud558\uace0 \ub2e4\ub978 \ubaa8\ub4e0 \uacf3\uc5d0\uc11c \ubd88\ub7ec\uc624\ub294 \ub4f1) \uc774\ub7ec\ud55c \ucd5c\uc801\ud654\ub97c \uc0ac\uc6a9\ud558\ub294 \uacbd\uc6b0, \uc800\uc7a5\uc774 \uc644\ub8cc\ub418\uae30 \uc804\uc5d0 \ubd88\ub7ec\uc624\ub294 \uc5b4\ub5a0\ud55c \uc791\uc5c5\ub3c4 \uc2dc\uc791\ud558\uc9c0 \uc54a\ub3c4\ub85d \ud574\uc57c \ud569\ub2c8\ub2e4. \ub354\ubd88\uc5b4, \ubaa8\ub4c8\uc744 \uc77d\uc5b4\uc62c \ub54c \uc791\uc5c5\uc774 \ub2e4\ub978 \uae30\uae30\uc5d0 \uc811\uadfc\ud558\uc9c0 \uc54a\ub3c4\ub85d \uc801\uc808\ud55c map_location \uc778\uc790\ub97c \uc81c\uacf5\ud574\uc57c\ud569\ub2c8\ub2e4. map_location\uac12\uc774 \uc5c6\uc744 \uacbd\uc6b0, torch.load\ub294 \uba3c\uc800 \ubaa8\ub4c8\uc744 CPU\uc5d0 \uc77d\uc5b4\uc628 \ub2e4\uc74c \uac01 \ub9e4\uac1c\ubcc0\uc218\uac00 \uc800\uc7a5\ub41c \uc704\uce58\ub85c \ubcf5\uc0ac\ud558\uc5ec \ub3d9\uc77c\ud55c \uc7a5\uce58\ub97c \uc0ac\uc6a9\ud558\ub294 \ub3d9\uc77c\ud55c \uae30\uae30\uc5d0\uc11c \ubaa8\ub4e0 \uc791\uc5c5\uc744 \ubc1c\uc0dd\uc2dc\ud0b5\ub2c8\ub2e4. \ub354 \ucd94\uac00\uc801\uc778 \uc2e4\ud328 \ubcf5\uad6c\uc640 \uc5d8\ub77c\uc2a4\ud2f1(elasticity support)\uc740 TorchElastic\uc744 \ucc38\uace0\ud558\uc138\uc694. def demo_checkpoint(rank, world_size): print(f\"Running DDP checkpoint example on rank {rank}.\") setup(rank, world_size) model = ToyModel().to(rank) ddp_model = DDP(model, device_ids=[rank]) CHECKPOINT_PATH = tempfile.gettempdir() + \"/model.checkpoint\" if rank == 0: # \ubaa8\ub4e0 \uc791\uc5c5\uc740 \uac19\uc740 \ub9e4\uac1c\ubcc0\uc218\ub85c\ubd80\ud130 \uc2dc\uc791\ub41c\ub2e4\uace0 \uc0dd\uac01\ud574\uc57c \ud569\ub2c8\ub2e4. # \ubb34\uc791\uc704\uc758 \ub9e4\uac1c\ubcc0\uc218\uc640 \ubcc0\ud654\ub3c4\ub294 \uc5ed\uc804\ud30c \uc804\ub2ec\ub85c \ub3d9\uae30\ud654\ub429\ub2c8\ub2e4. # \uadf8\ub7fc\uc73c\ub85c, \ud558\ub098\uc758 \uc791\uc5c5\uc740 \ubaa8\ub378\uc744 \uc800\uc7a5\ud558\uae30\uc5d0 \ucda9\ubd84\ud569\ub2c8\ub2e4. torch.save(ddp_model.state_dict(), CHECKPOINT_PATH) # \uc791\uc5c5 0\uc774 \uc800\uc7a5\ud55c \ud6c4 \uc791\uc5c5 1\uc774 \ubaa8\ub378\uc744 \uc77d\uc5b4\uc624\ub3c4\ub85d barrier()\ub97c \uc0ac\uc6a9\ud569\ub2c8\ub2e4. dist.barrier() # configure map_location properly map_location = {\u0027cuda:%d\u0027 % 0: \u0027cuda:%d\u0027 % rank} ddp_model.load_state_dict( torch.load(CHECKPOINT_PATH, map_location=map_location, weights_only=True)) loss_fn = nn.MSELoss() optimizer = optim.SGD(ddp_model.parameters(), lr=0.001) optimizer.zero_grad() outputs = ddp_model(torch.randn(20, 10)) labels = torch.randn(20, 5).to(rank) loss_fn(outputs, labels).backward() optimizer.step() # \ud30c\uc77c\uc0ad\uc81c\ub97c \ubcf4\ud638\ud558\uae30 \uc704\ud574 \uc544\ub798\uc5d0 dist.barrier()\ub97c \uc0ac\uc6a9\ud560 \ud544\uc694\ub294 \uc5c6\uc2b5\ub2c8\ub2e4. # DDP\uc758 \uc5ed\uc804\ud30c \uc804\ub2ec \uacfc\uc815\uc5d0 \uc788\ub294 AllReduce \uc635\uc2a4(ops)\uac00 \ub3d9\uae30\ud654 \uae30\ub2a5\uc744 \uc218\ud589\ud588\uae30 \ub54c\ubb38\uc5d0 if rank == 0: os.remove(CHECKPOINT_PATH) cleanup() print(f\"Finished running DDP checkpoint example on rank {rank}.\") \ubaa8\ub378 \ubcd1\ub82c \ucc98\ub9ac\ub97c \ud65c\uc6a9\ud55c DDP# DDP\ub294 \ub2e4\uc911 GPU \ubaa8\ub378\uc5d0\uc11c\ub3c4 \uc791\ub3d9\ud569\ub2c8\ub2e4. \ub2e4\uc911 GPU \ubaa8\ub378\uc744 \ud65c\uc6a9\ud55c DDP\ub294 \ub300\uc6a9\ub7c9\uc758 \ub370\uc774\ud130\ub97c \uac00\uc9c4 \ub300\uc6a9\ub7c9 \ubaa8\ub378\uc744 \ud559\uc2b5\uc2dc\ud0ac \ub54c \ud2b9\ud788 \uc720\uc6a9\ud569\ub2c8\ub2e4. class ToyMpModel(nn.Module): def __init__(self, dev0, dev1): super(ToyMpModel, self).__init__() self.dev0 = dev0 self.dev1 = dev1 self.net1 = torch.nn.Linear(10, 10).to(dev0) self.relu = torch.nn.ReLU() self.net2 = torch.nn.Linear(10, 5).to(dev1) def forward(self, x): x = x.to(self.dev0) x = self.relu(self.net1(x)) x = x.to(self.dev1) return self.net2(x) \ub2e4\uc911 GPU \ubaa8\ub378\uc744 DDP\ub85c \uc804\ub2ec\ud560 \ub54c, device_ids\uc640 output_device\ub97c \uc124\uc815\ud558\uc9c0 \uc54a\uc544\uc57c \ud569\ub2c8\ub2e4. \uc785\ub825 \ubc0f \ucd9c\ub825 \ub370\uc774\ud130\ub294 \uc5b4\ud50c\ub9ac\ucf00\uc774\uc158 \ub610\ub294 \ubaa8\ub378 forward()\uc5d0 \uc758\ud574 \uc801\uc808\ud55c \uc7a5\uce58\uc5d0 \ubc30\uce58\ub429\ub2c8\ub2e4. def demo_model_parallel(rank, world_size): print(f\"Running DDP with model parallel example on rank {rank}.\") setup(rank, world_size) # \uc791\uc5c5\uc744 \uc704\ud55c mp_model \ubc0f \uc7a5\uce58 \uc124\uc815 dev0 = rank * 2 dev1 = rank * 2 + 1 mp_model = ToyMpModel(dev0, dev1) ddp_mp_model = DDP(mp_model) loss_fn = nn.MSELoss() optimizer = optim.SGD(ddp_mp_model.parameters(), lr=0.001) optimizer.zero_grad() # \ucd9c\ub825\uac12\uc740 dev1\uc5d0 \uc800\uc7a5 outputs = ddp_mp_model(torch.randn(20, 10)) labels = torch.randn(20, 5).to(dev1) loss_fn(outputs, labels).backward() optimizer.step() cleanup() print(f\"Finished running DDP with model parallel example on rank {rank}.\") if __name__ == \"__main__\": n_gpus = torch.cuda.device_count() assert n_gpus \u003e= 2, f\"Requires at least 2 GPUs to run, but got {n_gpus}\" world_size = n_gpus run_demo(demo_basic, world_size) run_demo(demo_checkpoint, world_size) run_demo(demo_model_parallel, world_size) Initialize DDP with torch.distributed.run/torchrun# We can leverage PyTorch Elastic to simplify the DDP code and initialize the job more easily. Let\u2019s still use the Toymodel example and create a file named elastic_ddp.py. import os import torch import torch.distributed as dist import torch.nn as nn import torch.optim as optim from torch.nn.parallel import DistributedDataParallel as DDP class ToyModel(nn.Module): def __init__(self): super(ToyModel, self).__init__() self.net1 = nn.Linear(10, 10) self.relu = nn.ReLU() self.net2 = nn.Linear(10, 5) def forward(self, x): return self.net2(self.relu(self.net1(x))) def demo_basic(): torch.cuda.set_device(int(os.environ[\"LOCAL_RANK\"])) dist.init_process_group(\"nccl\") rank = dist.get_rank() print(f\"Start running basic DDP example on rank {rank}.\") # create model and move it to GPU with id rank device_id = rank % torch.cuda.device_count() model = ToyModel().to(device_id) ddp_model = DDP(model, device_ids=[device_id]) loss_fn = nn.MSELoss() optimizer = optim.SGD(ddp_model.parameters(), lr=0.001) optimizer.zero_grad() outputs = ddp_model(torch.randn(20, 10)) labels = torch.randn(20, 5).to(device_id) loss_fn(outputs, labels).backward() optimizer.step() dist.destroy_process_group() print(f\"Finished running basic DDP example on rank {rank}.\") if __name__ == \"__main__\": demo_basic() In the example above, we are running the DDP script on two hosts and we run with 8 processes on each host. That is, we are running this job on 16 GPUs. Note that $MASTER_ADDR must be the same across all nodes. Here torchrun will launch 8 processes and invoke elastic_ddp.py on each process on the node it is launched on, but user also needs to apply cluster management tools like slurm to actually run this command on 2 nodes. For example, on a SLURM enabled cluster, we can write a script to run the command above and set MASTER_ADDR as: export MASTER_ADDR=$(scontrol show hostname ${SLURM_NODELIST} | head -n 1) Then we can just run this script using the SLURM command: srun --nodes=2 ./torchrun_script.sh. This is just an example; you can choose your own cluster scheduling tools to initiate the torchrun job. For more information about Elastic run, please see the quick start document.",
       "author": {
         "@type": "Organization",
         "name": "PyTorch Contributors",
         "url": "https://pytorch.org"
       },
       "image": "../_static/img/pytorch_seo.png",
       "mainEntityOfPage": {
         "@type": "WebPage",
         "@id": "/intermediate/ddp_tutorial.html"
       },
       "datePublished": "2023-01-01T00:00:00Z",
       "dateModified": "2023-01-01T00:00:00Z"
     }