René's URL Explorer Experiment

    {
       "@context": "https://schema.org",
       "@type": "Article",
       "name": "\ud150\uc11c\ubcf4\ub4dc\ub97c \uc774\uc6a9\ud55c \ud30c\uc774\ud1a0\uce58 \ud504\ub85c\ud30c\uc77c\ub7ec",
       "headline": "\ud150\uc11c\ubcf4\ub4dc\ub97c \uc774\uc6a9\ud55c \ud30c\uc774\ud1a0\uce58 \ud504\ub85c\ud30c\uc77c\ub7ec",
       "description": "PyTorch Documentation. Explore PyTorch, an open-source machine learning library that accelerates the path from research prototyping to production deployment. Discover tutorials, API references, and guides to help you build and deploy deep learning models efficiently.",
       "url": "/intermediate/tensorboard_profiler_tutorial.html",
       "articleBody": "\ucc38\uace0 Go to the end to download the full example code. \ud150\uc11c\ubcf4\ub4dc\ub97c \uc774\uc6a9\ud55c \ud30c\uc774\ud1a0\uce58 \ud504\ub85c\ud30c\uc77c\ub7ec# \ubc88\uc5ed: \uc190\ub3d9\uc6b0 \uc774 \ud29c\ud1a0\ub9ac\uc5bc\uc5d0\uc11c\ub294 \ud30c\uc774\ud1a0\uce58(PyTorch) \ud504\ub85c\ud30c\uc77c\ub7ec(profiler)\uc640 \ud568\uaed8 \ud150\uc11c\ubcf4\ub4dc(TensorBoard) \ud50c\ub7ec\uadf8\uc778(plugin)\uc744 \uc0ac\uc6a9\ud558\uc5ec \ubaa8\ub378\uc758 \uc131\ub2a5 \ubcd1\ubaa9 \ud604\uc0c1\uc744 \ud0d0\uc9c0\ud558\ub294 \ubc29\ubc95\uc744 \ubcf4\uc5ec \uc90d\ub2c8\ub2e4. \uacbd\uace0 TensorBoard\uc640 PyTorch \ud504\ub85c\ud30c\uc77c\ub7ec\uc758 \ud1b5\ud569\uc740 \uc774\uc81c \ub354 \uc774\uc0c1 \uc0ac\uc6a9\ub418\uc9c0 \uc54a\uc2b5\ub2c8\ub2e4. \ub300\uc2e0 Perfetto \ub610\ub294 Chrome \ud2b8\ub808\uc774\uc2a4\ub97c \uc0ac\uc6a9\ud558\uc5ec trace.json \ud30c\uc77c\uc744 \ubcfc \uc218 \uc788\uc2b5\ub2c8\ub2e4. \ud2b8\ub808\uc774\uc2a4 \uc0dd\uc131 \ud6c4, trace.json \ud30c\uc77c\uc744 Perfetto UI \ub610\ub294 chrome://tracing \uc5d0 \ub4dc\ub798\uadf8\ud558\uc5ec \ud504\ub85c\ud30c\uc77c\uc744 \uc2dc\uac01\ud654\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc18c\uac1c# \ud30c\uc774\ud1a0\uce58(PyTorch) 1.8\ubd80\ud130 GPU\uc5d0\uc11c CUDA \ucee4\ub110(kernel) \uc2e4\ud589 \ubfd0\ub9cc \uc544\ub2c8\ub77c CPU \uc791\uc5c5\uc744 \uae30\ub85d\ud560 \uc218 \uc788\ub294 \uc5c5\ub370\uc774\ud2b8\ub41c \ud504\ub85c\ud30c\uc77c\ub7ec API\uac00 \ud3ec\ud568\ub418\uc5b4 \uc788\uc2b5\ub2c8\ub2e4. \ud504\ub85c\ud30c\uc77c\ub7ec\ub294 \ud150\uc11c\ubcf4\ub4dc \ud50c\ub7ec\uadf8\uc778\uc5d0\uc11c \uc774\ub7f0 \uc815\ubcf4\ub97c \uc2dc\uac01\ud654\ud558\uace0 \uc131\ub2a5 \ubcd1\ubaa9 \ud604\uc0c1\uc5d0 \ub300\ud55c \ubd84\uc11d\uc744 \uc81c\uacf5\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc774 \ud29c\ud1a0\ub9ac\uc5bc\uc5d0\uc11c\ub294 \uac04\ub2e8\ud55c Resnet \ubaa8\ub378\uc744 \uc0ac\uc6a9\ud558\uc5ec \ud150\uc11c\ubcf4\ub4dc \ud50c\ub7ec\uadf8\uc778\uc744 \ud65c\uc6a9\ud55c \ubaa8\ub378 \uc131\ub2a5 \ubd84\uc11d \ubc29\ubc95\uc744 \ubcf4\uc5ec\ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4. \uc900\ube44# \uc544\ub798 \uba85\ub839\uc5b4\ub97c \uc2e4\ud589\ud558\uc5ec ``torch``\uc640 ``torchvision``\uc744 \uc124\uce58\ud569\ub2c8\ub2e4: pip install torch torchvision \uacfc\uc815# \ub370\uc774\ud130 \ubc0f \ubaa8\ub378 \uc900\ube44 \ud504\ub85c\ud30c\uc77c\ub7ec\ub97c \uc0ac\uc6a9\ud558\uc5ec \uc2e4\ud589 \uc774\ubca4\ud2b8(execution events) \uae30\ub85d \ud504\ub85c\ud30c\uc77c\ub7ec \uc2e4\ud589 \ud150\uc11c\ubcf4\ub4dc\ub97c \uc0ac\uc6a9\ud558\uc5ec \uacb0\uacfc \ud655\uc778 \ubc0f \ubaa8\ub378 \uc131\ub2a5 \ubd84\uc11d \ud504\ub85c\ud30c\uc77c\ub7ec\uc758 \ub3c4\uc6c0\uc73c\ub85c \uc131\ub2a5 \uac1c\uc120 \ub2e4\ub978 \uace0\uae09 \uae30\ub2a5\uc73c\ub85c \uc131\ub2a5 \ubd84\uc11d \ucd94\uac00 \uc5f0\uc2b5: AMD GPU\uc5d0\uc11c PyTorch \ud504\ub85c\ud30c\uc77c\ub9c1 1. \ub370\uc774\ud130 \ubc0f \ubaa8\ub378 \uc900\ube44# \uba3c\uc800 \ud544\uc694\ud55c \ub77c\uc774\ube0c\ub7ec\ub9ac\ub97c \ubaa8\ub450 \ubd88\ub7ec\uc635\ub2c8\ub2e4: import torch import torch.nn import torch.optim import torch.profiler import torch.utils.data import torchvision.datasets import torchvision.models import torchvision.transforms as T \uc774\ud6c4 \uc785\ub825 \ub370\uc774\ud130\ub97c \uc900\ube44\ud569\ub2c8\ub2e4. \uc774 \ud29c\ud1a0\ub9ac\uc5bc\uc758 \uacbd\uc6b0 CIFAR10 \ub370\uc774\ud130\uc14b\uc744 \uc0ac\uc6a9\ud569\ub2c8\ub2e4. \uc6d0\ud558\ub294 \ud615\uc2dd\uc73c\ub85c \ubcc0\ud658\ud558\uace0 ``DataLoader``\ub97c \uc0ac\uc6a9\ud558\uc5ec \uac01 \ubc30\uce58(batch)\ub97c \ub85c\ub4dc\ud569\ub2c8\ub2e4. transform = T.Compose( [T.Resize(224), T.ToTensor(), T.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]) train_set = torchvision.datasets.CIFAR10(root=\u0027./data\u0027, train=True, download=True, transform=transform) train_loader = torch.utils.data.DataLoader(train_set, batch_size=32, shuffle=True) 0%| | 0.00/170M [00:00\u003c?, ?B/s] 0%| | 32.8k/170M [00:00\u003c14:23, 197kB/s] 0%| | 65.5k/170M [00:00\u003c14:23, 197kB/s] 0%| | 98.3k/170M [00:00\u003c14:18, 199kB/s] 0%| | 229k/170M [00:00\u003c06:33, 432kB/s] 0%| | 459k/170M [00:00\u003c03:39, 775kB/s] 1%| | 918k/170M [00:00\u003c01:57, 1.45MB/s] 1%| | 1.84M/170M [00:01\u003c01:00, 2.78MB/s] 2%|\u258f | 3.70M/170M [00:01\u003c00:30, 5.47MB/s] 4%|\u258d | 7.14M/170M [00:01\u003c00:16, 9.93MB/s] 6%|\u258c | 10.0M/170M [00:01\u003c00:13, 12.1MB/s] 8%|\u258a | 13.1M/170M [00:01\u003c00:11, 13.8MB/s] 9%|\u2589 | 16.0M/170M [00:02\u003c00:10, 14.9MB/s] 11%|\u2588 | 18.8M/170M [00:02\u003c00:09, 15.5MB/s] 13%|\u2588\u258e | 21.8M/170M [00:02\u003c00:09, 16.1MB/s] 14%|\u2588\u258d | 24.7M/170M [00:02\u003c00:08, 16.2MB/s] 16%|\u2588\u258b | 27.8M/170M [00:02\u003c00:08, 16.7MB/s] 18%|\u2588\u258a | 30.7M/170M [00:02\u003c00:08, 16.9MB/s] 20%|\u2588\u2589 | 33.7M/170M [00:03\u003c00:07, 17.1MB/s] 22%|\u2588\u2588\u258f | 36.7M/170M [00:03\u003c00:07, 17.2MB/s] 23%|\u2588\u2588\u258e | 39.6M/170M [00:03\u003c00:07, 17.2MB/s] 25%|\u2588\u2588\u258d | 42.4M/170M [00:03\u003c00:07, 17.1MB/s] 27%|\u2588\u2588\u258b | 45.3M/170M [00:03\u003c00:07, 16.5MB/s] 28%|\u2588\u2588\u258a | 48.3M/170M [00:03\u003c00:07, 16.7MB/s] 30%|\u2588\u2588\u2588 | 51.2M/170M [00:04\u003c00:07, 16.8MB/s] 32%|\u2588\u2588\u2588\u258f | 54.1M/170M [00:04\u003c00:06, 16.9MB/s] 33%|\u2588\u2588\u2588\u258e | 57.0M/170M [00:04\u003c00:06, 17.0MB/s] 35%|\u2588\u2588\u2588\u258c | 60.1M/170M [00:04\u003c00:06, 17.2MB/s] 37%|\u2588\u2588\u2588\u258b | 63.0M/170M [00:04\u003c00:06, 17.1MB/s] 39%|\u2588\u2588\u2588\u2589 | 66.1M/170M [00:04\u003c00:06, 17.2MB/s] 41%|\u2588\u2588\u2588\u2588 | 69.2M/170M [00:05\u003c00:05, 17.3MB/s] 42%|\u2588\u2588\u2588\u2588\u258f | 72.2M/170M [00:05\u003c00:05, 17.3MB/s] 44%|\u2588\u2588\u2588\u2588\u258d | 75.2M/170M [00:05\u003c00:05, 17.3MB/s] 46%|\u2588\u2588\u2588\u2588\u258c | 78.0M/170M [00:05\u003c00:05, 17.1MB/s] 47%|\u2588\u2588\u2588\u2588\u258b | 81.0M/170M [00:05\u003c00:05, 17.2MB/s] 49%|\u2588\u2588\u2588\u2588\u2589 | 84.0M/170M [00:05\u003c00:05, 17.2MB/s] 51%|\u2588\u2588\u2588\u2588\u2588 | 87.0M/170M [00:06\u003c00:04, 17.2MB/s] 53%|\u2588\u2588\u2588\u2588\u2588\u258e | 89.9M/170M [00:06\u003c00:04, 17.2MB/s] 54%|\u2588\u2588\u2588\u2588\u2588\u258d | 92.8M/170M [00:06\u003c00:04, 17.2MB/s] 56%|\u2588\u2588\u2588\u2588\u2588\u258c | 95.7M/170M [00:06\u003c00:04, 17.1MB/s] 58%|\u2588\u2588\u2588\u2588\u2588\u258a | 98.5M/170M [00:06\u003c00:04, 17.0MB/s] 59%|\u2588\u2588\u2588\u2588\u2588\u2589 | 101M/170M [00:07\u003c00:04, 17.0MB/s] 61%|\u2588\u2588\u2588\u2588\u2588\u2588 | 104M/170M [00:07\u003c00:03, 17.1MB/s] 63%|\u2588\u2588\u2588\u2588\u2588\u2588\u258e | 107M/170M [00:07\u003c00:03, 17.0MB/s] 65%|\u2588\u2588\u2588\u2588\u2588\u2588\u258d | 110M/170M [00:07\u003c00:03, 17.2MB/s] 66%|\u2588\u2588\u2588\u2588\u2588\u2588\u258b | 113M/170M [00:07\u003c00:03, 17.2MB/s] 68%|\u2588\u2588\u2588\u2588\u2588\u2588\u258a | 116M/170M [00:07\u003c00:03, 17.1MB/s] 70%|\u2588\u2588\u2588\u2588\u2588\u2588\u2589 | 119M/170M [00:08\u003c00:02, 17.1MB/s] 72%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f | 122M/170M [00:08\u003c00:02, 17.1MB/s] 73%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e | 125M/170M [00:08\u003c00:02, 17.2MB/s] 75%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258c | 128M/170M [00:08\u003c00:02, 17.2MB/s] 77%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258b | 131M/170M [00:08\u003c00:02, 17.0MB/s] 78%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a | 134M/170M [00:08\u003c00:02, 17.0MB/s] 80%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588 | 137M/170M [00:09\u003c00:01, 17.0MB/s] 82%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f | 140M/170M [00:09\u003c00:01, 17.0MB/s] 84%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e | 143M/170M [00:09\u003c00:01, 17.1MB/s] 85%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258c | 146M/170M [00:09\u003c00:01, 17.3MB/s] 87%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258b | 148M/170M [00:09\u003c00:01, 17.1MB/s] 89%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2589 | 151M/170M [00:09\u003c00:01, 17.2MB/s] 91%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588 | 154M/170M [00:10\u003c00:00, 17.2MB/s] 92%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f| 157M/170M [00:10\u003c00:00, 17.1MB/s] 94%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258d| 160M/170M [00:10\u003c00:00, 17.0MB/s] 96%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258c| 163M/170M [00:10\u003c00:00, 16.9MB/s] 97%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258b| 166M/170M [00:10\u003c00:00, 16.9MB/s] 99%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2589| 169M/170M [00:10\u003c00:00, 17.0MB/s] 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 170M/170M [00:10\u003c00:00, 15.5MB/s] \uadf8\ub7f0 \ub2e4\uc74c Resnet \ubaa8\ub378, \uc190\uc2e4 \ud568\uc218 \ubc0f \uc635\ud2f0\ub9c8\uc774\uc800 \uac1d\uccb4\ub97c \uc0dd\uc131\ud569\ub2c8\ub2e4. GPU\uc5d0\uc11c \uc2e4\ud589\ud558\uae30 \uc704\ud574 \ubaa8\ub378 \ubc0f \uc190\uc2e4\uc744 GPU \uc7a5\uce58\ub85c \uc774\ub3d9\ud569\ub2c8\ub2e4. device = torch.device(\"cuda:0\") model = torchvision.models.resnet18(weights=\u0027IMAGENET1K_V1\u0027).cuda(device) criterion = torch.nn.CrossEntropyLoss().cuda(device) optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9) model.train() ResNet( (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): ReLU(inplace=True) (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False) (layer1): Sequential( (0): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): ReLU(inplace=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (1): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): ReLU(inplace=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (layer2): Sequential( (0): BasicBlock( (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): ReLU(inplace=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (downsample): Sequential( (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): ReLU(inplace=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (layer3): Sequential( (0): BasicBlock( (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): ReLU(inplace=True) (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (downsample): Sequential( (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): ReLU(inplace=True) (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (layer4): Sequential( (0): BasicBlock( (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): ReLU(inplace=True) (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (downsample): Sequential( (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): ReLU(inplace=True) (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (avgpool): AdaptiveAvgPool2d(output_size=(1, 1)) (fc): Linear(in_features=512, out_features=1000, bias=True) ) \uac01 \uc785\ub825 \ub370\uc774\ud130 \ubc30\uce58\uc5d0 \ub300\ud55c \ud559\uc2b5 \ub2e8\uacc4\ub97c \uc815\uc758\ud569\ub2c8\ub2e4. def train(data): inputs, labels = data[0].to(device=device), data[1].to(device=device) outputs = model(inputs) loss = criterion(outputs, labels) optimizer.zero_grad() loss.backward() optimizer.step() 2. \ud504\ub85c\ud30c\uc77c\ub7ec\ub97c \uc0ac\uc6a9\ud558\uc5ec \uc2e4\ud589 \uc774\ubca4\ud2b8 \uae30\ub85d# \ud504\ub85c\ud30c\uc77c\ub7ec\ub294 \ucee8\ud14d\uc2a4\ud2b8(context) \uad00\ub9ac\uc790\ub97c \ud1b5\ud574 \ud65c\uc131\ud654\ub418\uace0 \uba87 \uac00\uc9c0 \ub9e4\uac1c\ubcc0\uc218\ub97c \uc0ac\uc6a9\ud560 \uc218 \uc788\uc73c\uba70, \uac00\uc7a5 \uc720\uc6a9\ud55c \uac83\uc740 \uc544\ub798\uc640 \uac19\uc2b5\ub2c8\ub2e4: schedule - step (int)\uc744 \ub2e8\uc77c \ub9e4\uac1c\ubcc0\uc218\ub85c \ubc1b\uc544\ub4e4\uc774\uace0, \uac01 \ub2e8\uacc4\uc5d0\uc11c \uc218\ud589\ud560 \ud504\ub85c\ud30c\uc77c\ub7ec \uc791\uc5c5\uc744 \ubc18\ud658\ud558\ub294 \ud638\ucd9c \uac00\ub2a5\ud55c \ud568\uc218\uc785\ub2c8\ub2e4. \uc774 \uc608\uc2dc\uc5d0\uc11c\ub294 wait=1, warmup=1, active=3, repeat=1 \ub85c \uc124\uc815\ub418\uc5b4 \uc788\uc73c\uba70, \ud504\ub85c\ud30c\uc77c\ub7ec\ub294 \uccab \ubc88\uc9f8 \ub2e8\uacc4/\ubc18\ubcf5(step/iteration)\uc744 \uac74\ub108\ub701\ub2c8\ub2e4. \ub450 \ubc88\uc9f8\ubd80\ud130 \uc6cc\ubc0d\uc5c5(warming up)\uc744 \uc2dc\uc791\ud558\uba74, \ub2e4\uc74c \uc138 \ubc88\uc758 \ubc18\ubcf5\uc744 \uae30\ub85d\ud558\uace0, \uadf8 \ud6c4 \ucd94\uc801(trace)\uc744 \uc0ac\uc6a9\ud560 \uc218 \uc788\uac8c \ub418\uace0 on_trace_ready (\uc124\uc815\ub41c \uacbd\uc6b0)\uac00 \ud638\ucd9c\ub429\ub2c8\ub2e4. \uc804\uccb4\uc801\uc73c\ub85c \uc774 \uc8fc\uae30\uac00 \ud55c \ubc88 \ubc18\ubcf5\ub429\ub2c8\ub2e4. \ud150\uc11c\ubcf4\ub4dc \ud50c\ub7ec\uadf8\uc778\uc5d0\uc11c \uac01 \uc8fc\uae30\ub294 \u201cspan\u201d\uc774\ub77c\uace0 \ud569\ub2c8\ub2e4. wait \ub2e8\uacc4\uc778 \ub3d9\uc548 \ud504\ub85c\ud30c\uc77c\ub7ec\ub294 \ube44\ud65c\uc131\ud654\ub429\ub2c8\ub2e4. warmup \ub2e8\uacc4\uc778 \ub3d9\uc548\uc5d4 \ud504\ub85c\ud30c\uc77c\ub7ec\uac00 \ucd94\uc801(tracing)\uc744 \uc2dc\uc791\ud558\uc9c0\ub9cc \uacb0\uacfc\ub294 \ubb34\uc2dc\ub429\ub2c8\ub2e4. \uc774\ub294 \ud504\ub85c\ud30c\uc77c\ub9c1 \uacfc\ubd80\ud558(overhead)\ub97c \uc904\uc774\uae30 \uc704\ud568\uc785\ub2c8\ub2e4. \ud504\ub85c\ud30c\uc77c\ub9c1\uc744 \uc2dc\uc791\ud560 \ub54c \uacfc\ubd80\ud558\ub294 \ud06c\uace0 \ud504\ub85c\ud30c\uc77c\ub9c1 \uacb0\uacfc\uc5d0 \uc65c\uace1\uc744 \uac00\uc838\uc624\uae30 \uc27d\uc2b5\ub2c8\ub2e4. active \ub2e8\uacc4\uc5d0\uc120 \ud504\ub85c\ud30c\uc77c\ub7ec\uac00 \uc791\ub3d9\ud558\uba70 \uc774\ubca4\ud2b8\ub97c \uae30\ub85d\ud569\ub2c8\ub2e4. on_trace_ready - \uac01 \uc8fc\uae30 \ub9c8\uc9c0\ub9c9\uc5d0 \ud638\ucd9c\ub418\ub294 \ud568\uc218\uc785\ub2c8\ub2e4; \uc774 \uc608\uc2dc\uc5d0\uc11c\ub294 torch.profiler.tensorboard_trace_handler``\ub97c \uc0ac\uc6a9\ud558\uc5ec \ud150\uc11c\ubcf4\ub4dc\uc758 \uacb0\uacfc \ud30c\uc77c\uc744 \uc0dd\uc131\ud569\ub2c8\ub2e4. \ud504\ub85c\ud30c\uc77c\ub9c1 \ud6c4 \uacb0\uacfc \ud30c\uc77c\uc740 ``./log/resnet18 \ub514\ub809\ud1a0\ub9ac\uc5d0 \uc800\uc7a5\ub429\ub2c8\ub2e4. \ud150\uc11c\ubcf4\ub4dc\uc5d0\uc11c \ud504\ub85c\ud30c\uc77c(profile)\uc744 \ubd84\uc11d\ud558\ub824\uba74 \uc774 \ub514\ub809\ud1a0\ub9ac\ub97c logdir \ub9e4\uac1c\ubcc0\uc218\ub85c \uc9c0\uc815\ud574\uc57c \ud569\ub2c8\ub2e4. record_shapes - \uc5f0\uc0b0\uc790 \uc785\ub825\uc758 shape\uc744 \uae30\ub85d\ud560\uc9c0 \uc5ec\ubd80\ub97c \ub098\ud0c0\ub0c5\ub2c8\ub2e4. profile_memory - Track tensor memory \ud560\ub2f9/\ud560\ub2f9 \ud574\uc81c \uc5ec\ubd80\ub97c \ub098\ud0c0\ub0c5\ub2c8\ub2e4. \uc8fc\uc758, 1.10 \uc774\uc804 \ubc84\uc804\uc758 \ud30c\uc774\ud1a0\uce58\ub97c \uc0ac\uc6a9\ud558\ub294 \uacbd\uc6b0 \ud504\ub85c\ud30c\uc77c\ub9c1 \uc2dc\uac04\uc774 \uae38\ub2e4\uba74 \uc774 \uae30\ub2a5\uc744 \ube44\ud65c\uc131\ud654\ud558\uac70\ub098 \uc0c8 \ubc84\uc804\uc73c\ub85c \uc5c5\uadf8\ub808\uc774\ub4dc\ud574 \uc8fc\uc138\uc694. with_stack - ops\uc5d0 \ub300\ud55c \uc18c\uc2a4 \uc815\ubcf4(\ud30c\uc77c \ubc0f \ub77c\uc778 \ubc88\ud638)\ub97c \uae30\ub85d \uc5ec\ubd80\ub97c \ub098\ud0c0\ub0c5\ub2c8\ub2e4. \ub9cc\uc57d VS Code\uc5d0\uc11c \ud150\uc11c\ubcf4\ub4dc\ub97c \uc2e4\ud589\ud558\ub294 \uacbd\uc6b0 (\ucc38\uace0), \uc2a4\ud0dd \ud504\ub808\uc784(stack frame)\uc744 \ud074\ub9ad\ud558\uba74 \ud2b9\uc815 \ucf54\ub4dc \ub77c\uc778\uc73c\ub85c \uc774\ub3d9\ud569\ub2c8\ub2e4. with torch.profiler.profile( schedule=torch.profiler.schedule(wait=1, warmup=1, active=3, repeat=1), on_trace_ready=torch.profiler.tensorboard_trace_handler(\u0027./log/resnet18\u0027), record_shapes=True, profile_memory=True, with_stack=True ) as prof: for step, batch_data in enumerate(train_loader): prof.step() # \uac01 \ub2e8\uacc4\uc5d0\uc11c \ud638\ucd9c\ud558\uc5ec \ud504\ub85c\ud30c\uc77c\ub7ec\uc5d0\uac8c \ub2e8\uacc4\uc758 \uacbd\uacc4\ub97c \uc54c\ub824\uc57c \ud569\ub2c8\ub2e4. if step \u003e= 1 + 1 + 3: break train(batch_data) \ub610\ud55c, \ub2e4\uc74c\uc758 non-context \uad00\ub9ac\uc790(manager)\ub294 \uc2dc\uc791(start)/\uc815\uc9c0(stop) \uae30\ub2a5\ub3c4 \uc9c0\uc6d0\ub429\ub2c8\ub2e4. prof = torch.profiler.profile( schedule=torch.profiler.schedule(wait=1, warmup=1, active=3, repeat=1), on_trace_ready=torch.profiler.tensorboard_trace_handler(\u0027./log/resnet18\u0027), record_shapes=True, with_stack=True) prof.start() for step, batch_data in enumerate(train_loader): prof.step() if step \u003e= 1 + 1 + 3: break train(batch_data) prof.stop() 3. \ud504\ub85c\ud30c\uc77c\ub7ec \uc2e4\ud589# \uc704 \ucf54\ub4dc\ub97c \uc2e4\ud589\ud569\ub2c8\ub2e4. \ud504\ub85c\ud30c\uc77c\ub9c1 \uacb0\uacfc\ub294 ./log/resnet18 \ub514\ub809\ud1a0\ub9ac\uc5d0 \uc800\uc7a5\ub429\ub2c8\ub2e4. 4. \ud150\uc11c\ubcf4\ub4dc\ub97c \uc0ac\uc6a9\ud558\uc5ec \uacb0\uacfc \ud655\uc778 \ubc0f \ubaa8\ub378 \uc131\ub2a5 \ubd84\uc11d# \ucc38\uace0 \ud150\uc11c\ubcf4\ub4dc \ud50c\ub7ec\uadf8\uc778(Tensorboard Plugin) \uc9c0\uc6d0\uc774 \uc911\ub2e8\ub418\uc5c8\uc73c\ubbc0\ub85c, \uc544\ub798 \uae30\ub2a5\ub4e4 \uc911 \uc77c\ubd80\ub294 \uc774\uc804\ucc98\ub7fc \ub3d9\uc791\ud558\uc9c0 \uc54a\uc744 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc774\uc5d0 \ub300\ud55c \ub300\uc548\uc73c\ub85c HTA \ub97c \uc0ac\uc6a9\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \ud30c\uc774\ud1a0\uce58 \ud504\ub85c\ud30c\uc77c\ub7ec \ud150\uc11c\ubcf4\ub4dc \ud50c\ub7ec\uadf8\uc778\uc744 \uc124\uce58\ud569\ub2c8\ub2e4. pip install torch_tb_profiler \ud150\uc11c\ubcf4\ub4dc\ub97c \uc2e4\ud589\ud569\ub2c8\ub2e4. tensorboard --logdir=./log \uad6c\uae00 \ud06c\ub86c(Google Chrome) \ube0c\ub77c\uc6b0\uc800 \ub610\ub294 \ub9c8\uc774\ud06c\ub85c\uc18c\ud504\ud2b8 \uc5e3\uc9c0(Microsoft Edge) \ube0c\ub77c\uc6b0\uc800\uc5d0\uc11c \ud150\uc11c\ubcf4\ub4dc \ud504\ub85c\ud30c\uc77c(profile) URL\uc5d0 \uc811\uc18d\ud569\ub2c8\ub2e4. (Safari \ube0c\ub77c\uc6b0\uc800\ub294 \uc9c0\uc6d0\ud558\uc9c0 \uc54a\uc2b5\ub2c8\ub2e4.) http://localhost:6006/#pytorch_profiler \uc544\ub798\uc640 \uac19\uc774 \ud504\ub85c\ud30c\uc77c\ub7ec \ud50c\ub7ec\uadf8\uc778 \ud398\uc774\uc9c0\ub97c \ubcfc \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uac1c\uc694(Overview) \uac1c\uc694 \ud398\uc774\uc9c0\uc5d0\ub294 \ubaa8\ub378 \uc131\ub2a5\uc5d0 \ub300\ud55c \ub300\ub7b5\uc801\uc778 \uc694\uc57d\uc774 \ud45c\uc2dc\ub429\ub2c8\ub2e4. \u201cGPU \uc694\uc57d(GPU Summary)\u201d \ud328\ub110\uc5d0\ub294 GPU \uad6c\uc131, GPU \uc0ac\uc6a9\ub7c9 \ubc0f Tensor \ucf54\uc5b4 \uc0ac\uc6a9\ub7c9\uc774 \ud45c\uc2dc\ub429\ub2c8\ub2e4. \uc774 \uc608\uc81c\uc5d0\uc11c\ub294 GPU \uc0ac\uc6a9\ub7c9\uc774 \ub0ae\uc2b5\ub2c8\ub2e4. \uc774\ub7ec\ud55c \uce21\uc815 \uc9c0\ud45c(metrics)\uc5d0 \ub300\ud55c \uc790\uc138\ud55c \ub0b4\uc6a9\uc740 \uc5ec\uae30 \uc5d0\uc11c \ud655\uc778\ud574\uc8fc\uc138\uc694. \u201c\ub2e8\uacc4 \uc2dc\uac04 \uc138\ubd84\ud654(Step Time Breakdown)\u201d\ub294 \uac01 \ub2e8\uacc4\uc5d0\uc11c \uc218\ud589\ub41c \uc2dc\uac04\uc758 \ubd84\ud3ec\ub97c \ubcf4\uc5ec\uc90d\ub2c8\ub2e4. \uc774 \uc608\uc81c\uc5d0\uc11c\ub294 DataLoader \uacfc\ubd80\ud558\uac00 \uc0c1\ub2f9\ud55c \uac83\uc744 \ubcfc \uc218 \uc788\uc2b5\ub2c8\ub2e4. \ud558\ub2e8\uc758 \u201c\uc131\ub2a5 \uad8c\uc7a5\uc0ac\ud56d(Performance Recommendation)\u201d\uc740 \ud504\ub85c\ud30c\uc77c\ub9c1 \ub370\uc774\ud130\ub97c \uc0ac\uc6a9\ud558\uc5ec \ubc1c\uc0dd \uac00\ub2a5\ud55c \ubcd1\ubaa9 \ud604\uc0c1\uc744 \uc790\ub3d9\uc73c\ub85c \uac15\uc870\ud558\uace0, \uc2e4\ud589 \uac00\ub2a5\ud55c \ucd5c\uc801\ud654 \uc81c\uc548\uc744 \uc81c\uacf5\ud569\ub2c8\ub2e4. \uc67c\ucabd \u201c\ubcf4\uae30(Views)\u201d \ub4dc\ub86d\ub2e4\uc6b4(dropdown) \ubaa9\ub85d\uc5d0\uc11c \ubcf4\uae30 \ud398\uc774\uc9c0\ub97c \ubcc0\uacbd\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc5f0\uc0b0 \ubcf4\uae30(Operator view) \uc5f0\uc0b0 \ubcf4\uae30\ub294 \ud638\uc2a4\ud2b8 \ub610\ub294 \uc7a5\uce58\uc5d0\uc11c \uc2e4\ud589\ub418\ub294 \ubaa8\ub4e0 \ud30c\uc774\ud1a0\uce58 \uc5f0\uc0b0\uc790\uc758 \uc131\ub2a5\uc744 \ud45c\uc2dc\ud569\ub2c8\ub2e4. \u201c\uc140\ud504(Self)\u201d \uae30\uac04\uc5d0\ub294 \ud558\uc704 \uc5f0\uc0b0\uc758 \uc2dc\uac04\uc774 \ud3ec\ud568\ub418\uc9c0 \uc54a\uc2b5\ub2c8\ub2e4. \u201c\uc804\uccb4(Total)\u201d \uae30\uac04\uc5d0\ub294 \ud558\uc704 \uc5f0\uc0b0\uc758 \uc2dc\uac04\uc774 \ud3ec\ud568\ub429\ub2c8\ub2e4. \ud638\ucd9c \uc2a4\ud0dd \ubcf4\uae30(View call stack) \uc5f0\uc0b0\uc790\uc758 ``View Callstack``\ub97c \ud074\ub9ad\ud558\uba74, \uc774\ub984\uc740 \uac19\uc9c0\ub9cc \uc11c\ub85c \ub2e4\ub978 \uc5f0\uc0b0\uc790\uac00 \ud45c\uc2dc\ub429\ub2c8\ub2e4. \ud558\uc704 \ud14c\uc774\ube14\uc758 ``View Callstack``\ub97c \ud074\ub9ad\ud558\uba74, \ud638\ucd9c \uc2a4\ud0dd \ud504\ub808\uc784(call stack frames)\uc774 \ud45c\uc2dc\ub429\ub2c8\ub2e4. VS Code \ub0b4\ubd80\uc5d0\uc11c \ud150\uc11c\ubcf4\ub4dc\uac00 \uc2e4\ud589\ub418\ub294 \uacbd\uc6b0 (\uc2e4\ud589 \uac00\uc774\ub4dc), \ud638\ucd9c \uc2a4\ud0dd \ud504\ub808\uc784(call stack frame)\uc744 \ud074\ub9ad\ud558\uba74 \ud2b9\uc815 \ucf54\ub4dc \ub77c\uc778\uc73c\ub85c \uc774\ub3d9\ud569\ub2c8\ub2e4. \ucee4\ub110 \ubcf4\uae30(Kernel view) GPU \ucee4\ub110 \ubcf4\uae30(GPU kernel view)\ub294 \ubaa8\ub4e0 \ucee4\ub110(kernel)\uc774 GPU\uc5d0 \uc18c\ube44\ud55c \uc2dc\uac04\uc744 \ubcf4\uc5ec\uc90d\ub2c8\ub2e4. \uc0ac\uc6a9\ub41c Tensor \ucf54\uc5b4: \uc774 \ucee4\ub110(kernel)\uc774 tensor \ucf54\uc5b4\ub97c \uc0ac\uc6a9\ud558\ub294\uc9c0 \uc5ec\ubd80\ub8f0 \ub098\ud0c0\ub0c5\ub2c8\ub2e4. SM\ub2f9 \ud3c9\uade0 \ube14\ub7ed \uc218: SM\ub2f9 \ube14\ub7ed \uc218 = \ucee4\ub110(kernel)\uc758 \ube14\ub7ed / GPU\uc758 SM \uc218. \uc774 \uc218\uce58\uac00 1\ubcf4\ub2e4 \uc791\uc73c\uba74 GPU \uba40\ud2f0\ud504\ub85c\uc138\uc11c\uac00 \uc644\uc804\ud788 \uc0ac\uc6a9\ub418\uc9c0 \uc54a\uc74c\uc744 \ub098\ud0c0\ub0c5\ub2c8\ub2e4. \u201cSM\ub2f9 \ud3c9\uade0 \ube14\ub7ed \uc218(Mean Blocks per SM)\u201d\ub294 \uc774 \ucee4\ub110 \uc774\ub984\uc758 \ubaa8\ub4e0 \uc2e4\ud589\uc5d0 \ub300\ud55c \uac00\uc911 \ud3c9\uade0\uc774\uace0, \uac01 \uc2e4\ud589 \uae30\uac04\uc744 \uac00\uc911\uce58\ub85c \uc0ac\uc6a9\ud558\uc600\uc2b5\ub2c8\ub2e4. \ud3c9\uade0 \uc608\uc0c1 \ub2ec\uc131 \uc810\uc720\uc728(Mean Est. Achieved Occupancy): \uc608\uc0c1 \ub2ec\uc131 \uc810\uc720\uc728(Est. Achieved Occupancy)\uc740 \uc5f4\uc758 \ud234\ud301(column\u2019s tooltip)\uc5d0 \uc815\uc758\ub418\uc5b4 \uc788\uc2b5\ub2c8\ub2e4. \uba54\ubaa8\ub9ac \ub300\uc5ed\ud3ed \uacbd\uacc4 \ucee4\ub110\uacfc \uac19\uc740 \ub300\ubd80\ubd84\uc758 \uacbd\uc6b0, \ub192\uc744\uc218\ub85d \uc88b\uc2b5\ub2c8\ub2e4. \u201c\ud3c9\uade0 \uc608\uc0c1 \ub2ec\uc131 \uc810\uc720\uc728(Mean Est. Achieved Occupancy)\u201d\uc740 \ucee4\ub110 \uc774\ub984\uc758 \ubaa8\ub4e0 \uc2e4\ud589\uc5d0 \ub300\ud55c \uac00\uc911 \ud3c9\uade0\uc774\uba70, \uac01 \uc2e4\ud589\uc758 \uc9c0\uc18d \uc2dc\uac04\uc744 \uac00\uc911\uce58\ub85c \uc0ac\uc6a9\ud569\ub2c8\ub2e4. \ucd94\uc801 \ubcf4\uae30(Trace view) \ucd94\uc801 \ubcf4\uae30\ub294 \ud504\ub85c\ud30c\uc77c\ub41c \uc5f0\uc0b0\uc790\uc640 GPU \ucee4\ub110\uc758 \ud0c0\uc784\ub77c\uc778\uc744 \ubcf4\uc5ec\uc90d\ub2c8\ub2e4. \uc544\ub798\uc640 \uac19\uc774 \uc120\ud0dd\ud558\uc5ec \uc138\ubd80 \uc815\ubcf4\ub97c \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc624\ub978\ucabd \ub3c4\uad6c \ubaa8\uc74c\uc744 \uc0ac\uc6a9\ud558\uc5ec \uadf8\ub798\ud504\ub97c \uc774\ub3d9\ud558\uace0 \ud655\ub300/\ucd95\uc18c\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \ub610\ud55c \ud0a4\ubcf4\ub4dc\ub97c \uc0ac\uc6a9\ud558\uc5ec \ud0c0\uc784\ub77c\uc778 \uc548\uc5d0\uc11c \ud655\ub300/\uc774\ub3d9\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \u2018w\u2019\ubc0f \u2018s\u2019 \ud0a4\ub294 \ub9c8\uc6b0\uc2a4 \uc911\uc2ec\uc73c\ub85c \ud655\ub300\ub418\uba70, \u2018a\u2019\uc640 \u2018d\u2019 \ud0a4\ub294 \ud0c0\uc784\ub77c\uc778\uc744 \uc88c\uc6b0\ub85c \uc774\ub3d9\ud569\ub2c8\ub2e4. \uc77d\uc744 \uc218 \uc788\ub294 \ud45c\ud604\uc774 \ubcf4\uc77c \ub54c\uae4c\uc9c0 \uc774 \ud0a4\ub97c \uc5ec\ub7ec \ubc88 \ub204\ub97c \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc5ed\ubc29\ud5a5 \uc5f0\uc0b0\uc790(backward operator)\uc758 \u201cIncoming Flow\u201d \ud544\ub4dc\uac00 \u201cforward correspond to backward\u201d \uac12\uc778 \uacbd\uc6b0, \ud14d\uc2a4\ud2b8\ub97c \ud074\ub9ad\ud558\uc5ec \uc2dc\uc791\ub418\ub294 \uc804\uc9c4 \uc5f0\uc0b0\uc790(forward operator)\ub97c \uac00\uc838\uc62c \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc774 \uc608\uc2dc\uc5d0\uc11c\ub294 ``enumerate(DataLoader)``\ub85c \uc811\ub450\uc0ac\uac00 \ubd99\uc740 \uc774\ubca4\ud2b8\uc5d0 \ub9ce\uc740 \uc2dc\uac04\uc774 \uc18c\uc694\ub418\ub294 \uac83\uc744 \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uadf8\ub9ac\uace0 \ub300\ubd80\ubd84\uc758 \uae30\uac04 \ub3d9\uc548 GPU\ub294 \uc26c\ub294 \uc0c1\ud0dc\uc785\ub2c8\ub2e4. \uc774 \uae30\ub2a5\uc740 \ud638\uc2a4\ud2b8 \uce21\uc5d0\uc11c \ub370\uc774\ud130\ub97c \ub85c\ub4dc\ud558\uace0 \ub370\uc774\ud130\ub97c \ubcc0\ud658\ud558\ub294 \uae30\ub2a5\uc774\uae30 \ub54c\ubb38\uc5d0, GPU \ub9ac\uc18c\uc2a4\uac00 \ub0ad\ube44\ub429\ub2c8\ub2e4. 5. \ud504\ub85c\ud30c\uc77c\ub7ec\uc758 \ub3c4\uc6c0\uc73c\ub85c \uc131\ub2a5 \uac1c\uc120# \u201c\uac1c\uc694(Overview)\u201d \ud398\uc774\uc9c0 \ud558\ub2e8\uc758 \u201c\uc131\ub2a5 \ucd94\ucc9c(Performance Recommendation)\u201d \uc81c\uc548\uc740 \ubcd1\ubaa9 \ud604\uc0c1\uc774 \u201c ``DataLoader``\uc784\uc744 \uc554\uc2dc\ud569\ub2c8\ub2e4. \ud30c\uc774\ud1a0\uce58 ``DataLoader``\ub294 \uae30\ubcf8\uc801\uc73c\ub85c \ub2e8\uc77c \ud504\ub85c\uc138\uc2a4\ub97c \uc0ac\uc6a9\ud569\ub2c8\ub2e4. \uc0ac\uc6a9\uc790\ub294 \ub9e4\uac1c\ubcc0\uc218 ``num_workers``\ub97c \uc124\uc815\ud558\uc5ec \ub2e4\uc911 \ud504\ub85c\uc138\uc2a4 \ub370\uc774\ud130 \ub85c\ub4dc\ub97c \ud65c\uc131\ud654\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc790\uc138\ud55c \ub0b4\uc6a9\uc740 \uc5ec\uae30 \uc5d0 \uc788\uc2b5\ub2c8\ub2e4. \uc774 \uc608\uc2dc\uc5d0\uc11c \u201c\uc131\ub2a5 \uad8c\uc7a5\uc0ac\ud56d(Performance Recommendation)\u201d\uc5d0 \ub530\ub77c \uc544\ub798\uc640 \uac19\uc774 ``num_workers``\ub97c \uc124\uc815\ud558\uace0, ``./log/resnet18_4workers``\uc640 \uac19\uc740 \ub2e4\ub978 \uc774\ub984\uc744 ``tensorboard_trace_handler``\ub85c \uc804\ub2ec\ud55c \ud6c4 \ub2e4\uc2dc \uc2e4\ud589\ud569\ub2c8\ub2e4. train_loader = torch.utils.data.DataLoader(train_set, batch_size=32, shuffle=True, num_workers=4) \uadf8\ub7f0 \ub2e4\uc74c \uc67c\ucabd \u201c\uc2e4\ud589(Runs)\u201d \ub4dc\ub86d\ub2e4\uc6b4(dropdown) \ubaa9\ub85d\uc5d0\uc11c \ucd5c\uadfc \ud504\ub85c\ud30c\uc77c\ub41c \uc2e4\ud589\uc744 \uc120\ud0dd\ud569\ub2c8\ub2e4. \uc704\uc758 \ubcf4\uae30(view)\uc5d0\uc11c \uc774\uc804 \uc2e4\ud589\uc778 132ms\uc5d0 \ube44\ud574 \ub2e8\uacc4(step) \uc2dc\uac04\uc774 \uc57d 76ms\ub85c \uac10\uc18c\ud558\uace0, ``DataLoader``\uc758 \uc2dc\uac04 \uac10\uc18c\uac00 \uc8fc\ub85c \uae30\uc5ec\ud55c\ub2e4\ub294 \uac83\uc744 \uc54c \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc704\uc758 \ubcf4\uae30(view)\uc5d0\uc11c ``enumerate(DataLoader)``\uc758 \ub7f0\ud0c0\uc784\uc774 \uac10\uc18c\ud558\uace0, GPU \ud65c\uc6a9\ub3c4\uac00 \uc99d\uac00\ud558\ub294 \uac83\uc744 \uc54c \uc218 \uc788\uc2b5\ub2c8\ub2e4. 6. \ub2e4\ub978 \uace0\uae09 \uae30\ub2a5\uc73c\ub85c \uc131\ub2a5 \ubd84\uc11d# \uba54\ubaa8\ub9ac \ubcf4\uae30(Memory view) \uba54\ubaa8\ub9ac \ud504\ub85c\ud30c\uc77c\uc744 \uc124\uc815\ud558\ub824\uba74 torch.profiler.profile \uc778\uc218\uc5d0\uc11c ``profile_memory``\ub97c ``True``\ub85c \uc124\uc815\ud574\uc57c \ud569\ub2c8\ub2e4. Azure\uc758 \uae30\uc874 \uc608\uc81c\ub97c \uc0ac\uc6a9\ud574 \ubcfc \uc218 \uc788\uc2b5\ub2c8\ub2e4. pip install azure-storage-blob tensorboard --logdir=https://torchtbprofiler.blob.core.windows.net/torchtbprofiler/demo/memory_demo_1_10 \ud504\ub85c\ud30c\uc77c\ub7ec\ub294 \ud504\ub85c\ud30c\uc77c\ub9c1 \uc911\uc5d0 \ubaa8\ub4e0 \uba54\ubaa8\ub9ac \ud560\ub2f9/\ud574\uc81c \uc774\ubca4\ud2b8\uc640 \ud560\ub2f9\uc790\uc758 \ub0b4\ubd80 \uc0c1\ud0dc\ub97c \uae30\ub85d\ud569\ub2c8\ub2e4. \uba54\ubaa8\ub9ac \ubcf4\uae30(memory view)\ub294 \ub2e4\uc74c\uacfc \uac19\uc774 \uc138 \uac00\uc9c0 \uc694\uc18c\ub85c \uad6c\uc131\ub429\ub2c8\ub2e4. \uad6c\uc131 \uc694\uc18c\ub294 \uac01\uac01 \uba54\ubaa8\ub9ac \uace1\uc120 \uadf8\ub798\ud504, \uba54\ubaa8\ub9ac \uc774\ubca4\ud2b8 \ud14c\uc774\ube14 \ubc0f \uba54\ubaa8\ub9ac \ud1b5\uacc4 \ud14c\uc774\ube14\uc785\ub2c8\ub2e4. \uba54\ubaa8\ub9ac \uc720\ud615\uc740 \u201c\uc7a5\uce58(Device)\u201d \uc120\ud0dd \uc0c1\uc790\uc5d0\uc11c \uc120\ud0dd\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc608\ub97c \ub4e4\uba74, \ub2e4\uc74c \ud45c\uc5d0\uc11c \u201cGPU0\u201d\uc740 GPU 0\uc5d0\uc11c\uc758 \uac01 \uc5f0\uc0b0\uc790\uc758 \uba54\ub85c\ub9ac \uc0ac\uc6a9\ub7c9\ub9cc \ubcf4\uc5ec\uc8fc\uace0, CPU \ub610\ub294 \ub2e4\ub978 GPU\ub97c \ud3ec\ud568\ud558\uc9c0 \uc54a\ub294\ub2e4\ub294 \uac83\uc744 \uc758\ubbf8\ud569\ub2c8\ub2e4. \uba54\ubaa8\ub9ac \uace1\uc120\uc740 \uba54\ubaa8\ub9ac \uc18c\ube44\uc758 \ucd94\uc138\ub97c \ubcf4\uc5ec\uc90d\ub2c8\ub2e4. \u201cAllocated\u201d \uace1\uc120\uc740 \uc2e4\uc81c \uc0ac\uc6a9 \uc911\uc778 \ucd1d \uba54\ubaa8\ub9ac, \uc608\ub97c \ub4e4\uba74 tensor\ub97c \ubcf4\uc5ec\uc90d\ub2c8\ub2e4. \ud30c\uc774\ud1a0\uce58\uc5d0\uc11c \uce90\uc2f1 \uba54\ucee4\ub2c8\uc998(caching mechanism)\uc740 CUDA \ud560\ub2f9\uae30 \ubc0f \uc77c\ubd80 \ub2e4\ub978 \ud560\ub2f9\uae30\uc5d0 \uc0ac\uc6a9\ub429\ub2c8\ub2e4. \u201cReserved\u201d \uace1\uc120\uc740 \ud560\ub2f9\uc790\uc5d0 \uc758\ud574 \uc608\uc57d\ub41c \ucd1d \uba54\ubaa8\ub9ac\ub97c \ubcf4\uc5ec\uc90d\ub2c8\ub2e4. \uadf8\ub798\ud504\ub97c \uc88c\ud074\ub9ad\ud558\uace0 \ub04c\uc5b4\uc11c \uc6d0\ud558\ub294 \ubc94\uc704\uc758 \uc774\ubca4\ud2b8\ub97c \uc120\ud0dd\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4: \uc120\ud0dd\ud55c \ud6c4\uc5d0\ub294 \uc138 \uac00\uc9c0 \uad6c\uc131 \uc694\uc18c\uac00 \uc81c\ud55c\ub41c \ubc94\uc704\uc5d0 \ub9de\uac8c \uc5c5\ub370\uc774\ud2b8\ub418\uc5b4 \uc790\uc138\ud55c \uc815\ubcf4\ub97c \uc5bb\uc744 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc774 \ud504\ub85c\uc138\uc2a4\ub97c \ubc18\ubcf5\ud558\uba74, \ub9e4\uc6b0 \uc138\ubd84\ud654\ub41c \uc138\ubd80 \uc815\ubcf4\ub97c \ud655\ub300\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uadf8\ub798\ud504\ub97c \uc6b0\ud074\ub9ad\ud558\uba74 \uadf8\ub798\ud504\uac00 \ucd08\uae30 \uc0c1\ud0dc\ub85c \uc7ac\uc124\uc815\ub429\ub2c8\ub2e4. \uba54\ubaa8\ub9ac \uc774\ubca4\ud2b8 \ud14c\uc774\ube14\uc5d0\uc11c \ud560\ub2f9 \ubc0f \ud574\uc81c \uc774\ubca4\ud2b8\ub294 \ud558\ub098\uc758 \ud56d\ubaa9\uc73c\ub85c \uc30d\uc73c\ub85c \uad6c\uc131\ub429\ub2c8\ub2e4. \u201coperator\u201d \uc5f4\uc5d0\ub294 \ud560\ub2f9\uc744 \ubc1c\uc0dd\uc2dc\ud0a4\ub294 \uc989\uc2dc ATen \uc5f0\uc0b0\uc790\uac00 \ud45c\uc2dc\ub429\ub2c8\ub2e4. \ud30c\uc774\ud1a0\uce58\uc5d0\uc11c ATen \uc5f0\uc0b0\uc790\ub294 \uc77c\ubc18\uc801\uc73c\ub85c aten::empty``\ub97c \uc0ac\uc6a9\ud558\uc5ec \uba54\ubaa8\ub9ac\ub97c \ud560\ub2f9\ud569\ub2c8\ub2e4. \uc608\ub97c \ub4e4\uc5b4, ``aten::ones``\uc740 ``aten::empty \ub2e4\uc74c\uc5d0 ``aten::fill_``\ub85c \uad6c\ud604\ub429\ub2c8\ub2e4. \uc5f0\uc0b0\uc790 \uc774\ub984\ub9cc ``aten::empty``\ub85c \ud45c\uc2dc\ud574\ub3c4 \ubcc4 \ub3c4\uc6c0\uc774 \ub418\uc9c0 \uc54a\uc2b5\ub2c8\ub2e4. \uc774 \ud2b9\uc218\ud55c \uacbd\uc6b0\uc5d0\ub294 ``aten::ones (aten::empty)``\ub85c \ud45c\uc2dc\ub429\ub2c8\ub2e4. \u201c\ud560\ub2f9 \uc2dc\uac04(Allocation Time)\u201d, \u201c\ud574\uc81c \uc2dc\uac04(Release Time)\u201d \ubc0f \u201c\uae30\uac04(Duration)\u201d\uc740 \uc774\ubca4\ud2b8\uac00 \uc2dc\uac04 \ubc94\uc704\ub97c \ubc97\uc5b4\ub098\ub294 \uacbd\uc6b0 \uc5f4\uc758 \ub370\uc774\ud130\uac00 \ub204\ub77d\ub420 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uba54\ubaa8\ub9ac \ud1b5\uacc4 \ud14c\uc774\ube14\uc5d0\uc11c, \u201c\ud06c\uae30 \uc99d\uac00(Size Increase)\u201d \uc5f4\uc740 \ubaa8\ub4e0 \ud560\ub2f9 \ud06c\uae30\ub97c \ud569\uc0b0\ud558\uace0 \ubaa8\ub4e0 \uba54\ubaa8\ub9ac \ub9b4\ub9ac\uc2a4(release) \ud06c\uae30\ub97c \ube80 \uac12, \uc989, \uc774 \uc5f0\uc0b0\uc790 \uc774\ud6c4\uc758 \uba54\ubaa8\ub9ac \uc0ac\uc6a9\ub7c9 \uc21c \uc99d\uac00 \uac12\uc785\ub2c8\ub2e4. \u201c\uc790\uccb4 \ud06c\uae30 \uc99d\uac00(Self Size Increase)\u201d \uc5f4\uc740 \u201c\ud06c\uae30 \uc99d\uac00(Size Increase)\u201d\uc640 \uc720\uc0ac \ud558\uc9c0\ub9cc, \ud558\uc704 \uc5f0\uc0b0\uc790\uc758 \ud560\ub2f9\uc740 \uacc4\uc0b0\ud558\uc9c0 \uc54a\uc2b5\ub2c8\ub2e4. ATen \uc5f0\uc0b0\uc790\uc758 \uad6c\ud604 \uc138\ubd80 \uc0ac\ud56d\uacfc \uad00\ub828\ud558\uc5ec, \uc77c\ubd80 \uc5f0\uc0b0\uc790\ub294 \ub2e4\ub978 \uc5f0\uc0b0\uc790\ub97c \ud638\ucd9c\ud560 \uc218 \uc788\uc73c\ubbc0\ub85c, \uba54\ubaa8\ub9ac \ud560\ub2f9\uc740 \ucf5c \uc2a4\ud0dd\uc758 \ubaa8\ub4e0 \uc218\uc900\uc5d0\uc11c \ubc1c\uc0dd\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc989, \u201c\uc790\uccb4 \ud06c\uae30 \uc99d\uac00(Self Size Increase)\u201d\ub294 \ud604\uc7ac \uc218\uc900\uc758 \ucf5c \uc2a4\ud0dd\uc5d0\uc11c \uba54\ubaa8\ub9ac \uc0ac\uc6a9\ub7c9 \uc99d\uac00\ub9cc\uc744 \uacc4\uc0b0\ud569\ub2c8\ub2e4. \ub9c8\uc9c0\ub9c9\uc73c\ub85c, \u201c\ud560\ub2f9 \ud06c\uae30(Allocation Size)\u201d \uc5f4\uc740 \uba54\ubaa8\ub9ac \ub9b4\ub9ac\uc2a4\ub97c \uace0\ub824\ud558\uc9c0 \uc54a\uace0 \ubaa8\ub4e0 \ud560\ub2f9\uc744 \ud569\uc0b0\ud569\ub2c8\ub2e4. \ubd84\uc0b0 \ubcf4\uae30(Distributed view) \uc774\uc81c \ud50c\ub7ec\uadf8\uc778\uc740 NCCL/GLOO\ub97c \ubc31\uc5d4\ub4dc\ub85c \uc0ac\uc6a9\ud558\ub294 DDP \ud504\ub85c\ud30c\uc77c\ub9c1\uc5d0 \ub300\ud55c \ubd84\uc0b0 \ubcf4\uae30\ub97c \uc9c0\uc6d0\ud569\ub2c8\ub2e4. Azure\uc758 \uae30\uc874 \uc608\uc81c\ub97c \uc0ac\uc6a9\ud574 \ubcfc \uc218 \uc788\uc2b5\ub2c8\ub2e4: pip install azure-storage-blob tensorboard --logdir=https://torchtbprofiler.blob.core.windows.net/torchtbprofiler/demo/distributed_bert \u201c\ucef4\ud4e8\ud305/\ucee4\ubba4\ub2c8\ucf00\uc774\uc158 \uac1c\uc694(Computation/Communication Overview)\u201d\uc5d0\ub294 \ucef4\ud4e8\ud305/\ucee4\ubba4\ub2c8\ucf00\uc774\uc158 \ube44\uc728\uacfc \uc911\ubcf5 \uc815\ub3c4\uac00 \ud45c\uc2dc\ub429\ub2c8\ub2e4. \uc774 \ubcf4\uae30\uc5d0\uc11c, \uc0ac\uc6a9\uc790\ub294 \uc791\uc5c5\uc790 \uac04\uc758 \ub85c\ub4dc \ubc38\ub7f0\uc2f1 \ubb38\uc81c\ub97c \ud30c\uc545\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc608\ub97c \ub4e4\uc5b4, \ud55c \uc791\uc5c5\uc790\uc758 \uc5f0\uc0b0 + \uc911\ubcf5 \uc2dc\uac04\uc774 \ub2e4\ub978 \uc791\uc5c5\uc790\ubcf4\ub2e4 \ud6e8\uc52c \ud070 \uacbd\uc6b0, \ub85c\ub4dc \ubc38\ub7f0\uc2f1\uc5d0 \ubb38\uc81c\uac00 \uc788\uac70\ub098 \uc774 \uc791\uc5c5\uc790\uac00 \ub099\uc624\uc790(straggler)\uc77c \uc218 \uc788\uc2b5\ub2c8\ub2e4. \u201c\ub3d9\uae30\ud654/\ucee4\ubba4\ub2c8\ucf00\uc774\uc158 \uac1c\uc694(Synchronizing/Communication Overview)\u201d\ub294 \ud1b5\uc2e0\uc758 \ud6a8\uc728\uc131\uc744 \ubcf4\uc5ec\uc90d\ub2c8\ub2e4. \u201c\ub370\uc774\ud130 \uad50\ud658 \uc2dc\uac04(Data Transfer Time)\u201d\uc740 \uc2e4\uc81c \ub370\uc774\ud130\ub97c \uad50\ud658\ud558\ub294 \uc2dc\uac04\uc785\ub2c8\ub2e4. \u201c\ub3d9\uae30\ud654 \uc2dc\uac04(Synchronizing Time)\u201d\uc740 \ub2e4\ub978 \uc791\uc5c5\uc790\uc640 \ub300\uae30 \ubc0f \ub3d9\uae30\ud654\ud558\ub294 \uc2dc\uac04\uc785\ub2c8\ub2e4. \ud55c \uc791\uc5c5\uc790\uc758 \u201c\ub3d9\uae30\ud654 \uc2dc\uac04\u201d\uc774 \ub2e4\ub978 \uc791\uc5c5\uc790 \ubcf4\ub2e4 \ud6e8\uc52c \uc9e7\ub2e4\uba74\u2019, \uc774 \uc791\uc5c5\uc790\ub294 \ub2e4\ub978 \uc791\uc5c5\uc790\ubcf4\ub2e4 \ub354 \ub9ce\uc740 \uacc4\uc0b0 \uc791\uc5c5\ub7c9\uc744 \uac00\uc9c8 \uc218 \uc788\ub294 \ub099\uc624\uc790(straggler)\uc77c \uc218 \uc788\uc2b5\ub2c8\ub2e4\u2019. \u201c\ucee4\ubba4\ub2c8\ucf00\uc774\uc158 \uc791\uc5c5 \ud1b5\uacc4(Communication Operations Stats)\u201d\ub294 \uac01 \uc791\uc5c5\uc790\uc758 \ubaa8\ub4e0 \ud1b5\uc2e0 \uc791\uc5c5\uc5d0 \ub300\ud55c \uc138\ubd80 \ud1b5\uacc4\ub97c \uc694\uc57d\ud569\ub2c8\ub2e4. 7. \ucd94\uac00 \uc5f0\uc2b5: AMD GPU\uc5d0\uc11c PyTorch \ud504\ub85c\ud30c\uc77c\ub9c1# The AMD ROCm Platform is an open-source software stack designed for GPU computation, consisting of drivers, development tools, and APIs. We can run the above mentioned steps on AMD GPUs. In this section, we will use Docker to install the ROCm base development image before installing PyTorch. For the purpose of example, let\u2019s create a directory called profiler_tutorial, and save the code in Step 1 as test_cifar10.py in this directory. mkdir ~/profiler_tutorial cd profiler_tutorial vi test_cifar10.py At the time of this writing, the Stable(2.1.1) Linux version of PyTorch on ROCm Platform is ROCm 5.6. Obtain a base Docker image with the correct user-space ROCm version installed from Docker Hub. It is rocm/dev-ubuntu-20.04:5.6. Start the ROCm base Docker container: docker run -it --network=host --device=/dev/kfd --device=/dev/dri --group-add=video --ipc=host --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --shm-size 8G -v ~/profiler_tutorial:/profiler_tutorial rocm/dev-ubuntu-20.04:5.6 Inside the container, install any dependencies needed for installing the wheels package. sudo apt update sudo apt install libjpeg-dev python3-dev -y pip3 install wheel setuptools sudo apt install python-is-python3 Install the wheels: pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.6 Install the torch_tb_profiler, and then, run the Python file test_cifar10.py: pip install torch_tb_profiler cd /profiler_tutorial python test_cifar10.py Now, we have all the data needed to view in TensorBoard: tensorboard --logdir=./log Choose different views as described in Step 4. For example, below is the Operator View: At the time this section is written, Trace view does not work and it displays nothing. You can work around by typing chrome://tracing in your Chrome Browser. Copy the trace.json file under ~/profiler_tutorial/log/resnet18 directory to the Windows. You may need to copy the file by using scp if the file is located in a remote location. Click Load button to load the trace JSON file from the chrome://tracing page in the browser. As mentioned previously, you can move the graph and zoom in and out. You can also use keyboard to zoom and move around inside the timeline. The w and s keys zoom in centered around the mouse, and the a and d keys move the timeline left and right. You can hit these keys multiple times until you see a readable representation. \ub354 \uc54c\uc544\ubcf4\uae30# \ud559\uc2b5\uc744 \uacc4\uc18d\ud558\ub824\uba74 \ub2e4\uc74c \ubb38\uc11c\ub97c \ucc38\uc870\ud558\uc2dc\uace0, \uc5ec\uae30 \uc5d0\uc11c \uc790\uc720\ub86d\uac8c \uc774\uc288\ub97c \uc5f4\uc5b4\ubcf4\uc138\uc694. PyTorch TensorBoard Profiler Github torch.profiler API HTA Total running time of the script: (0 minutes 19.780 seconds) Download Jupyter notebook: tensorboard_profiler_tutorial.ipynb Download Python source code: tensorboard_profiler_tutorial.py Download zipped: tensorboard_profiler_tutorial.zip",
       "author": {
         "@type": "Organization",
         "name": "PyTorch Contributors",
         "url": "https://pytorch.org"
       },
       "image": "../_static/img/pytorch_seo.png",
       "mainEntityOfPage": {
         "@type": "WebPage",
         "@id": "/intermediate/tensorboard_profiler_tutorial.html"
       },
       "datePublished": "2023-01-01T00:00:00Z",
       "dateModified": "2023-01-01T00:00:00Z"
     }