René's URL Explorer Experiment

    {
       "@context": "https://schema.org",
       "@type": "Article",
       "name": "TorchVision Object Detection Finetuning Tutorial",
       "headline": "TorchVision Object Detection Finetuning Tutorial",
       "description": "PyTorch Documentation. Explore PyTorch, an open-source machine learning library that accelerates the path from research prototyping to production deployment. Discover tutorials, API references, and guides to help you build and deploy deep learning models efficiently.",
       "url": "/intermediate/torchvision_tutorial.html",
       "articleBody": "\ucc38\uace0 Go to the end to download the full example code. TorchVision Object Detection Finetuning Tutorial# For this tutorial, we will be finetuning a pre-trained Mask R-CNN model on the Penn-Fudan Database for Pedestrian Detection and Segmentation. It contains 170 images with 345 instances of pedestrians, and we will use it to illustrate how to use the new features in torchvision in order to train an object detection and instance segmentation model on a custom dataset. \ucc38\uace0 This tutorial works only with torchvision version \u003e=0.16 or nightly. If you\u2019re using torchvision\u003c=0.15, please follow this tutorial instead. Defining the Dataset# The reference scripts for training object detection, instance segmentation and person keypoint detection allows for easily supporting adding new custom datasets. The dataset should inherit from the standard torch.utils.data.Dataset class, and implement __len__ and __getitem__. The only specificity that we require is that the dataset __getitem__ should return a tuple: image: torchvision.tv_tensors.Image of shape [3, H, W], a pure tensor, or a PIL Image of size (H, W) target: a dict containing the following fields boxes, torchvision.tv_tensors.BoundingBoxes of shape [N, 4]: the coordinates of the N bounding boxes in [x0, y0, x1, y1] format, ranging from 0 to W and 0 to H labels, integer torch.Tensor of shape [N]: the label for each bounding box. 0 represents always the background class. image_id, int: an image identifier. It should be unique between all the images in the dataset, and is used during evaluation area, float torch.Tensor of shape [N]: the area of the bounding box. This is used during evaluation with the COCO metric, to separate the metric scores between small, medium and large boxes. iscrowd, uint8 torch.Tensor of shape [N]: instances with iscrowd=True will be ignored during evaluation. (optionally) masks, torchvision.tv_tensors.Mask of shape [N, H, W]: the segmentation masks for each one of the objects If your dataset is compliant with above requirements then it will work for both training and evaluation codes from the reference script. Evaluation code will use scripts from pycocotools which can be installed with pip install pycocotools. \ucc38\uace0 For Windows, please install pycocotools from gautamchitnis with command pip install git+https://github.com/gautamchitnis/cocoapi.git@cocodataset-master#subdirectory=PythonAPI One note on the labels. The model considers class 0 as background. If your dataset does not contain the background class, you should not have 0 in your labels. For example, assuming you have just two classes, cat and dog, you can define 1 (not 0) to represent cats and 2 to represent dogs. So, for instance, if one of the images has both classes, your labels tensor should look like [1, 2]. Additionally, if you want to use aspect ratio grouping during training (so that each batch only contains images with similar aspect ratios), then it is recommended to also implement a get_height_and_width method, which returns the height and the width of the image. If this method is not provided, we query all elements of the dataset via __getitem__ , which loads the image in memory and is slower than if a custom method is provided. Writing a custom dataset for PennFudan# Let\u2019s write a dataset for the PennFudan dataset. First, let\u2019s download the dataset and extract the zip file: wget https://www.cis.upenn.edu/~jshi/ped_html/PennFudanPed.zip -P data cd data \u0026\u0026 unzip PennFudanPed.zip We have the following folder structure: PennFudanPed/ PedMasks/ FudanPed00001_mask.png FudanPed00002_mask.png FudanPed00003_mask.png FudanPed00004_mask.png ... PNGImages/ FudanPed00001.png FudanPed00002.png FudanPed00003.png FudanPed00004.png Here is one example of a pair of images and segmentation masks import matplotlib.pyplot as plt from torchvision.io import read_image image = read_image(\"data/PennFudanPed/PNGImages/FudanPed00046.png\") mask = read_image(\"data/PennFudanPed/PedMasks/FudanPed00046_mask.png\") plt.figure(figsize=(16, 8)) plt.subplot(121) plt.title(\"Image\") plt.imshow(image.permute(1, 2, 0)) plt.subplot(122) plt.title(\"Mask\") plt.imshow(mask.permute(1, 2, 0)) \u003cmatplotlib.image.AxesImage object at 0x7f0d933dbb90\u003e So each image has a corresponding segmentation mask, where each color correspond to a different instance. Let\u2019s write a torch.utils.data.Dataset class for this dataset. In the code below, we are wrapping images, bounding boxes and masks into torchvision.tv_tensors.TVTensor classes so that we will be able to apply torchvision built-in transformations (new Transforms API) for the given object detection and segmentation task. Namely, image tensors will be wrapped by torchvision.tv_tensors.Image, bounding boxes into torchvision.tv_tensors.BoundingBoxes and masks into torchvision.tv_tensors.Mask. As torchvision.tv_tensors.TVTensor are torch.Tensor subclasses, wrapped objects are also tensors and inherit the plain torch.Tensor API. For more information about torchvision tv_tensors see this documentation. import os import torch from torchvision.io import read_image from torchvision.ops.boxes import masks_to_boxes from torchvision import tv_tensors from torchvision.transforms.v2 import functional as F class PennFudanDataset(torch.utils.data.Dataset): def __init__(self, root, transforms): self.root = root self.transforms = transforms # load all image files, sorting them to # ensure that they are aligned self.imgs = list(sorted(os.listdir(os.path.join(root, \"PNGImages\")))) self.masks = list(sorted(os.listdir(os.path.join(root, \"PedMasks\")))) def __getitem__(self, idx): # load images and masks img_path = os.path.join(self.root, \"PNGImages\", self.imgs[idx]) mask_path = os.path.join(self.root, \"PedMasks\", self.masks[idx]) img = read_image(img_path) mask = read_image(mask_path) # instances are encoded as different colors obj_ids = torch.unique(mask) # first id is the background, so remove it obj_ids = obj_ids[1:] num_objs = len(obj_ids) # split the color-encoded mask into a set # of binary masks masks = (mask == obj_ids[:, None, None]).to(dtype=torch.uint8) # get bounding box coordinates for each mask boxes = masks_to_boxes(masks) # there is only one class labels = torch.ones((num_objs,), dtype=torch.int64) image_id = idx area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0]) # suppose all instances are not crowd iscrowd = torch.zeros((num_objs,), dtype=torch.int64) # Wrap sample and targets into torchvision tv_tensors: img = tv_tensors.Image(img) target = {} target[\"boxes\"] = tv_tensors.BoundingBoxes(boxes, format=\"XYXY\", canvas_size=F.get_size(img)) target[\"masks\"] = tv_tensors.Mask(masks) target[\"labels\"] = labels target[\"image_id\"] = image_id target[\"area\"] = area target[\"iscrowd\"] = iscrowd if self.transforms is not None: img, target = self.transforms(img, target) return img, target def __len__(self): return len(self.imgs) That\u2019s all for the dataset. Now let\u2019s define a model that can perform predictions on this dataset. Defining your model# In this tutorial, we will be using Mask R-CNN, which is based on top of Faster R-CNN. Faster R-CNN is a model that predicts both bounding boxes and class scores for potential objects in the image. Mask R-CNN adds an extra branch into Faster R-CNN, which also predicts segmentation masks for each instance. There are two common situations where one might want to modify one of the available models in TorchVision Model Zoo. The first is when we want to start from a pre-trained model, and just finetune the last layer. The other is when we want to replace the backbone of the model with a different one (for faster predictions, for example). Let\u2019s go see how we would do one or another in the following sections. 1 - Finetuning from a pretrained model# Let\u2019s suppose that you want to start from a model pre-trained on COCO and want to finetune it for your particular classes. Here is a possible way of doing it: import torchvision from torchvision.models.detection.faster_rcnn import FastRCNNPredictor # load a model pre-trained on COCO model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights=\"DEFAULT\") # replace the classifier with a new one, that has # num_classes which is user-defined num_classes = 2 # 1 class (person) + background # get number of input features for the classifier in_features = model.roi_heads.box_predictor.cls_score.in_features # replace the pre-trained head with a new one model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes) Downloading: \"https://download.pytorch.org/models/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth\" to /root/.cache/torch/hub/checkpoints/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth 0%| | 0.00/160M [00:00\u003c?, ?B/s] 0%| | 128k/160M [00:00\u003c07:42, 362kB/s] 0%| | 256k/160M [00:00\u003c05:26, 512kB/s] 0%| | 384k/160M [00:00\u003c04:41, 593kB/s] 0%| | 768k/160M [00:00\u003c02:26, 1.14MB/s] 1%| | 1.62M/160M [00:01\u003c01:07, 2.44MB/s] 2%|\u258f | 3.12M/160M [00:01\u003c00:36, 4.46MB/s] 4%|\u258d | 6.25M/160M [00:01\u003c00:18, 8.73MB/s] 6%|\u258b | 10.1M/160M [00:01\u003c00:12, 12.9MB/s] 9%|\u2589 | 14.0M/160M [00:01\u003c00:09, 15.6MB/s] 11%|\u2588 | 17.8M/160M [00:02\u003c00:08, 17.4MB/s] 13%|\u2588\u258e | 20.4M/160M [00:02\u003c00:08, 16.8MB/s] 15%|\u2588\u258c | 24.2M/160M [00:02\u003c00:07, 18.3MB/s] 18%|\u2588\u258a | 28.1M/160M [00:02\u003c00:07, 19.5MB/s] 20%|\u2588\u2588 | 32.0M/160M [00:02\u003c00:06, 20.2MB/s] 22%|\u2588\u2588\u258f | 35.9M/160M [00:02\u003c00:06, 20.8MB/s] 25%|\u2588\u2588\u258d | 39.8M/160M [00:03\u003c00:05, 21.2MB/s] 27%|\u2588\u2588\u258b | 43.6M/160M [00:03\u003c00:05, 21.5MB/s] 30%|\u2588\u2588\u2589 | 47.4M/160M [00:03\u003c00:05, 21.4MB/s] 32%|\u2588\u2588\u2588\u258f | 51.2M/160M [00:03\u003c00:05, 21.6MB/s] 35%|\u2588\u2588\u2588\u258d | 55.1M/160M [00:03\u003c00:05, 21.8MB/s] 37%|\u2588\u2588\u2588\u258b | 59.1M/160M [00:04\u003c00:04, 22.0MB/s] 39%|\u2588\u2588\u2588\u2589 | 63.0M/160M [00:04\u003c00:04, 22.0MB/s] 42%|\u2588\u2588\u2588\u2588\u258f | 66.9M/160M [00:04\u003c00:04, 22.0MB/s] 44%|\u2588\u2588\u2588\u2588\u258d | 70.8M/160M [00:04\u003c00:04, 22.0MB/s] 47%|\u2588\u2588\u2588\u2588\u258b | 74.5M/160M [00:04\u003c00:04, 21.9MB/s] 49%|\u2588\u2588\u2588\u2588\u2589 | 78.4M/160M [00:04\u003c00:03, 21.9MB/s] 51%|\u2588\u2588\u2588\u2588\u2588\u258f | 82.2M/160M [00:05\u003c00:03, 21.9MB/s] 54%|\u2588\u2588\u2588\u2588\u2588\u258d | 86.1M/160M [00:05\u003c00:03, 22.0MB/s] 56%|\u2588\u2588\u2588\u2588\u2588\u258b | 90.0M/160M [00:05\u003c00:03, 22.0MB/s] 59%|\u2588\u2588\u2588\u2588\u2588\u2589 | 93.9M/160M [00:05\u003c00:03, 21.8MB/s] 61%|\u2588\u2588\u2588\u2588\u2588\u2588 | 97.6M/160M [00:05\u003c00:02, 21.7MB/s] 64%|\u2588\u2588\u2588\u2588\u2588\u2588\u258e | 102M/160M [00:06\u003c00:02, 21.8MB/s] 66%|\u2588\u2588\u2588\u2588\u2588\u2588\u258c | 105M/160M [00:06\u003c00:02, 21.0MB/s] 68%|\u2588\u2588\u2588\u2588\u2588\u2588\u258a | 109M/160M [00:06\u003c00:02, 21.2MB/s] 70%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588 | 112M/160M [00:06\u003c00:02, 21.2MB/s] 73%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e | 116M/160M [00:06\u003c00:02, 21.5MB/s] 75%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258c | 120M/160M [00:06\u003c00:01, 21.4MB/s] 78%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a | 124M/160M [00:07\u003c00:01, 21.7MB/s] 80%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588 | 128M/160M [00:07\u003c00:01, 21.8MB/s] 82%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f | 132M/160M [00:07\u003c00:01, 21.5MB/s] 85%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258d | 136M/160M [00:07\u003c00:01, 21.6MB/s] 87%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258b | 139M/160M [00:07\u003c00:01, 21.2MB/s] 90%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2589 | 143M/160M [00:08\u003c00:00, 21.4MB/s] 92%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f| 147M/160M [00:08\u003c00:00, 21.4MB/s] 95%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258d| 151M/160M [00:08\u003c00:00, 21.5MB/s] 97%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258b| 155M/160M [00:08\u003c00:00, 21.2MB/s] 99%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a| 158M/160M [00:08\u003c00:00, 19.5MB/s] 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 160M/160M [00:08\u003c00:00, 18.8MB/s] 2 - Modifying the model to add a different backbone# import torchvision from torchvision.models.detection import FasterRCNN from torchvision.models.detection.rpn import AnchorGenerator # load a pre-trained model for classification and return # only the features backbone = torchvision.models.mobilenet_v2(weights=\"DEFAULT\").features # ``FasterRCNN`` needs to know the number of # output channels in a backbone. For mobilenet_v2, it\u0027s 1280 # so we need to add it here backbone.out_channels = 1280 # let\u0027s make the RPN generate 5 x 3 anchors per spatial # location, with 5 different sizes and 3 different aspect # ratios. We have a Tuple[Tuple[int]] because each feature # map could potentially have different sizes and # aspect ratios anchor_generator = AnchorGenerator( sizes=((32, 64, 128, 256, 512),), aspect_ratios=((0.5, 1.0, 2.0),) ) # let\u0027s define what are the feature maps that we will # use to perform the region of interest cropping, as well as # the size of the crop after rescaling. # if your backbone returns a Tensor, featmap_names is expected to # be [0]. More generally, the backbone should return an # ``OrderedDict[Tensor]``, and in ``featmap_names`` you can choose which # feature maps to use. roi_pooler = torchvision.ops.MultiScaleRoIAlign( featmap_names=[\u00270\u0027], output_size=7, sampling_ratio=2 ) # put the pieces together inside a Faster-RCNN model model = FasterRCNN( backbone, num_classes=2, rpn_anchor_generator=anchor_generator, box_roi_pool=roi_pooler ) Downloading: \"https://download.pytorch.org/models/mobilenet_v2-7ebf99e0.pth\" to /root/.cache/torch/hub/checkpoints/mobilenet_v2-7ebf99e0.pth 0%| | 0.00/13.6M [00:00\u003c?, ?B/s] 52%|\u2588\u2588\u2588\u2588\u2588\u258f | 7.12M/13.6M [00:00\u003c00:00, 67.1MB/s] 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 13.6M/13.6M [00:00\u003c00:00, 79.2MB/s] Object detection and instance segmentation model for PennFudan Dataset# In our case, we want to finetune from a pre-trained model, given that our dataset is very small, so we will be following approach number 1. Here we want to also compute the instance segmentation masks, so we will be using Mask R-CNN: import torchvision from torchvision.models.detection.faster_rcnn import FastRCNNPredictor from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor def get_model_instance_segmentation(num_classes): # load an instance segmentation model pre-trained on COCO model = torchvision.models.detection.maskrcnn_resnet50_fpn(weights=\"DEFAULT\") # get number of input features for the classifier in_features = model.roi_heads.box_predictor.cls_score.in_features # replace the pre-trained head with a new one model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes) # now get the number of input features for the mask classifier in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels hidden_layer = 256 # and replace the mask predictor with a new one model.roi_heads.mask_predictor = MaskRCNNPredictor( in_features_mask, hidden_layer, num_classes ) return model That\u2019s it, this will make model be ready to be trained and evaluated on your custom dataset. Putting everything together# In references/detection/, we have a number of helper functions to simplify training and evaluating detection models. Here, we will use references/detection/engine.py and references/detection/utils.py. Just download everything under references/detection to your folder and use them here. On Linux if you have wget, you can download them using below commands: os.system(\"wget https://raw.githubusercontent.com/pytorch/vision/main/references/detection/engine.py\") os.system(\"wget https://raw.githubusercontent.com/pytorch/vision/main/references/detection/utils.py\") os.system(\"wget https://raw.githubusercontent.com/pytorch/vision/main/references/detection/coco_utils.py\") os.system(\"wget https://raw.githubusercontent.com/pytorch/vision/main/references/detection/coco_eval.py\") os.system(\"wget https://raw.githubusercontent.com/pytorch/vision/main/references/detection/transforms.py\") 0 Since v0.15.0 torchvision provides new Transforms API to easily write data augmentation pipelines for Object Detection and Segmentation tasks. Let\u2019s write some helper functions for data augmentation / transformation: from torchvision.transforms import v2 as T def get_transform(train): transforms = [] if train: transforms.append(T.RandomHorizontalFlip(0.5)) transforms.append(T.ToDtype(torch.float, scale=True)) transforms.append(T.ToPureTensor()) return T.Compose(transforms) Testing forward() method (Optional)# Before iterating over the dataset, it\u2019s good to see what the model expects during training and inference time on sample data. import utils model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights=\"DEFAULT\") dataset = PennFudanDataset(\u0027data/PennFudanPed\u0027, get_transform(train=True)) data_loader = torch.utils.data.DataLoader( dataset, batch_size=2, shuffle=True, collate_fn=utils.collate_fn ) # For Training images, targets = next(iter(data_loader)) images = list(image for image in images) targets = [{k: v for k, v in t.items()} for t in targets] output = model(images, targets) # Returns losses and detections print(output) # For inference model.eval() x = [torch.rand(3, 300, 400), torch.rand(3, 500, 400)] predictions = model(x) # Returns predictions print(predictions[0]) {\u0027loss_classifier\u0027: tensor(0.0503, grad_fn=\u003cNllLossBackward0\u003e), \u0027loss_box_reg\u0027: tensor(0.0387, grad_fn=\u003cDivBackward0\u003e), \u0027loss_objectness\u0027: tensor(0.0080, grad_fn=\u003cBinaryCrossEntropyWithLogitsBackward0\u003e), \u0027loss_rpn_box_reg\u0027: tensor(0.0018, grad_fn=\u003cDivBackward0\u003e)} {\u0027boxes\u0027: tensor([], size=(0, 4), grad_fn=\u003cStackBackward0\u003e), \u0027labels\u0027: tensor([], dtype=torch.int64), \u0027scores\u0027: tensor([], grad_fn=\u003cIndexBackward0\u003e)} We want to be able to train our model on an accelerator such as CUDA, MPS, MTIA, or XPU. Let\u2019s now write the main function which performs the training and the validation: from engine import train_one_epoch, evaluate # train on the accelerator or on the CPU, if an accelerator is not available device = torch.accelerator.current_accelerator() if torch.accelerator.is_available() else torch.device(\u0027cpu\u0027) # our dataset has two classes only - background and person num_classes = 2 # use our dataset and defined transformations dataset = PennFudanDataset(\u0027data/PennFudanPed\u0027, get_transform(train=True)) dataset_test = PennFudanDataset(\u0027data/PennFudanPed\u0027, get_transform(train=False)) # split the dataset in train and test set indices = torch.randperm(len(dataset)).tolist() dataset = torch.utils.data.Subset(dataset, indices[:-50]) dataset_test = torch.utils.data.Subset(dataset_test, indices[-50:]) # define training and validation data loaders data_loader = torch.utils.data.DataLoader( dataset, batch_size=2, shuffle=True, collate_fn=utils.collate_fn ) data_loader_test = torch.utils.data.DataLoader( dataset_test, batch_size=1, shuffle=False, collate_fn=utils.collate_fn ) # get the model using our helper function model = get_model_instance_segmentation(num_classes) # move model to the right device model.to(device) # construct an optimizer params = [p for p in model.parameters() if p.requires_grad] optimizer = torch.optim.SGD( params, lr=0.005, momentum=0.9, weight_decay=0.0005 ) # and a learning rate scheduler lr_scheduler = torch.optim.lr_scheduler.StepLR( optimizer, step_size=3, gamma=0.1 ) # let\u0027s train it just for 2 epochs num_epochs = 2 for epoch in range(num_epochs): # train for one epoch, printing every 10 iterations train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10) # update the learning rate lr_scheduler.step() # evaluate on the test dataset evaluate(model, data_loader_test, device=device) print(\"That\u0027s it!\") Downloading: \"https://download.pytorch.org/models/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth\" to /root/.cache/torch/hub/checkpoints/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth 0%| | 0.00/170M [00:00\u003c?, ?B/s] 0%| | 128k/170M [00:00\u003c07:48, 380kB/s] 0%| | 256k/170M [00:00\u003c05:30, 539kB/s] 0%| | 384k/170M [00:00\u003c04:44, 624kB/s] 0%| | 768k/170M [00:00\u003c02:27, 1.20MB/s] 1%| | 1.62M/170M [00:01\u003c01:08, 2.56MB/s] 2%|\u258f | 3.12M/170M [00:01\u003c00:37, 4.67MB/s] 4%|\u258e | 6.25M/170M [00:01\u003c00:18, 9.19MB/s] 6%|\u258c | 10.0M/170M [00:01\u003c00:12, 13.3MB/s] 8%|\u258a | 13.8M/170M [00:01\u003c00:10, 16.1MB/s] 10%|\u2588 | 17.5M/170M [00:01\u003c00:08, 18.1MB/s] 13%|\u2588\u258e | 21.4M/170M [00:02\u003c00:07, 19.5MB/s] 15%|\u2588\u258d | 25.1M/170M [00:02\u003c00:07, 20.2MB/s] 17%|\u2588\u258b | 28.5M/170M [00:02\u003c00:07, 20.2MB/s] 19%|\u2588\u2589 | 32.4M/170M [00:02\u003c00:06, 21.2MB/s] 21%|\u2588\u2588\u258f | 36.1M/170M [00:02\u003c00:06, 21.6MB/s] 24%|\u2588\u2588\u258e | 40.0M/170M [00:02\u003c00:06, 22.1MB/s] 26%|\u2588\u2588\u258c | 43.9M/170M [00:03\u003c00:05, 22.3MB/s] 28%|\u2588\u2588\u258a | 46.8M/170M [00:03\u003c00:06, 20.8MB/s] 30%|\u2588\u2588\u2589 | 50.6M/170M [00:03\u003c00:05, 21.5MB/s] 32%|\u2588\u2588\u2588\u258f | 54.4M/170M [00:03\u003c00:05, 21.7MB/s] 34%|\u2588\u2588\u2588\u258d | 58.1M/170M [00:03\u003c00:05, 21.9MB/s] 36%|\u2588\u2588\u2588\u258b | 61.9M/170M [00:04\u003c00:05, 22.1MB/s] 39%|\u2588\u2588\u2588\u258a | 65.6M/170M [00:04\u003c00:04, 22.2MB/s] 41%|\u2588\u2588\u2588\u2588 | 69.5M/170M [00:04\u003c00:04, 22.5MB/s] 43%|\u2588\u2588\u2588\u2588\u258e | 73.4M/170M [00:04\u003c00:04, 22.7MB/s] 45%|\u2588\u2588\u2588\u2588\u258c | 77.2M/170M [00:04\u003c00:04, 22.8MB/s] 48%|\u2588\u2588\u2588\u2588\u258a | 81.0M/170M [00:04\u003c00:04, 22.7MB/s] 50%|\u2588\u2588\u2588\u2588\u2589 | 84.8M/170M [00:05\u003c00:03, 22.7MB/s] 52%|\u2588\u2588\u2588\u2588\u2588\u258f | 88.6M/170M [00:05\u003c00:03, 22.7MB/s] 54%|\u2588\u2588\u2588\u2588\u2588\u258d | 92.5M/170M [00:05\u003c00:03, 22.9MB/s] 57%|\u2588\u2588\u2588\u2588\u2588\u258b | 96.4M/170M [00:05\u003c00:03, 23.0MB/s] 59%|\u2588\u2588\u2588\u2588\u2588\u2589 | 100M/170M [00:05\u003c00:03, 23.0MB/s] 61%|\u2588\u2588\u2588\u2588\u2588\u2588\u258f | 104M/170M [00:05\u003c00:02, 23.2MB/s] 64%|\u2588\u2588\u2588\u2588\u2588\u2588\u258e | 108M/170M [00:06\u003c00:02, 22.9MB/s] 66%|\u2588\u2588\u2588\u2588\u2588\u2588\u258c | 112M/170M [00:06\u003c00:02, 22.5MB/s] 68%|\u2588\u2588\u2588\u2588\u2588\u2588\u258a | 116M/170M [00:06\u003c00:02, 22.8MB/s] 70%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588 | 119M/170M [00:06\u003c00:02, 22.8MB/s] 72%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f | 123M/170M [00:06\u003c00:02, 22.7MB/s] 75%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258d | 127M/170M [00:07\u003c00:01, 22.8MB/s] 77%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258b | 131M/170M [00:07\u003c00:01, 22.9MB/s] 79%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2589 | 135M/170M [00:07\u003c00:01, 22.9MB/s] 82%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f | 138M/170M [00:07\u003c00:01, 22.8MB/s] 84%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258d | 142M/170M [00:07\u003c00:01, 22.9MB/s] 86%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258c | 146M/170M [00:07\u003c00:01, 23.0MB/s] 88%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a | 150M/170M [00:08\u003c00:00, 22.8MB/s] 91%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588 | 154M/170M [00:08\u003c00:00, 22.8MB/s] 93%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e| 158M/170M [00:08\u003c00:00, 22.7MB/s] 95%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258c| 161M/170M [00:08\u003c00:00, 22.8MB/s] 97%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258b| 165M/170M [00:08\u003c00:00, 22.7MB/s] 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2589| 169M/170M [00:08\u003c00:00, 22.9MB/s] 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 170M/170M [00:08\u003c00:00, 19.9MB/s] /workspace/tutorials-kr/intermediate_source/engine.py:30: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast(\u0027cuda\u0027, args...)` instead. Epoch: [0] [ 0/60] eta: 0:01:34 lr: 0.000090 loss: 2.9703 (2.9703) loss_classifier: 0.3432 (0.3432) loss_box_reg: 0.1413 (0.1413) loss_mask: 2.4372 (2.4372) loss_objectness: 0.0474 (0.0474) loss_rpn_box_reg: 0.0012 (0.0012) time: 1.5690 data: 0.0186 max mem: 1923 Epoch: [0] [10/60] eta: 0:00:16 lr: 0.000936 loss: 1.5786 (1.8444) loss_classifier: 0.2983 (0.2945) loss_box_reg: 0.2995 (0.2872) loss_mask: 0.8385 (1.2297) loss_objectness: 0.0300 (0.0288) loss_rpn_box_reg: 0.0038 (0.0042) time: 0.3270 data: 0.0194 max mem: 2590 Epoch: [0] [20/60] eta: 0:00:09 lr: 0.001783 loss: 0.8604 (1.2920) loss_classifier: 0.1766 (0.2155) loss_box_reg: 0.2502 (0.2794) loss_mask: 0.4121 (0.7682) loss_objectness: 0.0239 (0.0237) loss_rpn_box_reg: 0.0044 (0.0051) time: 0.1650 data: 0.0198 max mem: 2590 Epoch: [0] [30/60] eta: 0:00:05 lr: 0.002629 loss: 0.5386 (1.0609) loss_classifier: 0.0845 (0.1710) loss_box_reg: 0.2282 (0.2734) loss_mask: 0.2187 (0.5925) loss_objectness: 0.0129 (0.0192) loss_rpn_box_reg: 0.0040 (0.0048) time: 0.1235 data: 0.0199 max mem: 2590 Epoch: [0] [40/60] eta: 0:00:03 lr: 0.003476 loss: 0.4560 (0.9019) loss_classifier: 0.0506 (0.1408) loss_box_reg: 0.1597 (0.2458) loss_mask: 0.2087 (0.4952) loss_objectness: 0.0015 (0.0149) loss_rpn_box_reg: 0.0043 (0.0052) time: 0.1091 data: 0.0187 max mem: 2590 Epoch: [0] [50/60] eta: 0:00:01 lr: 0.004323 loss: 0.3754 (0.8045) loss_classifier: 0.0468 (0.1240) loss_box_reg: 0.1402 (0.2293) loss_mask: 0.1805 (0.4328) loss_objectness: 0.0015 (0.0126) loss_rpn_box_reg: 0.0062 (0.0058) time: 0.1009 data: 0.0190 max mem: 2634 Epoch: [0] [59/60] eta: 0:00:00 lr: 0.005000 loss: 0.3970 (0.7452) loss_classifier: 0.0479 (0.1133) loss_box_reg: 0.1487 (0.2184) loss_mask: 0.1736 (0.3959) loss_objectness: 0.0017 (0.0111) loss_rpn_box_reg: 0.0078 (0.0065) time: 0.0974 data: 0.0203 max mem: 2638 Epoch: [0] Total time: 0:00:08 (0.1489 s / it) creating index... index created! Test: [ 0/50] eta: 0:00:04 model_time: 0.0849 (0.0849) evaluator_time: 0.0016 (0.0016) time: 0.0932 data: 0.0064 max mem: 2638 Test: [49/50] eta: 0:00:00 model_time: 0.0185 (0.0464) evaluator_time: 0.0040 (0.0050) time: 0.0491 data: 0.0111 max mem: 2638 Test: Total time: 0:00:03 (0.0624 s / it) Averaged stats: model_time: 0.0185 (0.0464) evaluator_time: 0.0040 (0.0050) Accumulating evaluation results... DONE (t=0.01s). Accumulating evaluation results... DONE (t=0.01s). IoU metric: bbox Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.656 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.980 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.855 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.445 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.494 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.674 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.311 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.716 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.716 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.580 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.657 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.726 IoU metric: segm Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.726 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.980 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.924 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.267 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.532 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.748 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.329 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.764 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.765 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.580 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.729 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.776 Epoch: [1] [ 0/60] eta: 0:00:04 lr: 0.005000 loss: 0.3154 (0.3154) loss_classifier: 0.0460 (0.0460) loss_box_reg: 0.1283 (0.1283) loss_mask: 0.1313 (0.1313) loss_objectness: 0.0016 (0.0016) loss_rpn_box_reg: 0.0083 (0.0083) time: 0.0792 data: 0.0213 max mem: 2638 Epoch: [1] [10/60] eta: 0:00:03 lr: 0.005000 loss: 0.2622 (0.2920) loss_classifier: 0.0339 (0.0385) loss_box_reg: 0.0836 (0.0961) loss_mask: 0.1357 (0.1468) loss_objectness: 0.0016 (0.0026) loss_rpn_box_reg: 0.0077 (0.0079) time: 0.0783 data: 0.0167 max mem: 2638 Epoch: [1] [20/60] eta: 0:00:03 lr: 0.005000 loss: 0.2753 (0.3052) loss_classifier: 0.0339 (0.0390) loss_box_reg: 0.0836 (0.0997) loss_mask: 0.1450 (0.1564) loss_objectness: 0.0017 (0.0025) loss_rpn_box_reg: 0.0060 (0.0076) time: 0.0803 data: 0.0174 max mem: 2645 Epoch: [1] [30/60] eta: 0:00:02 lr: 0.005000 loss: 0.2605 (0.2878) loss_classifier: 0.0387 (0.0370) loss_box_reg: 0.0674 (0.0902) loss_mask: 0.1391 (0.1520) loss_objectness: 0.0010 (0.0021) loss_rpn_box_reg: 0.0045 (0.0065) time: 0.0880 data: 0.0179 max mem: 2645 Epoch: [1] [40/60] eta: 0:00:01 lr: 0.005000 loss: 0.2437 (0.2865) loss_classifier: 0.0378 (0.0380) loss_box_reg: 0.0732 (0.0883) loss_mask: 0.1324 (0.1519) loss_objectness: 0.0008 (0.0019) loss_rpn_box_reg: 0.0045 (0.0064) time: 0.0933 data: 0.0180 max mem: 2877 Epoch: [1] [50/60] eta: 0:00:00 lr: 0.005000 loss: 0.2518 (0.2789) loss_classifier: 0.0349 (0.0369) loss_box_reg: 0.0737 (0.0857) loss_mask: 0.1295 (0.1484) loss_objectness: 0.0006 (0.0018) loss_rpn_box_reg: 0.0045 (0.0060) time: 0.0978 data: 0.0192 max mem: 2881 Epoch: [1] [59/60] eta: 0:00:00 lr: 0.005000 loss: 0.2416 (0.2792) loss_classifier: 0.0333 (0.0366) loss_box_reg: 0.0716 (0.0852) loss_mask: 0.1318 (0.1499) loss_objectness: 0.0005 (0.0017) loss_rpn_box_reg: 0.0035 (0.0058) time: 0.0933 data: 0.0206 max mem: 2881 Epoch: [1] Total time: 0:00:05 (0.0893 s / it) creating index... index created! Test: [ 0/50] eta: 0:00:01 model_time: 0.0142 (0.0142) evaluator_time: 0.0013 (0.0013) time: 0.0221 data: 0.0064 max mem: 2881 Test: [49/50] eta: 0:00:00 model_time: 0.0152 (0.0153) evaluator_time: 0.0030 (0.0031) time: 0.0304 data: 0.0113 max mem: 2881 Test: Total time: 0:00:01 (0.0296 s / it) Averaged stats: model_time: 0.0152 (0.0153) evaluator_time: 0.0030 (0.0031) Accumulating evaluation results... DONE (t=0.01s). Accumulating evaluation results... DONE (t=0.01s). IoU metric: bbox Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.766 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.976 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.946 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.438 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.582 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.788 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.359 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.808 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.808 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.520 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.786 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.823 IoU metric: segm Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.758 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.976 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.941 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.349 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.506 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.787 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.343 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.793 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.793 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.480 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.729 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.812 That\u0027s it! So after one epoch of training, we obtain a COCO-style mAP \u003e 50, and a mask mAP of 65. But what do the predictions look like? Let\u2019s take one image in the dataset and verify import matplotlib.pyplot as plt from torchvision.utils import draw_bounding_boxes, draw_segmentation_masks image = read_image(\"data/PennFudanPed/PNGImages/FudanPed00046.png\") eval_transform = get_transform(train=False) model.eval() with torch.no_grad(): x = eval_transform(image) # convert RGBA -\u003e RGB and move to device x = x[:3, ...].to(device) predictions = model([x, ]) pred = predictions[0] image = (255.0 * (image - image.min()) / (image.max() - image.min())).to(torch.uint8) image = image[:3, ...] pred_labels = [f\"pedestrian: {score:.3f}\" for label, score in zip(pred[\"labels\"], pred[\"scores\"])] pred_boxes = pred[\"boxes\"].long() output_image = draw_bounding_boxes(image, pred_boxes, pred_labels, colors=\"red\") masks = (pred[\"masks\"] \u003e 0.7).squeeze(1) output_image = draw_segmentation_masks(output_image, masks, alpha=0.5, colors=\"blue\") plt.figure(figsize=(12, 12)) plt.imshow(output_image.permute(1, 2, 0)) \u003cmatplotlib.image.AxesImage object at 0x7f0daa4d6910\u003e The results look good! Wrapping up# In this tutorial, you have learned how to create your own training pipeline for object detection models on a custom dataset. For that, you wrote a torch.utils.data.Dataset class that returns the images and the ground truth boxes and segmentation masks. You also leveraged a Mask R-CNN model pre-trained on COCO train2017 in order to perform transfer learning on this new dataset. For a more complete example, which includes multi-machine / multi-GPU training, check references/detection/train.py, which is present in the torchvision repository. Total running time of the script: (0 minutes 52.968 seconds) Download Jupyter notebook: torchvision_tutorial.ipynb Download Python source code: torchvision_tutorial.py Download zipped: torchvision_tutorial.zip",
       "author": {
         "@type": "Organization",
         "name": "PyTorch Contributors",
         "url": "https://pytorch.org"
       },
       "image": "../_static/img/pytorch_seo.png",
       "mainEntityOfPage": {
         "@type": "WebPage",
         "@id": "/intermediate/torchvision_tutorial.html"
       },
       "datePublished": "2023-01-01T00:00:00Z",
       "dateModified": "2023-01-01T00:00:00Z"
     }