René's URL Explorer Experiment

    {
       "@context": "https://schema.org",
       "@type": "Article",
       "name": "Normalization Layers",
       "headline": "Normalization Layers",
       "description": "Normalization layers in PyTorch C++ \u2014 BatchNorm, LayerNorm, GroupNorm, InstanceNorm, and LocalResponseNorm.",
       "url": "/api/nn/normalization.html",
       "articleBody": "Normalization Layers# Normalization layers stabilize and accelerate training by normalizing intermediate activations. They help with gradient flow and allow higher learning rates. BatchNorm: Normalizes across batch dimension; most common in CNNs InstanceNorm: Normalizes each sample independently; popular in style transfer LayerNorm: Normalizes across feature dimension; standard in transformers GroupNorm: Normalizes within groups of channels; works with small batches LocalResponseNorm: Lateral inhibition inspired by neuroscience (less common today) BatchNorm1d / BatchNorm2d / BatchNorm3d# class BatchNorm1d : public torch::nn::ModuleHolder\u003cBatchNorm1dImpl\u003e# A ModuleHolder subclass for BatchNorm1dImpl. See the documentation for BatchNorm1dImpl class to learn what methods it provides, and examples of how to use BatchNorm1d with torch::nn::BatchNorm1dOptions. See the documentation for ModuleHolder to learn about PyTorch\u2019s module storage semantics. Public Types using Impl = BatchNorm1dImpl# class BatchNorm1dImpl : public torch::nn::BatchNormImplBase\u003c1, BatchNorm1dImpl\u003e# Applies the BatchNorm1d function. See https://pytorch.org/docs/main/nn.html#torch.nn.BatchNorm1d to learn about the exact behavior of this module. See the documentation for torch::nn::BatchNorm1dOptions class to learn what constructor arguments are supported for this module. Example: BatchNorm1d model(BatchNorm1dOptions(4).eps(0.5).momentum(0.1).affine(false).track_running_stats(true)); class BatchNorm2d : public torch::nn::ModuleHolder\u003cBatchNorm2dImpl\u003e# A ModuleHolder subclass for BatchNorm2dImpl. See the documentation for BatchNorm2dImpl class to learn what methods it provides, and examples of how to use BatchNorm2d with torch::nn::BatchNorm2dOptions. See the documentation for ModuleHolder to learn about PyTorch\u2019s module storage semantics. Public Types using Impl = BatchNorm2dImpl# class BatchNorm2dImpl : public torch::nn::BatchNormImplBase\u003c2, BatchNorm2dImpl\u003e# Applies the BatchNorm2d function. See https://pytorch.org/docs/main/nn.html#torch.nn.BatchNorm2d to learn about the exact behavior of this module. See the documentation for torch::nn::BatchNorm2dOptions class to learn what constructor arguments are supported for this module. Example: BatchNorm2d model(BatchNorm2dOptions(4).eps(0.5).momentum(0.1).affine(false).track_running_stats(true)); class BatchNorm3d : public torch::nn::ModuleHolder\u003cBatchNorm3dImpl\u003e# A ModuleHolder subclass for BatchNorm3dImpl. See the documentation for BatchNorm3dImpl class to learn what methods it provides, and examples of how to use BatchNorm3d with torch::nn::BatchNorm3dOptions. See the documentation for ModuleHolder to learn about PyTorch\u2019s module storage semantics. Public Types using Impl = BatchNorm3dImpl# class BatchNorm3dImpl : public torch::nn::BatchNormImplBase\u003c3, BatchNorm3dImpl\u003e# Applies the BatchNorm3d function. See https://pytorch.org/docs/main/nn.html#torch.nn.BatchNorm3d to learn about the exact behavior of this module. See the documentation for torch::nn::BatchNorm3dOptions class to learn what constructor arguments are supported for this module. Example: BatchNorm3d model(BatchNorm3dOptions(4).eps(0.5).momentum(0.1).affine(false).track_running_stats(true)); Example: auto bn = torch::nn::BatchNorm2d( torch::nn::BatchNorm2dOptions(64) // num_features .eps(1e-5) .momentum(0.1) .affine(true) .track_running_stats(true)); InstanceNorm1d / InstanceNorm2d / InstanceNorm3d# class InstanceNorm1d : public torch::nn::ModuleHolder\u003cInstanceNorm1dImpl\u003e# A ModuleHolder subclass for InstanceNorm1dImpl. See the documentation for InstanceNorm1dImpl class to learn what methods it provides, and examples of how to use InstanceNorm1d with torch::nn::InstanceNorm1dOptions. See the documentation for ModuleHolder to learn about PyTorch\u2019s module storage semantics. Public Types using Impl = InstanceNorm1dImpl# class InstanceNorm1dImpl : public torch::nn::InstanceNormImpl\u003c1, InstanceNorm1dImpl\u003e# Applies the InstanceNorm1d function. See https://pytorch.org/docs/main/nn.html#torch.nn.InstanceNorm1d to learn about the exact behavior of this module. See the documentation for torch::nn::InstanceNorm1dOptions class to learn what constructor arguments are supported for this module. Example: InstanceNorm1d model(InstanceNorm1dOptions(4).eps(0.5).momentum(0.1).affine(false).track_running_stats(true)); class InstanceNorm2d : public torch::nn::ModuleHolder\u003cInstanceNorm2dImpl\u003e# A ModuleHolder subclass for InstanceNorm2dImpl. See the documentation for InstanceNorm2dImpl class to learn what methods it provides, and examples of how to use InstanceNorm2d with torch::nn::InstanceNorm2dOptions. See the documentation for ModuleHolder to learn about PyTorch\u2019s module storage semantics. Public Types using Impl = InstanceNorm2dImpl# class InstanceNorm2dImpl : public torch::nn::InstanceNormImpl\u003c2, InstanceNorm2dImpl\u003e# Applies the InstanceNorm2d function. See https://pytorch.org/docs/main/nn.html#torch.nn.InstanceNorm2d to learn about the exact behavior of this module. See the documentation for torch::nn::InstanceNorm2dOptions class to learn what constructor arguments are supported for this module. Example: InstanceNorm2d model(InstanceNorm2dOptions(4).eps(0.5).momentum(0.1).affine(false).track_running_stats(true)); class InstanceNorm3d : public torch::nn::ModuleHolder\u003cInstanceNorm3dImpl\u003e# A ModuleHolder subclass for InstanceNorm3dImpl. See the documentation for InstanceNorm3dImpl class to learn what methods it provides, and examples of how to use InstanceNorm3d with torch::nn::InstanceNorm3dOptions. See the documentation for ModuleHolder to learn about PyTorch\u2019s module storage semantics. Public Types using Impl = InstanceNorm3dImpl# class InstanceNorm3dImpl : public torch::nn::InstanceNormImpl\u003c3, InstanceNorm3dImpl\u003e# Applies the InstanceNorm3d function. See https://pytorch.org/docs/main/nn.html#torch.nn.InstanceNorm3d to learn about the exact behavior of this module. See the documentation for torch::nn::InstanceNorm3dOptions class to learn what constructor arguments are supported for this module. Example: InstanceNorm3d model(InstanceNorm3dOptions(4).eps(0.5).momentum(0.1).affine(false).track_running_stats(true)); LayerNorm# class LayerNorm : public torch::nn::ModuleHolder\u003cLayerNormImpl\u003e# A ModuleHolder subclass for LayerNormImpl. See the documentation for LayerNormImpl class to learn what methods it provides, and examples of how to use LayerNorm with torch::nn::LayerNormOptions. See the documentation for ModuleHolder to learn about PyTorch\u2019s module storage semantics. Public Types using Impl = LayerNormImpl# class LayerNormImpl : public torch::nn::Cloneable\u003cLayerNormImpl\u003e# Applies Layer Normalization over a mini-batch of inputs as described in the paper Layer Normalization_ . See https://pytorch.org/docs/main/nn.html#torch.nn.LayerNorm to learn about the exact behavior of this module. See the documentation for torch::nn::LayerNormOptions class to learn what constructor arguments are supported for this module. Example: LayerNorm model(LayerNormOptions({2, 2}).elementwise_affine(false).eps(2e-5)); Public Functions inline LayerNormImpl(std::vector\u003cint64_t\u003e normalized_shape)# explicit LayerNormImpl(LayerNormOptions options_)# virtual void reset() override# reset() must perform initialization of all members with reference semantics, most importantly parameters, buffers and submodules. void reset_parameters()# virtual void pretty_print(std::ostream \u0026stream) const override# Pretty prints the LayerNorm module into the given stream. Tensor forward(const Tensor \u0026input)# Applies layer normalization over a mini-batch of inputs as described in the paper Layer Normalization_ . The mean and standard-deviation are calculated separately over the last certain number dimensions which have to be of the shape specified by input normalized_shape. Layer Normalization: https://arxiv.org/abs/1607.06450 Public Members LayerNormOptions options# The options with which this module was constructed. Tensor weight# The learned weight. Initialized to ones if the elementwise_affine option is set to true upon construction. Tensor bias# The learned bias. Initialized to zeros elementwise_affine option is set to true upon construction. Example: auto ln = torch::nn::LayerNorm( torch::nn::LayerNormOptions({768})); // normalized_shape GroupNorm# class GroupNorm : public torch::nn::ModuleHolder\u003cGroupNormImpl\u003e# A ModuleHolder subclass for GroupNormImpl. See the documentation for GroupNormImpl class to learn what methods it provides, and examples of how to use GroupNorm with torch::nn::GroupNormOptions. See the documentation for ModuleHolder to learn about PyTorch\u2019s module storage semantics. Public Types using Impl = GroupNormImpl# class GroupNormImpl : public torch::nn::Cloneable\u003cGroupNormImpl\u003e# Applies Group Normalization over a mini-batch of inputs as described in the paper Group Normalization_ . See https://pytorch.org/docs/main/nn.html#torch.nn.GroupNorm to learn about the exact behavior of this module. See the documentation for torch::nn::GroupNormOptions class to learn what constructor arguments are supported for this module. Example: GroupNorm model(GroupNormOptions(2, 2).eps(2e-5).affine(false)); Public Functions inline GroupNormImpl(int64_t num_groups, int64_t num_channels)# explicit GroupNormImpl(const GroupNormOptions \u0026options_)# virtual void reset() override# reset() must perform initialization of all members with reference semantics, most importantly parameters, buffers and submodules. void reset_parameters()# virtual void pretty_print(std::ostream \u0026stream) const override# Pretty prints the GroupNorm module into the given stream. Tensor forward(const Tensor \u0026input)# Public Members GroupNormOptions options# The options with which this module was constructed. Tensor weight# The learned weight. Tensor bias# The learned bias. Example: auto gn = torch::nn::GroupNorm( torch::nn::GroupNormOptions(32, 256)); // num_groups, num_channels LocalResponseNorm# class LocalResponseNorm : public torch::nn::ModuleHolder\u003cLocalResponseNormImpl\u003e# A ModuleHolder subclass for LocalResponseNormImpl. See the documentation for LocalResponseNormImpl class to learn what methods it provides, and examples of how to use LocalResponseNorm with torch::nn::LocalResponseNormOptions. See the documentation for ModuleHolder to learn about PyTorch\u2019s module storage semantics. Public Types using Impl = LocalResponseNormImpl# class LocalResponseNormImpl : public torch::nn::Cloneable\u003cLocalResponseNormImpl\u003e# Applies local response normalization over an input signal composed of several input planes, where channels occupy the second dimension. Applies normalization across channels. See https://pytorch.org/docs/main/nn.html#torch.nn.LocalResponseNorm to learn about the exact behavior of this module. See the documentation for torch::nn::LocalResponseNormOptions class to learn what constructor arguments are supported for this module. Example: LocalResponseNorm model(LocalResponseNormOptions(2).alpha(0.0002).beta(0.85).k(2.)); Public Functions inline LocalResponseNormImpl(int64_t size)# explicit LocalResponseNormImpl(const LocalResponseNormOptions \u0026options_)# Tensor forward(const Tensor \u0026input)# virtual void reset() override# reset() must perform initialization of all members with reference semantics, most importantly parameters, buffers and submodules. virtual void pretty_print(std::ostream \u0026stream) const override# Pretty prints the LocalResponseNormImpl module into the given stream. Public Members LocalResponseNormOptions options# The options with which this Module was constructed.",
       "author": {
         "@type": "Organization",
         "name": "PyTorch Contributors",
         "url": "https://pytorch.org"
       },
       "image": "https://pytorch.org/docs/stable/_static/img/pytorch_seo.png",
       "mainEntityOfPage": {
         "@type": "WebPage",
         "@id": "/api/nn/normalization.html"
       },
       "datePublished": "2023-01-01T00:00:00Z",
       "dateModified": "2023-01-01T00:00:00Z"
     }