René's URL Explorer Experiment

    {
       "@context": "https://schema.org",
       "@type": "Article",
       "name": "Neural Network Modules (torch::nn)",
       "headline": "Neural Network Modules (torch::nn)",
       "description": "PyTorch C++ neural network modules \u2014 torch::nn API for defining and training models.",
       "url": "/api/nn/index.html",
       "articleBody": "Neural Network Modules (torch::nn)# The torch::nn namespace provides neural network building blocks that mirror Python\u2019s torch.nn module. It uses a PIMPL (Pointer to Implementation) pattern where user-facing classes like Conv2d wrap internal Conv2dImpl classes. When to use torch::nn: Building neural network models in C++ Creating custom layers and modules Porting Python models to C++ for production inference Training models entirely in C++ Basic usage: #include \u003ctorch/torch.h\u003e // Define a simple model struct Net : torch::nn::Module { torch::nn::Conv2d conv1{nullptr}; torch::nn::Linear fc1{nullptr}; Net() { conv1 = register_module(\"conv1\", torch::nn::Conv2d( torch::nn::Conv2dOptions(1, 32, 3).stride(1).padding(1))); fc1 = register_module(\"fc1\", torch::nn::Linear(32 * 28 * 28, 10)); } torch::Tensor forward(torch::Tensor x) { x = torch::relu(conv1-\u003eforward(x)); x = x.view({-1, 32 * 28 * 28}); return fc1-\u003eforward(x); } }; // Create and use the model auto model = std::make_shared\u003cNet\u003e(); auto input = torch::randn({1, 1, 28, 28}); auto output = model-\u003eforward(input); Header Files# torch/nn.h - Main neural network header (includes all modules) torch/nn/module.h - Base Module class torch/nn/modules.h - All module implementations torch/nn/options.h - Options structs for modules torch/nn/functional.h - Functional API Module Base Class# All neural network modules inherit from torch::nn::Module, which provides parameter management, serialization, device/dtype conversion, and hooks. class Module : public std::enable_shared_from_this\u003cModule\u003e# The base class for all modules in PyTorch. .. note:: The design and implementation of this class is largely based on the Python API. You may want to consult the python documentation for :py:class:pytorch:torch.nn.Module for further clarification on certain methods or behavior. A Module is an abstraction over the implementation of some function or algorithm, possibly associated with some persistent data. A Module may contain further Modules (\u201csubmodules\u201d), each with their own implementation, persistent data and further submodules. Modules can thus be said to form a recursive tree structure. A Module is registered as a submodule to another Module by calling register_module(), typically from within a parent module\u2019s constructor. A distinction is made between three kinds of persistent data that may be associated with a Module: Parameters: tensors that record gradients, typically weights updated during the backward step (e.g. the weight of a Linear module), Buffers: tensors that do not record gradients, typically updated during the forward step, such as running statistics (e.g. mean and variance in the BatchNorm module), Any additional state, not necessarily tensors, required for the implementation or configuration of a Module. The first two kinds of state are special in that they may be registered with the Module system to allow convenient access and batch configuration. For example, registered parameters in any Module may be iterated over via the parameters() accessor. Further, changing the data type of a Module\u2019s registered parameters can be done conveniently via Module::to(), e.g. module-\u003eto(torch::kCUDA) to move all parameters to GPU memory. Lastly, registered parameters and buffers are handled specially during a clone() operation, which performs a deepcopy of a cloneable Module hierarchy. Parameters are registered with a Module via register_parameter. Buffers are registered separately via register_buffer. These methods are part of the public API of Module and are typically invoked from within a concrete Modules constructor. Subclassed by torch::nn::Cloneable\u003c SoftshrinkImpl \u003e, torch::nn::Cloneable\u003c PReLUImpl \u003e, torch::nn::Cloneable\u003c LogSoftmaxImpl \u003e, torch::nn::Cloneable\u003c L1LossImpl \u003e, torch::nn::Cloneable\u003c SequentialImpl \u003e, torch::nn::Cloneable\u003c HardshrinkImpl \u003e, torch::nn::Cloneable\u003c GLUImpl \u003e, torch::nn::Cloneable\u003c RReLUImpl \u003e, torch::nn::Cloneable\u003c ParameterDictImpl \u003e, torch::nn::Cloneable\u003c IdentityImpl \u003e, torch::nn::Cloneable\u003c FoldImpl \u003e, torch::nn::Cloneable\u003c EmbeddingBagImpl \u003e, torch::nn::Cloneable\u003c BilinearImpl \u003e, torch::nn::Cloneable\u003c TripletMarginWithDistanceLossImpl \u003e, torch::nn::Cloneable\u003c SoftminImpl \u003e, torch::nn::Cloneable\u003c SmoothL1LossImpl \u003e, torch::nn::Cloneable\u003c MultiLabelMarginLossImpl \u003e, torch::nn::Cloneable\u003c LeakyReLUImpl \u003e, torch::nn::Cloneable\u003c FunctionalImpl \u003e, torch::nn::Cloneable\u003c ELUImpl \u003e, torch::nn::Cloneable\u003c TanhshrinkImpl \u003e, torch::nn::Cloneable\u003c PairwiseDistanceImpl \u003e, torch::nn::Cloneable\u003c LogSigmoidImpl \u003e, torch::nn::Cloneable\u003c HardtanhImpl \u003e, torch::nn::Cloneable\u003c FractionalMaxPool2dImpl \u003e, torch::nn::Cloneable\u003c FlattenImpl \u003e, torch::nn::Cloneable\u003c CrossMapLRN2dImpl \u003e, torch::nn::Cloneable\u003c TransformerEncoderLayerImpl \u003e, torch::nn::Cloneable\u003c ThresholdImpl \u003e, torch::nn::Cloneable\u003c SoftsignImpl \u003e, torch::nn::Cloneable\u003c MultiMarginLossImpl \u003e, torch::nn::Cloneable\u003c FractionalMaxPool3dImpl \u003e, torch::nn::Cloneable\u003c CTCLossImpl \u003e, torch::nn::Cloneable\u003c UnfoldImpl \u003e, torch::nn::Cloneable\u003c SiLUImpl \u003e, torch::nn::Cloneable\u003c ParameterListImpl \u003e, torch::nn::Cloneable\u003c MultiheadAttentionImpl \u003e, torch::nn::Cloneable\u003c CELUImpl \u003e, torch::nn::Cloneable\u003c UpsampleImpl \u003e, torch::nn::Cloneable\u003c TransformerImpl \u003e, torch::nn::Cloneable\u003c SELUImpl \u003e, torch::nn::Cloneable\u003c PixelUnshuffleImpl \u003e, torch::nn::Cloneable\u003c LinearImpl \u003e, torch::nn::Cloneable\u003c HingeEmbeddingLossImpl \u003e, torch::nn::Cloneable\u003c EmbeddingImpl \u003e, torch::nn::Cloneable\u003c MultiLabelSoftMarginLossImpl \u003e, torch::nn::Cloneable\u003c CrossEntropyLossImpl \u003e, torch::nn::Cloneable\u003c TripletMarginLossImpl \u003e, torch::nn::Cloneable\u003c TransformerDecoderLayerImpl \u003e, torch::nn::Cloneable\u003c SoftMarginLossImpl \u003e, torch::nn::Cloneable\u003c LocalResponseNormImpl \u003e, torch::nn::Cloneable\u003c BCELossImpl \u003e, torch::nn::Cloneable\u003c LayerNormImpl \u003e, torch::nn::Cloneable\u003c AdaptiveLogSoftmaxWithLossImpl \u003e, torch::nn::Cloneable\u003c ReLUImpl \u003e, torch::nn::Cloneable\u003c ModuleListImpl \u003e, torch::nn::Cloneable\u003c HuberLossImpl \u003e, torch::nn::Cloneable\u003c GELUImpl \u003e, torch::nn::Cloneable\u003c SoftmaxImpl \u003e, torch::nn::Cloneable\u003c Softmax2dImpl \u003e, torch::nn::Cloneable\u003c SoftplusImpl \u003e, torch::nn::Cloneable\u003c SigmoidImpl \u003e, torch::nn::Cloneable\u003c PoissonNLLLossImpl \u003e, torch::nn::Cloneable\u003c ModuleDictImpl \u003e, torch::nn::Cloneable\u003c MishImpl \u003e, torch::nn::Cloneable\u003c UnflattenImpl \u003e, torch::nn::Cloneable\u003c ReLU6Impl \u003e, torch::nn::Cloneable\u003c MSELossImpl \u003e, torch::nn::Cloneable\u003c CosineSimilarityImpl \u003e, torch::nn::Cloneable\u003c CosineEmbeddingLossImpl \u003e, torch::nn::Cloneable\u003c TransformerDecoderImpl \u003e, torch::nn::Cloneable\u003c TanhImpl \u003e, torch::nn::Cloneable\u003c NLLLossImpl \u003e, torch::nn::Cloneable\u003c MarginRankingLossImpl \u003e, torch::nn::Cloneable\u003c BCEWithLogitsLossImpl \u003e, torch::nn::Cloneable\u003c TransformerEncoderImpl \u003e, torch::nn::Cloneable\u003c PixelShuffleImpl \u003e, torch::nn::Cloneable\u003c KLDivLossImpl \u003e, torch::nn::Cloneable\u003c GroupNormImpl \u003e, torch::nn::Cloneable\u003c Derived \u003e Public Types using ModuleApplyFunction = std::function\u003cvoid(Module\u0026)\u003e# using ConstModuleApplyFunction = std::function\u003cvoid(const Module\u0026)\u003e# using NamedModuleApplyFunction = std::function\u003cvoid(const std::string\u0026, Module\u0026)\u003e# using ConstNamedModuleApplyFunction = std::function\u003cvoid(const std::string\u0026, const Module\u0026)\u003e# using ModulePointerApplyFunction = std::function\u003cvoid(const std::shared_ptr\u003cModule\u003e\u0026)\u003e# using NamedModulePointerApplyFunction = std::function\u003cvoid(const std::string\u0026, const std::shared_ptr\u003cModule\u003e\u0026)\u003e# Public Functions explicit Module(std::string name)# Tells the base Module about the name of the submodule. Module()# Constructs the module without immediate knowledge of the submodule\u2019s name. The name of the submodule is inferred via RTTI (if possible) the first time .name() is invoked. Module(const Module\u0026) = default# Module \u0026operator=(const Module\u0026) = default# Module(Module\u0026\u0026) noexcept = default# Module \u0026operator=(Module\u0026\u0026) noexcept = default# virtual ~Module() = default# const std::string \u0026name() const noexcept# Returns the name of the Module. A Module has an associated name, which is a string representation of the kind of concrete Module it represents, such as \"Linear\" for the Linear module. Under most circumstances, this name is automatically inferred via runtime type information (RTTI). In the unusual circumstance that you have this feature disabled, you may want to manually name your Modules by passing the string name to the Module base class\u2019 constructor. virtual std::shared_ptr\u003cModule\u003e clone(const std::optional\u003cDevice\u003e \u0026device = std::nullopt) const# Performs a recursive deep copy of the module and all its registered parameters, buffers and submodules. Optionally, this method sets the current device to the one supplied before cloning. If no device is given, each parameter and buffer will be moved to the device of its source. .. attention:: Attempting to call the clone() method inherited from the base Module class (the one documented here) will fail. To inherit an actual implementation of clone(), you must subclass Cloneable. Cloneable is templatized on the concrete module type, and can thus properly copy a Module. This method is provided on the base class\u2019 API solely for an easier-to-use polymorphic interface. void apply(const ModuleApplyFunction \u0026function)# Applies the function to the Module and recursively to every submodule. The function must accept a Module\u0026. .. code-block:: cpp MyModule module; module-\u003eapply([](nn::Module\u0026 module) { std::cout \u003c\u003c module.name() \u003c\u003c std::endl; }); void apply(const ConstModuleApplyFunction \u0026function) const# Applies the function to the Module and recursively to every submodule. The function must accept a const Module\u0026. .. code-block:: cpp MyModule module; module-\u003eapply([](const nn::Module\u0026 module) { std::cout \u003c\u003c module.name() \u003c\u003c std::endl; }); void apply(const NamedModuleApplyFunction \u0026function, const std::string \u0026name_prefix = std::string())# Applies the function to the Module and recursively to every submodule. The function must accept a const std::string\u0026 for the key of the module, and a Module\u0026. The key of the module itself is the empty string. If name_prefix is given, it is prepended to every key as \u003cname_prefix\u003e.\u003ckey\u003e (and just name_prefix for the module itself). .. code-block:: cpp MyModule module; module-\u003eapply([](const std::string\u0026 key, nn::Module\u0026 module) { std::cout \u003c\u003c key \u003c\u003c \": \" \u003c\u003c module.name() \u003c\u003c std::endl; }); void apply(const ConstNamedModuleApplyFunction \u0026function, const std::string \u0026name_prefix = std::string()) const# Applies the function to the Module and recursively to every submodule. The function must accept a const std::string\u0026 for the key of the module, and a const Module\u0026. The key of the module itself is the empty string. If name_prefix is given, it is prepended to every key as \u003cname_prefix\u003e.\u003ckey\u003e (and just name_prefix for the module itself). .. code-block:: cpp MyModule module; module-\u003eapply([](const std::string\u0026 key, const nn::Module\u0026 module) { std::cout \u003c\u003c key \u003c\u003c \": \" \u003c\u003c module.name() \u003c\u003c std::endl; }); void apply(const ModulePointerApplyFunction \u0026function) const# Applies the function to the Module and recursively to every submodule. The function must accept a const std::shared_ptr\u003cModule\u003e\u0026. .. code-block:: cpp MyModule module; module-\u003eapply([](const std::shared_ptr\u003cnn::Module\u003e\u0026 module) { std::cout \u003c\u003c module-\u003ename() \u003c\u003c std::endl; }); void apply(const NamedModulePointerApplyFunction \u0026function, const std::string \u0026name_prefix = std::string()) const# Applies the function to the Module and recursively to every submodule. The function must accept a const std::string\u0026 for the key of the module, and a const std::shared_ptr\u003cModule\u003e\u0026. The key of the module itself is the empty string. If name_prefix is given, it is prepended to every key as \u003cname_prefix\u003e.\u003ckey\u003e (and just name_prefix for the module itself). .. code-block:: cpp MyModule module; module-\u003eapply([](const std::string\u0026 key, const std::shared_ptr\u003cnn::Module\u003e\u0026 module) { std::cout \u003c\u003c key \u003c\u003c \": \" \u003c\u003c module-\u003ename() \u003c\u003c std::endl; }); std::vector\u003cTensor\u003e parameters(bool recurse = true) const# Returns the parameters of this Module and if recurse is true, also recursively of every submodule. OrderedDict\u003cstd::string, Tensor\u003e named_parameters(bool recurse = true) const# Returns an OrderedDict with the parameters of this Module along with their keys, and if recurse is true also recursively of every submodule. std::vector\u003cTensor\u003e buffers(bool recurse = true) const# Returns the buffers of this Module and if recurse is true, also recursively of every submodule. OrderedDict\u003cstd::string, Tensor\u003e named_buffers(bool recurse = true) const# Returns an OrderedDict with the buffers of this Module along with their keys, and if recurse is true also recursively of every submodule. std::vector\u003cstd::shared_ptr\u003cModule\u003e\u003e modules(bool include_self = true) const# Returns the submodules of this Module (the entire submodule hierarchy) and if include_self is true, also inserts a shared_ptr to this module in the first position. .. warning:: Only pass include_self as true if this Module is stored in a shared_ptr! Otherwise an exception will be thrown. You may still call this method with include_self set to false if your Module is not stored in a shared_ptr. OrderedDict\u003cstd::string, std::shared_ptr\u003cModule\u003e\u003e named_modules(const std::string \u0026name_prefix = std::string(), bool include_self = true) const# Returns an OrderedDict of the submodules of this Module (the entire submodule hierarchy) and their keys, and if include_self is true, also inserts a shared_ptr to this module in the first position. If name_prefix is given, it is prepended to every key as \u003cname_prefix\u003e.\u003ckey\u003e (and just name_prefix for the module itself). .. warning:: Only pass include_self as true if this Module is stored in a shared_ptr! Otherwise an exception will be thrown. You may still call this method with include_self set to false if your Module is not stored in a shared_ptr. std::vector\u003cstd::shared_ptr\u003cModule\u003e\u003e children() const# Returns the direct submodules of this Module. OrderedDict\u003cstd::string, std::shared_ptr\u003cModule\u003e\u003e named_children() const# Returns an OrderedDict of the direct submodules of this Module and their keys. virtual void train(bool on = true)# Enables \u201ctraining\u201d mode. void eval()# Calls train(false) to enable \u201ceval\u201d mode. Do not override this method, override train() instead. virtual bool is_training() const noexcept# True if the module is in training mode. Every Module has a boolean associated with it that determines whether the Module is currently in training mode (set via .train()) or in evaluation (inference) mode (set via .eval()). This property is exposed via is_training(), and may be used by the implementation of a concrete module to modify its runtime behavior. See the BatchNorm or Dropout modules for examples of Modules that use different code paths depending on this property. virtual void to(torch::Device device, torch::Dtype dtype, bool non_blocking = false)# Recursively casts all parameters to the given dtype and device. If non_blocking is true and the source is in pinned memory and destination is on the GPU or vice versa, the copy is performed asynchronously with respect to the host. Otherwise, the argument has no effect. virtual void to(torch::Dtype dtype, bool non_blocking = false)# Recursively casts all parameters to the given dtype. If non_blocking is true and the source is in pinned memory and destination is on the GPU or vice versa, the copy is performed asynchronously with respect to the host. Otherwise, the argument has no effect. virtual void to(torch::Device device, bool non_blocking = false)# Recursively moves all parameters to the given device. If non_blocking is true and the source is in pinned memory and destination is on the GPU or vice versa, the copy is performed asynchronously with respect to the host. Otherwise, the argument has no effect. virtual void zero_grad(bool set_to_none = true)# Recursively zeros out the grad value of each registered parameter. template\u003ctypename ModuleType\u003eModuleType::ContainedType *as() noexcept# Attempts to cast this Module to the given ModuleType. This method is useful when calling apply(). .. code-block:: cpp void initialize_weights(nn::Module\u0026 module) { torch::NoGradGuard no_grad; if (auto* linear = module.as\u003cnn::Linear\u003e()) { linear-\u003eweight.normal_(0.0, 0.02); } } MyModule module; module-\u003eapply(initialize_weights); template\u003ctypename ModuleType\u003econst ModuleType::ContainedType *as() const noexcept# Attempts to cast this Module to the given ModuleType. This method is useful when calling apply(). .. code-block:: cpp void initialize_weights(nn::Module\u0026 module) { torch::NoGradGuard no_grad; if (auto* linear = module.as\u003cnn::Linear\u003e()) { linear-\u003eweight.normal_(0.0, 0.02); } } MyModule module; module-\u003eapply(initialize_weights); template\u003ctypename ModuleType, typename = torch::detail::disable_if_module_holder_t\u003cModuleType\u003e\u003eModuleType *as() noexcept# Attempts to cast this Module to the given ModuleType. This method is useful when calling apply(). .. code-block:: cpp void initialize_weights(nn::Module\u0026 module) { torch::NoGradGuard no_grad; if (auto* linear = module.as\u003cnn::Linear\u003e()) { linear-\u003eweight.normal_(0.0, 0.02); } } MyModule module; module.apply(initialize_weights); template\u003ctypename ModuleType, typename = torch::detail::disable_if_module_holder_t\u003cModuleType\u003e\u003econst ModuleType *as() const noexcept# Attempts to cast this Module to the given ModuleType. This method is useful when calling apply(). .. code-block:: cpp void initialize_weights(nn::Module\u0026 module) { torch::NoGradGuard no_grad; if (auto* linear = module.as\u003cnn::Linear\u003e()) { linear-\u003eweight.normal_(0.0, 0.02); } } MyModule module; module.apply(initialize_weights); virtual void save(serialize::OutputArchive \u0026archive) const# Serializes the Module into the given OutputArchive. If the Module contains unserializable submodules (e.g. nn::Functional), those submodules are skipped when serializing. virtual void load(serialize::InputArchive \u0026archive)# Deserializes the Module from the given InputArchive. If the Module contains unserializable submodules (e.g. nn::Functional), we don\u2019t check the existence of those submodules in the InputArchive when deserializing. virtual void pretty_print(std::ostream \u0026stream) const# Streams a pretty representation of the Module into the given stream. By default, this representation will be the name of the module (taken from name()), followed by a recursive pretty print of all of the Module\u2019s submodules. Override this method to change the pretty print. The input stream should be returned from the method, to allow easy chaining. virtual bool is_serializable() const# Returns whether the Module is serializable. Tensor \u0026register_parameter(std::string name, Tensor tensor, bool requires_grad = true)# Registers a parameter with this Module. A parameter should be any gradient-recording tensor used in the implementation of your Module. Registering it makes it available to methods such as parameters(), clone() or to(). Note that registering an undefined Tensor (e.g. module.register_parameter(\"param\", Tensor())) is allowed, and is equivalent to module.register_parameter(\"param\", None) in Python API. .. code-block:: cpp MyModule::MyModule() { weight_ = register_parameter(\"weight\", torch::randn({A, B})); } Tensor \u0026register_buffer(std::string name, Tensor tensor)# Registers a buffer with this Module. A buffer is intended to be state in your module that does not record gradients, such as running statistics. Registering it makes it available to methods such as buffers(), clone() or to(). .. code-block:: cpp MyModule::MyModule() { mean_ = register_buffer(\"mean\", torch::empty({num_features_})); } template\u003ctypename ModuleType\u003estd::shared_ptr\u003cModuleType\u003e register_module(std::string name, std::shared_ptr\u003cModuleType\u003e module)# Registers a submodule with this Module. Registering a module makes it available to methods such as modules(), clone() or to(). .. code-block:: cpp MyModule::MyModule() { submodule_ = register_module(\"linear\", torch::nn::Linear(3, 4)); } template\u003ctypename ModuleType\u003estd::shared_ptr\u003cModuleType\u003e register_module(std::string name, ModuleHolder\u003cModuleType\u003e module_holder)# Registers a submodule with this Module. This method deals with ModuleHolders. Registering a module makes it available to methods such as modules(), clone() or to(). .. code-block:: cpp MyModule::MyModule() { submodule_ = register_module(\"linear\", torch::nn::Linear(3, 4)); } template\u003ctypename ModuleType\u003estd::shared_ptr\u003cModuleType\u003e replace_module(const std::string \u0026name, std::shared_ptr\u003cModuleType\u003e module)# Replaces a registered submodule with this Module. This takes care of the registration, if you used submodule members, you should assign the submodule as well, i.e. use as module-\u003esubmodule_ = module-\u003ereplace_module(\u201clinear\u201d, torch::nn::Linear(3, 4)); It only works when a module of the name is already registered. This is useful for replacing a module after initialization, e.g. for finetuning. template\u003ctypename ModuleType\u003estd::shared_ptr\u003cModuleType\u003e replace_module(const std::string \u0026name, ModuleHolder\u003cModuleType\u003e module_holder)# Replaces a registered submodule with this Module. This method deals with ModuleHolders. This takes care of the registration, if you used submodule members, you should assign the submodule as well, i.e. use as module-\u003esubmodule_ = module-\u003ereplace_module(\u201clinear\u201d, linear_holder); It only works when a module of the name is already registered. This is useful for replacing a module after initialization, e.g. for finetuning. void unregister_module(const std::string \u0026name)# Unregisters a submodule from this Module. If there is no such module with name an exception is thrown. Key features: register_module(): Register submodules for parameter tracking register_parameter(): Register learnable parameters register_buffer(): Register non-learnable state (e.g., running mean) parameters() / named_parameters(): Iterate over all parameters to(): Move module to a device or convert dtype train() / eval(): Toggle training/evaluation mode save() / load(): Serialize and deserialize module state Module Categories# Containers Convolution Layers Pooling Layers Linear Layers Activation Functions Normalization Layers Dropout Layers Embedding Layers Recurrent Layers Transformer Layers Loss Functions Functional API Utilities",
       "author": {
         "@type": "Organization",
         "name": "PyTorch Contributors",
         "url": "https://pytorch.org"
       },
       "image": "https://pytorch.org/docs/stable/_static/img/pytorch_seo.png",
       "mainEntityOfPage": {
         "@type": "WebPage",
         "@id": "/api/nn/index.html"
       },
       "datePublished": "2023-01-01T00:00:00Z",
       "dateModified": "2023-01-01T00:00:00Z"
     }