René's URL Explorer Experiment

    {
       "@context": "https://schema.org",
       "@type": "Article",
       "name": "Optimizers (torch::optim)",
       "headline": "Optimizers (torch::optim)",
       "description": "PyTorch C++ optimizer API \u2014 SGD, Adam, and other optimizers for training neural networks.",
       "url": "/api/optim/index.html",
       "articleBody": "Optimizers (torch::optim)# The torch::optim namespace provides optimization algorithms for training neural networks. These optimizers update model parameters based on computed gradients to minimize the loss function. When to use torch::optim: When training neural networks with gradient descent When you need different optimization strategies (SGD, Adam, etc.) When implementing learning rate schedules Basic usage: #include \u003ctorch/torch.h\u003e // Create model and optimizer auto model = std::make_shared\u003cNet\u003e(); auto optimizer = torch::optim::Adam( model-\u003eparameters(), torch::optim::AdamOptions(1e-3)); // Training loop for (auto\u0026 batch : *data_loader) { optimizer.zero_grad(); // Clear gradients auto loss = loss_fn(model-\u003eforward(batch.data), batch.target); loss.backward(); // Compute gradients optimizer.step(); // Update parameters } Header Files# torch/csrc/api/include/torch/optim.h - Main optim header torch/csrc/api/include/torch/optim/optimizer.h - Optimizer base class torch/csrc/api/include/torch/optim/sgd.h - SGD optimizer torch/csrc/api/include/torch/optim/adam.h - Adam optimizer Optimizer Base Class# All optimizers inherit from the Optimizer base class, which provides common functionality for parameter updates, gradient zeroing, and state management. class Optimizer# Subclassed by torch::optim::Adagrad, torch::optim::Adam, torch::optim::AdamW, torch::optim::LBFGS, torch::optim::RMSprop, torch::optim::SGD Public Types using LossClosure = std::function\u003cTensor()\u003e# Public Functions Optimizer(const Optimizer \u0026optimizer) = delete# Optimizer(Optimizer \u0026\u0026optimizer) = default# Optimizer \u0026operator=(const Optimizer \u0026optimizer) = delete# Optimizer \u0026operator=(Optimizer \u0026\u0026optimizer) = default# inline explicit Optimizer(const std::vector\u003cOptimizerParamGroup\u003e \u0026param_groups, std::unique_ptr\u003cOptimizerOptions\u003e defaults)# inline explicit Optimizer(std::vector\u003cTensor\u003e parameters, std::unique_ptr\u003cOptimizerOptions\u003e defaults)# Constructs the Optimizer from a vector of parameters. void add_param_group(const OptimizerParamGroup \u0026param_group)# Adds the given param_group to the optimizer\u2019s param_group list. virtual ~Optimizer() = default# virtual Tensor step(LossClosure closure = nullptr) = 0# A loss function closure, which is expected to return the loss value. void add_parameters(const std::vector\u003cTensor\u003e \u0026parameters)# Adds the given vector of parameters to the optimizer\u2019s parameter list. void zero_grad(bool set_to_none = true)# Zeros out the gradients of all parameters. const std::vector\u003cTensor\u003e \u0026parameters() const noexcept# Provides a const reference to the parameters in the first param_group this optimizer holds. std::vector\u003cTensor\u003e \u0026parameters() noexcept# Provides a reference to the parameters in the first param_group this optimizer holds. size_t size() const noexcept# Returns the number of parameters referenced by the optimizer. OptimizerOptions \u0026defaults() noexcept# const OptimizerOptions \u0026defaults() const noexcept# std::vector\u003cOptimizerParamGroup\u003e \u0026param_groups() noexcept# Provides a reference to the param_groups this optimizer holds. const std::vector\u003cOptimizerParamGroup\u003e \u0026param_groups() const noexcept# Provides a const reference to the param_groups this optimizer holds. ska::flat_hash_map\u003cvoid*, std::unique_ptr\u003cOptimizerParamState\u003e\u003e \u0026state() noexcept# Provides a reference to the state this optimizer holds. const ska::flat_hash_map\u003cvoid*, std::unique_ptr\u003cOptimizerParamState\u003e\u003e \u0026state() const noexcept# Provides a const reference to the state this optimizer holds. virtual void save(serialize::OutputArchive \u0026archive) const# Serializes the optimizer state into the given archive. virtual void load(serialize::InputArchive \u0026archive)# Deserializes the optimizer state from the given archive. OptimizerOptions# class OptimizerOptions# Public Functions OptimizerOptions() = default# OptimizerOptions(const OptimizerOptions\u0026) = default# OptimizerOptions \u0026operator=(const OptimizerOptions\u0026) = default# OptimizerOptions(OptimizerOptions\u0026\u0026) noexcept = default# OptimizerOptions \u0026operator=(OptimizerOptions\u0026\u0026) noexcept = default# virtual std::unique_ptr\u003cOptimizerOptions\u003e clone() const# virtual void serialize(torch::serialize::InputArchive \u0026archive)# virtual void serialize(torch::serialize::OutputArchive \u0026archive) const# virtual ~OptimizerOptions() = default# virtual double get_lr() const# virtual void set_lr(const double lr)# OptimizerParamGroup# class OptimizerParamGroup# Stores parameters in the param_group and stores a pointer to the OptimizerOptions. Public Functions inline OptimizerParamGroup(const OptimizerParamGroup \u0026param_group)# OptimizerParamGroup(OptimizerParamGroup \u0026\u0026param_group) = default# inline OptimizerParamGroup(std::vector\u003cTensor\u003e params)# inline OptimizerParamGroup(std::vector\u003cTensor\u003e params, std::unique_ptr\u003cOptimizerOptions\u003e options)# OptimizerParamGroup \u0026operator=(const OptimizerParamGroup \u0026param_group) = delete# OptimizerParamGroup \u0026operator=(OptimizerParamGroup \u0026\u0026param_group) noexcept = default# ~OptimizerParamGroup() = default# bool has_options() const# OptimizerOptions \u0026options()# const OptimizerOptions \u0026options() const# void set_options(std::unique_ptr\u003cOptimizerOptions\u003e options)# std::vector\u003cTensor\u003e \u0026params()# const std::vector\u003cTensor\u003e \u0026params() const# OptimizerParamState# class OptimizerParamState# Public Functions OptimizerParamState() = default# OptimizerParamState(const OptimizerParamState\u0026) = default# OptimizerParamState \u0026operator=(const OptimizerParamState\u0026) = default# OptimizerParamState(OptimizerParamState\u0026\u0026) noexcept = default# OptimizerParamState \u0026operator=(OptimizerParamState\u0026\u0026) noexcept = default# virtual std::unique_ptr\u003cOptimizerParamState\u003e clone() const# virtual void serialize(torch::serialize::InputArchive \u0026archive)# virtual void serialize(torch::serialize::OutputArchive \u0026archive) const# virtual ~OptimizerParamState() = default# Choosing an Optimizer# Selecting the right optimizer depends on your model architecture, dataset, and training requirements: Optimizer Best For Trade-offs SGD + Momentum CNNs, well-understood problems, when you can tune hyperparameters Requires careful learning rate tuning; often achieves best final accuracy Adam/AdamW General-purpose, transformers, quick prototyping Works well out-of-the-box; AdamW preferred with weight decay RMSprop RNNs, non-stationary objectives Good for recurrent architectures; handles varying gradient scales Adagrad Sparse data (NLP, embeddings) Learning rate decreases over time; good for infrequent features LBFGS Small models, fine-tuning, convex problems Memory-intensive; requires closure function Optimizer Categories# Gradient Descent Optimizers Adaptive Learning Rate Optimizers Second-Order Optimizers Learning Rate Schedulers",
       "author": {
         "@type": "Organization",
         "name": "PyTorch Contributors",
         "url": "https://pytorch.org"
       },
       "image": "https://pytorch.org/docs/stable/_static/img/pytorch_seo.png",
       "mainEntityOfPage": {
         "@type": "WebPage",
         "@id": "/api/optim/index.html"
       },
       "datePublished": "2023-01-01T00:00:00Z",
       "dateModified": "2023-01-01T00:00:00Z"
     }