René's URL Explorer Experiment

    {
       "@context": "https://schema.org",
       "@type": "Article",
       "name": "Gradient Descent Optimizers",
       "headline": "Gradient Descent Optimizers",
       "description": "SGD optimizer in PyTorch C++ \u2014 stochastic gradient descent with momentum and weight decay.",
       "url": "/api/optim/gradient_descent.html",
       "articleBody": "Gradient Descent Optimizers# These optimizers use gradient descent with optional enhancements like momentum. They are the foundation of neural network training and work well when you can afford careful hyperparameter tuning. SGD (Stochastic Gradient Descent)# The classic optimization algorithm. SGD with momentum is often the best choice for convolutional neural networks when properly tuned. While requiring more careful learning rate selection than adaptive methods, it frequently achieves the best final accuracy. When to use: Training CNNs (ResNet, VGG, etc.) where you want maximum accuracy When you have time for hyperparameter tuning When combined with learning rate schedules (warmup, cosine annealing) Key parameters: lr: Learning rate (typical: 0.01-0.1 for CNNs) momentum: Accelerates convergence (typical: 0.9) weight_decay: L2 regularization coefficient nesterov: Use Nesterov momentum (often improves convergence) class SGD : public torch::optim::Optimizer# Public Functions inline explicit SGD(const std::vector\u003cOptimizerParamGroup\u003e \u0026param_groups, SGDOptions defaults)# inline explicit SGD(std::vector\u003cTensor\u003e params, SGDOptions defaults)# virtual torch::Tensor step(LossClosure closure = nullptr) override# A loss function closure, which is expected to return the loss value. virtual void save(serialize::OutputArchive \u0026archive) const override# Serializes the optimizer state into the given archive. virtual void load(serialize::InputArchive \u0026archive) override# Deserializes the optimizer state from the given archive. Example: // Standard SGD with momentum - good for CNNs auto optimizer = torch::optim::SGD( model-\u003eparameters(), torch::optim::SGDOptions(0.01) // learning rate .momentum(0.9) // momentum factor .weight_decay(1e-4) // L2 regularization .nesterov(true)); // Nesterov momentum",
       "author": {
         "@type": "Organization",
         "name": "PyTorch Contributors",
         "url": "https://pytorch.org"
       },
       "image": "https://pytorch.org/docs/stable/_static/img/pytorch_seo.png",
       "mainEntityOfPage": {
         "@type": "WebPage",
         "@id": "/api/optim/gradient_descent.html"
       },
       "datePublished": "2023-01-01T00:00:00Z",
       "dateModified": "2023-01-01T00:00:00Z"
     }