René's URL Explorer Experiment

    {
       "@context": "https://schema.org",
       "@type": "Article",
       "name": "Second-Order Optimizers",
       "headline": "Second-Order Optimizers",
       "description": "Second-order optimizers in PyTorch C++ \u2014 LBFGS optimizer for full-batch optimization.",
       "url": "/api/optim/second_order.html",
       "articleBody": "Second-Order Optimizers# Second-order methods use curvature information (Hessian or its approximations) to make better optimization steps. They can converge faster but are more computationally expensive and memory-intensive. LBFGS (Limited-memory Broyden-Fletcher-Goldfarb-Shanno)# LBFGS is a quasi-Newton method that approximates the inverse Hessian using gradient history. It can converge much faster than first-order methods for smooth, convex-like loss surfaces. When to use: Small models where memory isn\u2019t a concern Fine-tuning pre-trained models Convex or near-convex optimization problems Full-batch training (not mini-batch) Key parameters: lr: Learning rate (often 1.0 for LBFGS) max_iter: Maximum iterations per step history_size: Number of past gradients to store Important: LBFGS requires a closure function that recomputes the loss. class LBFGS : public torch::optim::Optimizer# Public Functions inline explicit LBFGS(const std::vector\u003cOptimizerParamGroup\u003e \u0026param_groups, LBFGSOptions defaults = {})# inline explicit LBFGS(std::vector\u003cTensor\u003e params, LBFGSOptions defaults = {})# virtual Tensor step(LossClosure closure) override# A loss function closure, which is expected to return the loss value. virtual void save(serialize::OutputArchive \u0026archive) const override# Serializes the optimizer state into the given archive. virtual void load(serialize::InputArchive \u0026archive) override# Deserializes the optimizer state from the given archive. Example: auto optimizer = torch::optim::LBFGS( model-\u003eparameters(), torch::optim::LBFGSOptions(1.0) .max_iter(20) .history_size(10)); // LBFGS requires a closure that recomputes the model for (int epoch = 0; epoch \u003c num_epochs; ++epoch) { auto closure = [\u0026]() { optimizer.zero_grad(); auto output = model-\u003eforward(data); auto loss = loss_fn(output, target); loss.backward(); return loss; }; optimizer.step(closure); }",
       "author": {
         "@type": "Organization",
         "name": "PyTorch Contributors",
         "url": "https://pytorch.org"
       },
       "image": "https://pytorch.org/docs/stable/_static/img/pytorch_seo.png",
       "mainEntityOfPage": {
         "@type": "WebPage",
         "@id": "/api/optim/second_order.html"
       },
       "datePublished": "2023-01-01T00:00:00Z",
       "dateModified": "2023-01-01T00:00:00Z"
     }