Events

CILVR seminar: Tuning-Free Stochastic Optimization for Machine Learning

Speaker: Ahmed Khaled

Location: 60 Fifth Avenue, Room 7th floor open space
Videoconference link: https://nyu.zoom.us/s/97703722860

Date: Tuesday, April 29, 2025

Hyperparameter tuning is too costly to use in today’s large-scale machine learning problems, and we don’t really know how to do it well. We formalize the notion of “tuning-free” algorithms that can match the performance of optimally-tuned optimization algorithms up to polylogarithmic factors given only loose hints on the relevant problem parameters. We consider in particular algorithms that can match optimally-tuned Stochastic Gradient Descent (SGD). When the domain of optimization is bounded, we show tuning-free matching of SGD is possible and achieved by several existing algorithms. We prove that for the task of minimizing a convex and smooth or Lipschitz function over an unbounded domain, tuning-free optimization is impossible. We discuss conditions under which tuning-free optimization is possible even over unbounded domains. For the task of finding a stationary point of a smooth and potentially nonconvex function, we give a variant of SGD that matches the best-known high-probability convergence rate for tuned SGD at only an additional polylogarithmic cost. However, we also give an impossibility result that shows no algorithm can hope to match the optimal expected convergence rate for tuned SGD with high probability. Finally, I will also discuss ongoing work in making tuning-free distributed & adaptive algorithms.