Events

Global AI Frontier Lab: Hyperparameter Loss Surfaces Are Simple Near their Optima

Speaker: Nicholas Lourie

Location: 1 MetroTech Center, Room Floor 22

Date: Monday, September 22, 2025

We are pleased to announce the next fall session of the Global AI Frontier Lab: Seminar Series on September 22, 2025. Nicholas Lourie will be presenting on "Hyperparameter Loss Surfaces Are Simple Near their Optima". Dinner & networking will begin at 6:00 PM and the seminar will start at 7:00 PM EST. The seminar will be held at the Global AI Frontier Lab at 1 Metrotech Center, Brooklyn, NY 11201. This event will be in-person & online. In-person attendance is strongly encouraged for Lab researchers in NYC. Please RSVP by filling out this Google Form. For online attendees, a Zoom link will be sent out prior to the event. Please reach out to global-ai-frontier-lab@nyu.edu with any questions. We hope to see you there!

Abstract: Hyperparameters greatly impact models' capabilities; however, modern models are too large for extensive search. Instead, researchers design recipes that train well across scales based on their understanding of the hyperparameters. Unfortunately, few tools exist for understanding the hyperparameter loss surface. We discover novel structure in it that enables a new theory yielding such tools. This theory centers on a new family of probability distributions: the noisy quadratic. Its parameters correspond to the most important features of the hyperparameter loss surface, such as the best possible loss and its effective dimension. From these features, we derive an asymptotic law for random search that explains and extrapolates its convergence. These new tools enable new analyses, such as confidence intervals for the best possible performance or determining the effective number of hyperparameters. Join us and learn how to apply these tools in your own research! You can find them today at https://github.com/nicholaslourie/opda.

Bio: Nick Lourie takes seriously the project of building intelligent machines. By building them, he hopes to better understand how to make decisions and learn about the world. He's held roles across the machine learning lifecycle from software engineering to basic research. Previously, he investigated machine ethics, common sense, prompting, and the evaluation of natural language processing models at the Allen Institute for AI. Later, he applied deep learning to financial markets at Two Sigma Investments. Currently, he is pursuing a PhD at NYU advised by He He and Kyunghyun Cho, where he seeks to develop better statistical frameworks for designing, developing, and evaluating neural networks.