Events

Nuclear Norm Regularization for Deep Learning

Speaker: Christopher Scarvelis, MIT

Location: 60 Fifth Avenue, Room 650

Date: Friday, May 2, 2025

Penalizing the nuclear norm of a function’s Jacobian encourages it to locally behave like a low-rank linear map. Such functions vary locally along only a few directions, making the Jacobian nuclear norm a natural regularizer for machine learning problems whose data lies on a low-dimensional manifold in high-dimensional space. However, this regularizer is intractable in high dimensions, as it requires computing a large Jacobian matrix and taking its SVD. We show how to efficiently penalize the Jacobian nuclear norm
using techniques tailor-made for deep learning. Our key insight is that for functions parametrized as compositions – including all non-trivial neural networks – one may replace the expensive Jacobian nuclear norm computation with an average of layer-wise squared
Frobenius norms while exactly preserving the optimal solution. We then propose a denoising-style approximation that avoids Jacobian computations altogether. Our method is simple, efficient, and accurate, enabling Jacobian nuclear norm regularization to scale
to high-dimensional deep learning problems such as denoising and unsupervised representation learning