Events

Special ECE Seminar Series Modern Artificial Intelligence: Spectral Transformers

Speaker: Elad Hazan, Princeton University

Location: 370 Jay Street, Room 1201
Videoconference link: https://nyu.zoom.us/meeting/register/tJMsdeioqz0pE9LeSIp0lfjHTbgLUviF9Zr6#/registration

Date: Tuesday, November 26, 2024

We'll discuss a new technique for sequence modeling for prediction tasks with long range dependencies and fast inference/generation. At the heart of the method is a new formulation for state space models (SSMs) based on learning linear dynamical systems with the spectral filtering algorithm. This gives rise to a novel sequence prediction architecture we call a spectral state space model. Spectral state space models have two primary advantages. First, they have provable robustness properties as their performance depends on neither the spectrum of the underlying dynamics nor the dimensionality of the problem. Second, these models are constructed with fixed convolutional filters that do not require learning while still outperforming SSMs in both theory and practice. The resulting models are evaluated on synthetic dynamical systems as well as long-range prediction tasks of various modalities. These evaluations support the theoretical benefits of spectral filtering for tasks requiring very long range memory. We will also discuss recent work showing fast generation and provable length generalization of these models.