Julia Kempe

TL;DR

We show that LLMs stuck on sparse-reward, difficult math problems can self-improve by learning to generate a "stepping-stone" curriculum with meta-RL. This allows models to improve over reasoning plateaus without curated intermediate data or unstable intrinsic rewards.

When large language models try to show their work, things can go wrong. These models often improve at solving math or logic problems when they generate intermediate steps — a method known as Chain-of-Thought (CoT) prompting — but learning how to do that can make them rigid.

Rather than forcing the model to commit to one path of reasoning during training, this work lets it explore many possibilities at once using "soft" tokens — blurry combinations of multiple words or ideas instead of a single, fixed one. During inference, however, the CoT is generated in normal hard text. That combination — training soft, inferring hard — turned out to be the most effective.

In the race to build smarter AI systems, a counterintuitive pattern has emerged: models that generate shorter reasoning chains often arrive at correct answers more reliably than those that think longer. CDS PhD student Yunzhen Feng and his collaborators set out to understand why.

The results contradicted the "longer is better" narrative. When controlling for the question, it's actually the shorter, the better. The team discovered that abandoned reasoning branches — paths the model explored but eventually rejected — were the strongest predictor of incorrect answers.

This work provides new theoretical insights into how reinforcement learning after next-token prediction helps language models develop better reasoning capabilities. The research explains why this approach facilitates learning and when it provides the most benefit.

Understanding where and how concepts are represented in transformer models is crucial for interpretability and safety. This work introduces a concept-agnostic method for discovering attention modules in transformers, revealing how models organize information from safety guidelines to object recognition to mathematical reasoning.

Dataset pruning promises to reduce training costs, but traditional approaches can inadvertently harm model performance on underrepresented groups. DRoP introduces a distributionally robust approach to data pruning that maintains fairness while achieving efficient data reduction.

From a statistical perspective, jailbreaking LLMs may be fundamentally unavoidable. This work provides theoretical analysis of why language models remain vulnerable to adversarial attacks and proposes new approaches to safeguarding AI systems despite these inherent limitations.

Adversarial training is the gold standard for making AI models robust to attacks, but it comes with hidden costs. This research examines the implicit bias in adversarially robust generalization and reveals fundamental tradeoffs in achieving both security and performance.

As AI models increasingly train on synthetic data, a phenomenon called "model collapse" threatens their performance. This work provides new theoretical insights into model collapse as a change of scaling laws and offers solutions for maintaining model quality when training on generated data.

This research was featured in the New York Times.