Events

Designing and Evaluating Language Models for Human Interaction

Speaker: Mina Lee

Location: 60 Fifth Avenue, Room 150

Date: Tuesday, March 14, 2023

Despite the recent advancements in language models (LMs), most LMs are not optimized for, nor are they evaluated on, real-world usage with human interaction. In this talk, I will present my research on designing and evaluating LMs for human-LM interaction. Concretely, I will first describe how we can support human editing needs by enabling any LM to perform text infilling at any position in a document (i.e., “fill in the blanks”). I will then introduce CoAuthor, a platform for capturing human-LM interaction in collaborative writing as rich, replayable, keystroke-level interaction traces. With the platform, I demonstrate how collecting a large interaction dataset and analyzing the traces provide unique insights into LM capabilities regarding language, ideation, and collaboration. Lastly, I will propose a new framework, HALIE (Human-AI Language-based Interaction Evaluation), that defines the components of interactive systems and evaluation metrics for human-LM interaction beyond writing. I will conclude by discussing open challenges and future directions in this field.