Events

NLP/Text-As-Data: "Progress in Dynamic Adversarial Data Collection"

Speaker: Douwe Kiela

Location: 60 Fifth Avenue, Room 7th Floor Open Space

Date: Thursday, April 21, 2022

The current benchmarking paradigm in AI has many issues: benchmarks saturate quickly, are susceptible to overfitting, contain exploitable annotator artifacts, have unclear or imperfect evaluation metrics, and do not necessarily measure what we really care about. I will talk about our work in trying to rethink the way we do benchmarking in AI, specifically in natural language processing, focusing mostly on the Dynabench platform (dynabench.org), launched in the fall of 2020. In particular, I'll cover some recent work coming out of the Dynabench team that investigates various aspects of dynamic adversarial data collection. An alternative title would have been "humans and models in loops".