Events

Understanding the Physical World from Images

Speaker: David Fouhey

Location: 60 Fifth Avenue, Room 150
Videoconference link: https://nyu.zoom.us/j/98384212506

Date: Friday, March 17, 2023

To us, a photo can be effortlessly understood as a rich, 3D and physical world in which we can act. Our understanding goes far beyond sensing the visible surfaces and recognizing and naming objects as they currently exist.  We seamlessly plan to navigate, even in areas that we cannot fully see, and know how to interact with objects even when they are out of our reach. Despite our progress in computer vision, this level of understanding is still beyond computers. My research aims to address this gap, and I believe such an understanding will be critical for autonomous agents like robots, as well as enable new insights in a wide variety of other fields.

In this talk, I will discuss my research group’s work on realizing this long-term vision. I will start off in 3D, where I will show work that infers the full scene from a picture, including the hidden surfaces that a LiDAR does not see, and which learns to do so with easy-to-obtain supervision. Next, I will show our systems that recognize how humans are currently interacting with the scene, which pave the way towards understanding how interaction could happen. I will conclude by showing how this physical understanding can facilitate work in basic science, from inferring the solar magnetic field in data where each pixel is hundreds of kilometers wide to measuring millimeter-sized bird bones for testing old hypotheses at new scales.