Events

CS Colloquium: Learning Structured World Models From and For Physical Interactions

Speaker: Yunzhu Li

Location: 60 Fifth Avenue, Room 150

Date: Thursday, March 31, 2022

Humans have a strong intuitive understanding of the physical world. We
observe and interact with the environment through multiple sensory
modalities and build a mental model that predicts how the world would
change if we applied a specific action (i.e., intuitive physics). My
research draws on insights from humans and develops model-based
reinforcement learning (RL) agents that learn from their interactions
and build predictive models of the environment that generalize widely
across a range of objects made with different materials. The core idea
behind my research is to introduce novel representations and integrate
structural priors into the learning systems to model the dynamics at
different levels of abstraction. I will discuss how such structures can
make model-based planning algorithms more effective and help robots to
accomplish complicated manipulation tasks (e.g., manipulating an object
pile, pouring a cup of water, and shaping deformable foam into a target
configuration). Beyond visual perception, I will also discuss how we
built multi-modal sensing platforms with dense tactile sensors in
various forms (e.g., gloves, socks, vests, and robot sleeves) and how
they can lead to more structured and physically grounded models of the
world.