Events

Global AI Frontier Lab Seminar Series: Learning Point-based Dexterous Hand Policies from Human Demonstrations

Speaker: Seong Hyeon Park

Location: 1 MetroTech Center, Room Global AI Frontier Lab (Floor 22)

Date: Monday, June 15, 2026

Dinner & networking will begin at 6:00 PM and the seminar will start at 7:00 PM EST. This event will be in-person & online. In-person attendance is strongly encouraged for Lab researchers in NYC. All attendees must RSVP to participate. For online attendees, a Zoom link will be sent out prior to the event. Please reach out to global-ai-frontier-lab@nyu.edu with any questions. We hope to see you there!

Abstract:  Robotic foundation models pre-trained on human demonstration videos have shown promise, but a significant embodiment gap remains when the resulting policies are deployed on real robots. A common remedy is to fine-tune these models on robot-specific demonstrations. However, robot data collection can be prohibitively expensive and time-consuming, which is particularly acute in dexterous manipulation, e.g., teleoperating a multi-fingered hand for even a single atomic task can take days. To address this, we introduce Dexterous Point Policy, a framework that learns dexterous manipulation policies directly from human videos and requires no robot demonstrations. Our core insight is that a unified 3D keypoint representation can bridge human and robot embodiments when used for both observations and actions. Specifically, we extract 3D keypoints of task-relevant objects and human hands from raw videos, and train an autoregressive transformer over these keypoints. We observe that at the keypoint level, specifically the wrist and fingertips, human and robot behaviors closely align, enabling direct policy transfer. On a suite of real-robot tasks spanning pick-and-place and tool use, Dexterous Point Policy attains 75.0% success, whereas a state-of-the-art VLA baseline reaches only 1.0%. Furthermore, our method generalizes strongly to unseen scenarios, including multi-object environments and novel object categories. 

Bio: Seong-Hyeon is a PhD student at KAIST, advised by Professor Jinwoo Shin. His research focuses on efficient algorithms for robot learning and high-dimensional computer vision. In particular, he has worked on policies for dexterous (high-degree-of-freedom) robotic hands, 3D/4D reconstruction, and video motion estimation.