Copyright @2023 – All Right Reserved, NUS AI Lab
COM1, 13 Computing Drive Singapore 117417
Speaker: Assistant Professor Pulkit Agrawal, Department of Electrical Engineering and Computer Science at MIT
Chair: Dr SHAO Lin, Assistant Professor, School of Computing
Location: Hybrid (Zoom and in-person)
Abstract: I will present a vision and early steps towards a paradigm for building robots that can be tasked to perform any locomotion and manipulation task that a human can (i.e., Physical Intelligence). In recent years, large language and vision models have demonstrated unprecedented “common sense” understanding. However, reliable execution of sensorimotor skills (e.g., locomotion, opening doors, object manipulation, etc.) in the open world remains elusive. I will discuss a framework for learning new, complex, and generalizable sensorimotor skills in a manner that reduces human effort and is easily scaled to many tasks. I will elaborate using the following case studies:
Chair: Dr SHAO Lin, Assistant Professor, School of Computing
Location: Hybrid (Zoom and in-person)
Pulkit Agrawal is an Assistant Professor in the Department of Electrical Engineering and Computer Science at MIT, where he directs the Improbable AI Lab. He is interested in robotics and learning methods for control. Pulkit’s work received the Best Paper Award at the Conference on Robot Learning 2021 and the Best Student Paper Award at the Conference on Computer Supported Collaborative Learning 2011. He is a recipient of the Sony Faculty Research Award, Salesforce Research Award, Amazon Research Award, and a Fulbright fellowship. Before joining MIT, Pulkit received his Ph.D. from UC Berkeley and a Bachelor’s degree from IIT Kanpur, where he was awarded the Directors Gold Medal.
Abstract: I will present a vision and early steps towards a paradigm for building robots that can be tasked to perform any locomotion and manipulation task that a human can (i.e., Physical Intelligence). In recent years, large language and vision models have demonstrated unprecedented “common sense” understanding. However, reliable execution of sensorimotor skills (e.g., locomotion, opening doors, object manipulation, etc.) in the open world remains elusive. I will discuss a framework for learning new, complex, and generalizable sensorimotor skills in a manner that reduces human effort and is easily scaled to many tasks. I will elaborate using the following case studies:
- A dexterous manipulation system capable of re-orienting novel objects of complex shapes and peeling vegetables.
- A quadruped robot capable of fast locomotion, manipulation, and whole-body control on diverse natural terrains.
- A lifelong learning robotic agent that can request and learn new rigid-object manipulation skills in a few minutes.