Zhutian (Skye) Yang ztyang {at} mit {dot} edu Hello! My name is Skye and I'm a PhD student in robotics at MIT. I develop algorithms for solving multi-step manipulation problems in geometrically complex environments. I use a combination of deep learning and task and motion planning (TAMP) methods. I'm co-advised by Leslie Pack Kaelbling and Tomás Lozano-Pérez in the Learning and Intelligent Systems group. I'm currently a part-time research intern at Toyota Research Institute (TRI), Robotics division, Large Behavior Models team. I was an ex-NVIDIA intern in the Seattle Robotics Lab. I obtained bachelor's degree in information engineering and media at NTU, Singapore.

Research

• Developing generalizable long-horizon visuomotor policies by conbining a reactive local policy and a high-level planner (e.g., VLM or a planner).
  • Ongoing at MIT Develop a hierarchical policy network for multi-contact manipulation tasks by conbining multiple data sources collected in virtual and real environments.
  • Ongoing at TRI Train language-conditioned multi-skill policies to solve long-horizon mobile manipulation problems and chaining them by fine-tuning vision language models for skill planning.
  • PoPi Chain imitation learned policies conditioned on waypoints generated by motion planning for solving multi-step mobile manipulation of objects with unknown dynamics.
• Speeding up long-horizon task and motion planning using generative models.
  • VLM-TAMP Solve long-horizon manipulation problems for any robot embodiment by combining the geometric reasoning ability of TAMP and common-sense provided by pre-trained VLMs.
  • Diffusion-CCSP Learn to plan for motion while satisfacing geometric collision-free, physical stability, and data-defined spatial constraints (e.g. packing non-convex objects in boxes).
  • PiGi Learn to predict task plan feasibility from images of environments with articulated and movable obstacles using a transformer trained on a wide variety of procedurally generated scenes.

Publications

Guiding Long-Horizon Task and Motion Planning with Vision Language Models

Zhutian Yang, Caelan Reed Garrett, Leslie Pack Kaelbling, Tomás Lozano-Pérez, and Dieter Fox

CoRL 2024 LangRob Workshop (Spotlight)

Paper  |  Project Page  |  Bibtex  |  Talk  |  Poster (click to show) Poster (click to hide)



Combining Planning and Diffusion for Mobility with Unknown Dynamics

Yajvan Ravan, Zhutian Yang, Tao Chen, Leslie Pack Kaelbling, and Tomás Lozano-Pérez

In Submission

Paper  |  Project Page  |  Bibtex



Compositional Diffusion-Based Continuous Constraint Solvers

Zhutian Yang, Jiayuan Mao, Yilun Du, Jiajun Wu, Joshua Brett Tenenbaum, Tomás Lozano-Pérez, and Leslie Pack Kaelbling

CoRL 2023

Paper  |  Project Page  |  Code  |  Bibtex  |  Talk  |  MIT News  |  Poster (click to show) Poster (click to hide)



Sequence-Based Plan Feasibility Prediction for Efficient Task and Motion Planning

Zhutian Yang, Caelan Reed Garrett, Leslie Pack Kaelbling, Tomás Lozano-Pérez, and Dieter Fox

RSS 2023

🔥 We won Best Paper Runner-Up in CoRL 2022 Workshop on Learning, Perception, and Abstraction for Long-Horizon Planning

Paper  |  Project Page  |  Code  |  Bibtex  |  Talk  |  MIT News  |  Tech Crunch  |  Poster (click to show) Poster (click to hide)



Let’s Handle It: Generalizable Manipulation of Articulated Objects

Zhutian Yang, and Aidan Curtis

ICRL 2022 Workshop on Generalizable Policy Learning in the Physical World (Spotlight)

🔥 We won 2nd place in the ManiSkill Challenge 2022 Robotics Track

Paper  |  Poster (click to show) Poster (click to hide)



Flexibly Instructable Robots

Zhutian Yang, Patrick Henry Winston, and David Hsu

Undergraduate thesis work; Also appeared in Advances in Cognitive Systems 2019 and DSpace@MIT

Paper  |  Video Demo  |  Poster (click to show) Poster (click to hide)

Services

• Reviewed for ICLR, IROS, RA-L, AAAI, ICRA, and RSS.
• Co-organizing ICRA 2024 Workshop on Vision-Language Models for Navigation and Manipulation (VLMNM).
• Co-organized RSS 2023 Workshop on Learning for Task and Motion Planning (LTAMP).
• Served as a student counselor in EECS Resources for Easing Friction and Stress during 2020-2023 and helped more than 10 graduate students through stressful situations such as changing advisors and family conflicts.
• Served as Teaching Assistant to MIT 6.036 Introduction to Machine Learning in Spring 2022.

For What it's Worth

• I'm an AFAA certified group exercise instructor, specializing in kickboxing.
• If thrown out of robotics research and threatened to never code again, I would do improv musical comedy.
• Wait, I could actually fight back.