Zhutian (Skye) Yang / ju tin-yen young /      ztyang {at} mit {dot} edu Hello! My name is Zhutian and I'm a final-year PhD candidate in robotics at MIT. I develop algorithms for solving long-horizon manipulation problems in geometrically complex environments. I use a combination of deep learning and planning methods. I'm co-advised by Leslie Pack Kaelbling and Tomás Lozano-Pérez in the Learning and Intelligent Systems group. I'm currently a part-time research intern at Toyota Research Institute (TRI) Large Behavior Models team. I was an ex-NVIDIA intern in the Seattle Robotics Lab. I obtained bachelor's degree in information engineering and media at NTU, Singapore.

Research

• Developing generalizable long-horizon visuomotor policies.
  • Ongoing at TRI Developed a hierarchical multi-task policy architecture where a Robot Visual Planning Network generates language goals and guides a low-level language-conditioned multi-skill policy.
  • Ongoing at MIT Developing humanoid manipulation policies for multi-contact whole-body manipulation tasks by conbining diverse data sources collected in virtual and real environments.
  • PoPi Chain imitation learned policies conditioned on waypoints generated by motion planning for solving multi-step mobile manipulation of objects with unknown dynamics.
• Generating long-horizon manipulation trajectories by combining learning-based methods and planning techniques.
  • VLM-TAMP Solve long-horizon manipulation problems for any robot embodiment by combining the geometric reasoning ability of TAMP and common-sense provided by pre-trained VLMs.
  • Diffusion-CCSP Learn to plan for motion while satisfacing geometric collision-free, physical stability, and data-defined spatial constraints (e.g. packing non-convex objects in boxes).
  • PiGi Learn to predict task plan feasibility from images of environments with articulated and movable obstacles using a transformer trained on a wide variety of procedurally generated scenes.

Publications

Guiding Long-Horizon Task and Motion Planning with Vision Language Models

Zhutian Yang, Caelan Reed Garrett, Leslie Pack Kaelbling, Tomás Lozano-Pérez, and Dieter Fox

ICRA 2025; CoRL 2024 LangRob Workshop (Spotlight)

TLDR: Pretrained VLMs make mistakes in predicting robot actions when prompted with open language goals, so we use VLMs to break down long-horizon goals into subgoals, which are then solved by TAMP, in an interative replanning system. It's used to solve problems that involve interactions with 20+ objects and require 30-50 actions to complete.

Paper  |  Project Page  |  Code  |  Bibtex  |  Talk  |  Poster (click to show) Poster (click to hide)



Combining Planning and Diffusion for Mobility with Unknown Dynamics

Yajvan Ravan, Zhutian Yang, Tao Chen, Leslie Pack Kaelbling, and Tomás Lozano-Pérez

In Submission

TLDR: Rearranging large objects with unprediatble dynamics is hard because the relative pose between robot and object is changing. Diffusion policies that output global robot configurations struggle to generalize to new initial and goal conditions, or new environments and objects. So, we use motion planning to generating waypoints that guide a local diffusion policy, which is trained to achieve relative movements of the chair.

Paper  |  Project Page  |  Bibtex



Compositional Diffusion-Based Continuous Constraint Solvers

Zhutian Yang, Jiayuan Mao, Yilun Du, Jiajun Wu, Joshua Brett Tenenbaum, Tomás Lozano-Pérez, and Leslie Pack Kaelbling

CoRL 2023

TLDR: Multi-step manipulation problems involve a lot of collision-free, physical stability, and culture-defined spatial constraints. Conventional methods usually solve it by sampling then rejection, which is too slow. Therefore, we find global solutions by diffusion-based optimization, using diffusion models trained for each contraint type.

Paper  |  Project Page  |  Code  |  Bibtex  |  Talk  |  MIT News  |  Poster (click to show) Poster (click to hide)



Sequence-Based Plan Feasibility Prediction for Efficient Task and Motion Planning

Zhutian Yang, Caelan Reed Garrett, Leslie Pack Kaelbling, Tomás Lozano-Pérez, and Dieter Fox

RSS 2023

TLDR: In long-horizon mobile manipulation problems in complex environments with lots of articulated and movable obstacles, task and motion planners spend most computation on solving motion planning problems that aren't solvable. So we train a plan feasibility prediction model that quickly sort candidate plans by their likelihood of success using visual and language features of the problem, which cuts down planning time by 50 - 80 %.

🔥 We won Best Paper Runner-Up in CoRL 2022 Workshop on Learning, Perception, and Abstraction for Long-Horizon Planning

Paper  |  Project Page  |  Code  |  Bibtex  |  Talk  |  MIT News  |  Tech Crunch  |  Poster (click to show) Poster (click to hide)



Let’s Handle It: Generalizable Manipulation of Articulated Objects

Zhutian Yang, and Aidan Curtis

ICRL 2022 Workshop on Generalizable Policy Learning in the Physical World (Spotlight)

🔥 We won 2nd place in the ManiSkill Challenge 2022 Robotics Track

Paper  |  Poster (click to show) Poster (click to hide)



Flexibly Instructable Robots

Zhutian Yang, Patrick Henry Winston, and David Hsu

Undergraduate thesis work; Also appeared in Advances in Cognitive Systems 2019 and DSpace@MIT

Paper  |  Video Demo  |  Poster (click to show) Poster (click to hide)

Services

• Reviewed for ICLR, IROS, RA-L, AAAI, ICRA, and RSS.
• Co-organized ICRA 2024 Workshop on Vision-Language Models for Navigation and Manipulation (VLMNM).
• Co-organized RSS 2023 Workshop on Learning for Task and Motion Planning (LTAMP).

• Serving as a Student Counselor in EECS Resources for Easing Friction and Stress since 2020 and helped 10+ graduate students through stressful situations such as changing advisors and family conflicts.
• Served as Teaching Assistant to MIT 6.036 Introduction to Machine Learning in Spring 2022.

Misc.

• I'm an AFAA certified group exercise instructor, specializing in kickboxing.
• If thrown out of robotics research and threatened to never code again, I would do improv musical comedy.
• Wait, I could actually fight back.