Gaoyue (Kathy) Zhou

I am a Computer Science Ph.D. student at NYU Courant advised by Professors Lerrel Pinto and Yann LeCun.

Prior to NYU, I was a MS in Robotics student at CMU Robotics Institute advised by Professor Abhinav Gupta. I worked on enhancing robots' generalization capabilities and benchmarking. My thesis can be found here.

Prior to CMU, I did my undergraduate in Computer Science and Applied Mathematics at UC Berkeley. During my time at Berkeley, I had the pleasure to work with Professors Sergey Levine and John DeNero in the Berkeley Artificial Intelligence Research (BAIR) Lab.

Email  /  CV  /  Google Scholar  /  Github  /  Photography

I am broadly interested in World Models and Robotics. My motivation is to build agents with cognitive capabilities that can learn and infer like humans.

We introduce a hierarchical world model that decouples state-space reachability from local control: a high-level model samples feasible transitions across the state manifold, while a low-level world model grounds them into precise actions as an inverse dynamics solver, enabling long-horizon plans synthesized from short segments.

We introduce a minimal architectural extension that lets transformer policies consume dense pre-trained patch tokens directly, improving over global-pooled representations by 40% without the cost of a full VLM backbone.

Inspired by perceptual straightening in human vision, we add a curvature regularizer to JEPA world models so that Euclidean distance in latent space better approximates geodesic distance, improving gradient-based planning.

Through controlled pretraining experiments over text, image-text pairs, video, and action-conditioned video, we study how visual data complements language, and find that multimodal pretraining naturally gives rise to world modeling capabilities.

We represent dexterous actions as finger keypoints extracted from egocentric video, allowing a world model to train on over 900 hours of human and robot data and transfer zero-shot to unseen skills on a real dexterous hand.

We introduce a simple yet effective method for constructing versatile world models with pre-trained DINOv2 that generalize to complex environment dynamics, which enables zero-shot solutions at test time for arbitrary goals.

We train a video world model across diverse environments and embodiments, demonstrating its ability to plan novel trajectories in both known and unknown environments.

We introduce a new benchmark for tabletop manipulation: Train Offline, Test Online (TOTO) and showed performance of state-of-the-art models on two tasks.

In this work, we evaluate Offline RL algorithms' performance on real robot, on in-domain tasks and transfer learning settings.

In this paper, we collect and release a dataset for identifying deceptive actors through the game of Mafia, as well as train models to identify such actors and reveal features of their language.

We proposed a method for pre-training a behavioral prior for reinforcement learning using data from a diverse range of tasks, and used this behavioral prior to speed up learning of new tasks.

We proposed various ways to use CAD Tools for the Procedural Generation of 2-Manifold Sculpture Geometries with high-level control.

10725A: Convex Optimization , Fall 2022

CS 189: Introduction to Machine Learning , Fall 2020

Website template