Learning Visual Feature-based World Models via Residual Latent Action
NeurIPS 2026
Xinyu Zhang 张鑫语
Building robot policies grounded in 2D/3D world understanding.
I am a Research Scientist at Ai2 (Allen Institute for AI), working with Prof. Dieter Fox.
I obtained my PhD in Computer Science at Rutgers University, where I was fortunate to be advised by Prof. Abdeslam Boularias.
Before joining Rutgers, I lived a different life as a software engineer at Microsoft and Face++, where I gained exposure to large-scale representation learning, recommender systems, model compression, FPGA programming, and front-end web development.
I earned my Master’s at UC San Diego under Prof. Ken Kreutz-Delgado and my Bachelor’s at the University of Science and Technology of China.
Google Scholar · GitHub · CV · xz653 at rutgers dot edu
News
- Residual Latent Action World Model (RLA-WM) accepted to NeurIPS 2026.
- Glove2Hand accepted to CVPR 2026 (Highlight).
- Motion Blender Gaussian Splatting accepted to CoRL 2025.
- Joined Meta Reality Lab as a research intern.
- Autoregressive Action Sequence Learning accepted to RA-L 2025.
Research
I have published nine independent first-author papers in top-tier machine learning, robotics and vision venues — NeurIPS, CVPR, RSS, CoRL and more — spanning robot manipulation policy learning, world modeling, computer vision, 3D reconstruction, and video generation. My research goal is truly scaling robot learning with world understanding.
Publications
2026
Meta Reality Lab2025
Motion Blender Gaussian Splatting for Dynamic Scene Reconstruction
Conference on Robot Learning (CoRL) 2025
Autoregressive Action Sequence Learning for Robotic Manipulation
IEEE Robotics and Automation Letters (RA-L) 2025
TL;DR
- Robot actions as a language — but they are heterogeneous and often continuous.
- We propose a chunking causal transformer to adapt autoregressive models for robot actions.
- A universal architecture that establishes new state of the art on Push-T, ALOHA, and RLBench.
2024
One-Shot Imitation Learning with Invariance Matching for Robotic Manipulation
Robotics: Science and Systems (RSS) 2024
TL;DR
- Bind 3D spatial regions to robot actions, so actions carry semantics.
- Learn to discover these “key regions” and match regions between demonstrations.
- Through region matching, we transfer robot actions to new scenes in one shot.
Detect Everything with Few Examples
Conference on Robot Learning (CoRL) 2024
TL;DR
- Existing work mixes representation learning with detection.
- We don’t learn representations — we focus on how to use existing pretrained ones.
- Detect by propagating ROI regions in the attention map.
Scaling Manipulation Learning with Visual Kinematic Chain Prediction
Conference on Robot Learning (CoRL) 2024
TL;DR
- How to learn a single policy for diverse environments?
- Use a universal, visually grounded, analytically determined action space.
- That is, the visual projection of the robot kinematic structure.
2023
2022
2020
