SeeTraceAct: Visibility-Aware Latent Planning from Cross-Embodiment Demonstration Videos

Overview of SeeTraceAct. Given a demonstration video, current camera views, and a language instruction, the policy encodes task-relevant information into a visual latent plan (See). During training, the policy predicts future visual traces and their visibility for each camera view (Trace), while also predicting actions from the latent plan (Act). At inference time, the trace prediction component is discarded, and the policy generates actions from the latent plan (Act).

Vision-language-action models (VLAs) are promising general-purpose robot policies, but adapting them to new tasks typically requires costly task-specific teleoperation data. As an alternative, we study one-shot demo-conditioned VLAs, where a robot policy is conditioned on a single demonstration video of an unseen task. We find that existing end-to-end approaches often struggle when successful execution requires precisely localizing small target regions. To address this limitation, we propose SeeTraceAct, a demo-conditioned VLA framework that encourages precise spatial grounding through visibility-aware prediction of future end-effector traces. To enable reproducible evaluation with cross-embodiment demonstrations, we introduce and release RoboCasa-DC, a demo-conditioned extension of RoboCasa with episode-paired humanoid videos. Experiments on RoboCasa-DC and a real-world benchmark, where a Franka Panda arm is conditioned on human demonstrations, show that SeeTraceAct outperforms baselines, achieving the best success rate across all four RoboCasa-DC settings and improving real-world average success by 12.5 percentage points.

Architecture of SeeTraceAct. The model receives camera views, a language instruction, a demonstration video, and robot states, and outputs an action chunk. Learnable query tokens are appended after the input tokens; their final hidden states form a visual latent plan, which is decoded into future end-effector traces during training. The trace decoder is used only during training and discarded at inference time.

SeeTraceAct builds on the GR00T N1.5 architecture, which pairs a vision-language model (VLM) with a flow-matching action expert, and augments it with three components:

Video tokens. The demonstration video is encoded with an action-aware video encoder (V-JEPA 2) and compressed by a Perceiver Resampler into a compact set of 32 video tokens appended to the VLM input. Under causal attention, the video tokens attend to the image and language tokens, making the demonstration representation context-dependent rather than fixed across the episode.

Visual latent plan. A separate set of learnable query tokens is appended after the input tokens. These query tokens attend to the image, language, and video tokens, and their final hidden states constitute a latent representation trained to encode future task progression, which conditions action generation.

Visibility-aware trace decoder. During training, the latent plan is decoded into future end-effector traces in each static camera view. Because the end effector may leave some views, making its 2D coordinates ill-posed regression targets, the decoder uses two heads: a regression head that predicts normalized trace coordinates and a validity head that predicts whether each trace point lies within the image — so off-screen points still provide a learning signal. The policy is trained with a flow-matching action prediction loss plus this auxiliary visual trace loss, and the trace decoder is discarded after training.

Cross-embodiment benchmark dataset in RoboCasa-DC. For each of the 24 tasks, we pair 100 original Panda-arm trajectories with collected GR-1 humanoid demonstrations for training. For evaluation, we collect humanoid demonstrations for 50 pre-defined seeds per task.

RoboCasa-DC extends the RoboCasa simulation environment with episode-paired demonstrations for demo-conditioned policy learning. It supports both same-embodiment evaluation, where videos from Panda-arm trajectories serve as demonstrations, and cross-embodiment evaluation, where teleoperated GR-1 humanoid videos act as a simulated proxy for real-world human demonstrations. An evaluation split partitions the 24 tasks into seen training tasks and unseen evaluation tasks; at test time, each policy is conditioned on a single demonstration video paired with each evaluation seed. We publicly release RoboCasa-DC to support reproducible evaluation and future work on demo-conditioned policies.

RoboCasa-DC Benchmark

SeeTraceAct achieves the best success rate across all four evaluation settings, covering both category-balanced and precision-sensitive splits under same- and cross-embodiment demonstrations, with the largest margin in the precision-sensitive cross-embodiment setting.

Success rates averaged over the five unseen tasks in each split, with 50 evaluation episodes per task. In Same-emb demo, policies are conditioned on demonstrations from the same Panda-arm embodiment; in Cross-emb demo, policies are conditioned on simulated GR-1 humanoid demonstrations as a proxy for human videos.

Real-World Benchmark

We evaluate demo-conditioned policies on real-world tabletop manipulation with a Franka Panda arm, conditioning each policy on a one-shot human demonstration video. Policies are trained on four seen tasks and evaluated on four unseen tasks designed by recombining seen skills and objects, introducing a new contact interaction, and swapping the source and target objects.

Four seen tasks (top) and four unseen tasks (bottom) in the real-world benchmark. The yellow arrow indicates the desired path of the end effector.

Success rates over 10 trials for each of the four unseen tasks, along with their average. SeeTraceAct achieves the strongest performance across all four unseen tasks, improving average success from 37.5% to 50.0% over the strongest baseline.

@article{son2026seetraceact,
  title   = {SeeTraceAct: Visibility-Aware Latent Planning from Cross-Embodiment Demonstration Videos},
  author  = {Son, Jaehyeon and Kim, Junhyun and Kam, Kyle and Coholich, Jeremiah and Kim, Seok Joon and Kim, Jinhoo and Kim, Chris Dongjoo and Cho, Jaemin and Fox, Dieter and Kira, Zsolt},
  journal = {arXiv preprint},
  year    = {2026}
}