Spatially Prompted Visual Trajectory Prediction for Egocentric Manipulation

Accepted to WACV 2027

A vision-centric manipulation setting where object and target spatial prompts in the first egocentric frame condition future end-effector trajectory prediction.

1Michigan State University, 2NVIDIA Research
*Equal contribution

Updated November 4, 2026: three-seed comparisons and sequential predicted-state feedback. The Paper PDF includes the latest results; the arXiv link points to an earlier version.

SP-VTP overview

We study how first-frame spatial prompts can ground egocentric manipulation and drive future 3D end-effector trajectory prediction.

SP-VTP

We formulate egocentric manipulation as spatially prompted visual trajectory prediction from first-frame object-target grounding.

EgoSPT

We collect egocentric manipulation trajectories with spatial prompt annotations and recovered 3D end-effector motion.

SPOT

We propose a prompt-centric policy that fuses task prompts, current observations, and history to generate future trajectories.

Metrics & Protocol

We evaluate held-out tasks within three shared scenes and test predicted-state feedback with replayed visual observations.

Abstract

Language instructions specify task semantics but can be ambiguous when several visually similar objects and candidate targets are present. We study whether first-frame spatial prompts can specify which object to manipulate and where to place it as the egocentric viewpoint changes.

We formalize this setting as Spatially Prompted Visual Trajectory Prediction (SP-VTP), an offline trajectory-prediction setting. EgoSPT contains 2,841 fork pick-and-place episodes across three scenes, with first-frame object/target annotations and recovered 3D end-effector motion.

SPOT fuses a fixed first-frame task representation with the current image and recent motion history, without an explicit object tracker. Three-seed experiments show lower translation and endpoint errors with spatial prompts; interventions reveal different roles for object and target cues. Sequential predicted-state replay confirms that prompts reduce, but do not eliminate, accumulated drift. These results do not establish closed-loop robot success or transfer to new objects and tasks.

EgoSPT Dataset

EgoSPT overview

EgoSPT grounds manipulation goals with lightweight first-frame object-target prompts and supervises future 3D end-effector trajectory chunks from egocentric observations.

2,841 egocentric episodes
112,856 original sliding-window samples
3 scenes shared by train and validation

EgoSPT is collected with a modified Universal Manipulation Interface. Each episode contains egocentric video, object and target bounding-box annotations in the initial frame, and recovered end-effector poses. A deterministic scene–task-group split keeps all episodes of a selected task group in one partition. All three scenes appear in both training and validation: the standard benchmark uses 2,584 training episodes and 257 validation episodes (10,024 validation windows). Per-scene results characterize layout complexity, not unseen-scene transfer. The dataset covers one manipulated-object category and one task family: fork pick-and-place.

Modified UMI collection setup
Modified UMI setup for collecting egocentric manipulation trajectories.

Dataset Demos

Example egocentric manipulation videos from three scenes in EgoSPT.

Scene 1

put fork1 to bowl1

put fork1 to bowl2

put fork1 to bowl3

Scene 2

put fork1 to bowl1

put fork1 to bowl2

put fork1 to bowl3

Scene 3

subscene1

subscene10

subscene11

Processed Data Visualizations

Each processed demo synchronizes the egocentric video with recovered pose motion and gripper-width signals.

put fork1 to bowl1

put fork1 to bowl2

put fork1 to bowl3

put fork1 to cup1

put fork1 to cup2

put fork1 to plate1

SPOT Policy

Task Encoder Observation Encoder Trajectory Generator
SPOT policy architecture

Task Encoder

Encodes rendered boxes and coordinate tokens on the first frame. The resulting task representation stays fixed throughout the episode; no explicit object tracker is used.

Observation Encoder

Combines the current egocentric frame with the previous four EE states. Standard evaluation uses recorded history; the sequential diagnostic separately feeds back predictions.

Trajectory Generator

Uses flow matching to generate 16 relative EE waypoints with translation, 6D rotation, and gripper width. Each waypoint is relative to the chunk origin, not a per-step motion increment.

Experimental Results

Unless stated otherwise, results use 10,024 independent validation windows with recorded current RGB and four-step EE history. Tables report mean ± sample standard deviation across training seeds 42, 0, and 1. Translation errors are in cm; S@10 is the percentage of chunk endpoints within 10 cm of the recorded endpoint, not robot task success.

11.14 cm BBox+Visual fixed-16-step FDE
vs. 17.75 cm without prompts
+13.95 pp S@10 gain over No Prompt
53.14% vs. 39.19%
45.3% lower episode-end FDE with prompts
in cumulative state-feedback replay

How much do spatial prompts help?

On smaller screens, scroll tables horizontally to see all metrics.

Standard independent-window protocol. Lower errors and higher endpoint rates are better.
Task specificationPos. L2 (cm) ↓FDE (cm) ↓S@2 (%) ↑S@5 (%) ↑S@10 (%) ↑
No Prompt9.39 ± 0.2617.75 ± 0.394.31 ± 0.6319.47 ± 1.1739.19 ± 0.97
Target Only7.07 ± 0.0211.57 ± 0.123.35 ± 0.0820.16 ± 0.5850.14 ± 0.56
Visual Prompt6.83 ± 0.1811.25 ± 0.503.99 ± 0.3422.66 ± 0.9452.93 ± 3.17
BBox Prompt7.15 ± 0.0711.85 ± 0.143.00 ± 0.1719.01 ± 0.1949.02 ± 0.36
BBox+Visual6.90 ± 0.1011.14 ± 0.293.80 ± 0.4121.31 ± 0.6253.14 ± 0.41

Spatial prompts mainly reduce large endpoint errors rather than the finest local errors: S@10 improves while S@2 changes little. Target cues explain most endpoint gains; object context further improves the trajectory path. No Prompt still retains the unmarked first frame and recorded history—it is not a current-RGB-only baseline.

External VLA references

SmolVLA, OpenVLA-OFT, and GR00T N1.6 use current RGB, recorded four-step history, and category language without object or target instance IDs, e.g. pick fork and place to plate. They do not receive first-frame images or spatial prompts. SPOT instead receives an instance-specific spatial task specification. These comparisons do not isolate architecture or fine-tuning strategy.

Three-seed results under the standard recorded-history protocol. The π0.5 reference has a different prompt configuration, noted below.
MethodTask inputPos. L2 (cm) ↓FDE (cm) ↓Rot. geo. (°) ↓S@10 (%) ↑
SmolVLACategory language9.36 ± 0.2216.85 ± 0.4711.06 ± 0.1637.21 ± 1.72
OpenVLA-OFTCategory language8.16 ± 0.0615.33 ± 0.148.86 ± 0.0541.43 ± 0.74
GR00T N1.6Category language10.48 ± 1.0717.39 ± 1.1811.60 ± 0.4732.42 ± 4.47
π0.5*Generic language + visual prompt14.25 ± 0.1823.68 ± 0.2815.53 ± 0.0220.67 ± 0.47
SPOTBBox+Visual6.90 ± 0.1011.14 ± 0.299.86 ± 0.0953.14 ± 0.41

*π0.5 input distinction. Its original run metadata specifies a rendered first-frame object/target prompt, current RGB, four-step history, and the generic instruction “Move the object highlighted in green to the target highlighted in red.” It is not a language-only baseline. OpenVLA-OFT has lower rotation error than SPOT; the spatial-prompt advantage is not uniform across control variables.

Per-scene external-VLA results
Same train/validation setting, not unseen-scene transfer. Scene 1/2/3 contain 4,333/3,079/2,612 validation windows. Entries are mean ± sample standard deviation.
MethodScene 1Scene 2Scene 3
Pos. L2 (cm)FDE (cm)S@10 (%)Pos. L2 (cm)FDE (cm)S@10 (%)Pos. L2 (cm)FDE (cm)S@10 (%)
SmolVLA7.35 ± 0.2413.13 ± 0.4048.17 ± 1.3110.22 ± 0.4219.47 ± 0.9836.05 ± 3.5711.69 ± 0.0719.94 ± 0.1520.41 ± 0.57
OpenVLA-OFT6.10 ± 0.1010.98 ± 0.2253.55 ± 1.898.80 ± 0.1017.48 ± 0.1838.75 ± 0.2010.83 ± 0.1920.01 ± 0.4524.50 ± 1.21
GR00T N1.68.67 ± 1.2213.26 ± 1.4643.55 ± 6.8411.18 ± 1.0519.71 ± 1.2329.76 ± 3.9012.65 ± 0.8621.51 ± 0.9217.10 ± 1.61
π0.5*11.63 ± 0.2518.58 ± 0.4125.49 ± 0.9314.79 ± 0.2125.43 ± 0.3822.59 ± 0.4017.95 ± 0.1730.08 ± 0.1910.40 ± 0.28
SPOT6.71 ± 0.2610.59 ± 0.5858.54 ± 2.156.25 ± 0.0710.88 ± 0.2853.53 ± 1.408.01 ± 0.2112.37 ± 0.1743.13 ± 1.32

Architecture ablations

Seed-42 ablations favor frozen DINOv2, a four-step history, and flow matching. A longer history or an unfrozen backbone is not consistently better. Figure axes retain their explicitly labeled paper units.

Trajectory Head

Trajectory head ablation

History Horizon

History horizon ablation

Backbone Tuning

DINOv2 tuning ablation

What happens when history is predicted?

Predicted-history feedback with recorded RGB.

After a recorded warm start, each predicted 16-step chunk supplies the next history. Cumulative replay also uses predicted pose anchors, without GT resets. The first-frame prompt stays fixed and RGB remains recorded: predictions do not change the scene, so this is not a physical robot rollout.

Episode-end FDE (cm), averaged over 257 episodes, then reported as mean ± sample standard deviation over seeds 42/0/1.
History / pose anchorSPOT (BBox+Visual)No Prompt
Recorded / recorded4.55 ± 0.145.91 ± 0.29
Predicted / recorded6.53 ± 0.249.36 ± 0.75
Predicted / predicted22.59 ± 0.7841.30 ± 0.39

Prompts reduce, but do not eliminate, drift. Cumulative episode-end FDE drops by 45.3%, from 41.30 to 22.59 cm. End S@10 is 16.86% and Combined@10 is 5.45%—offline endpoint agreement, not physical task success.

Episode-end FDE scores the final valid waypoint, including partial tails; it is not the fixed-16-step FDE above. The recorded-history control uses the same corrected alignment and queries. Predicted poses can diverge from the replayed camera view, limiting deployment claims.

Position error versus episode progress for recorded history, predicted history with recorded anchors, and cumulative predicted-state feedback. Correct prompts reduce but do not remove cumulative drift.
Episode-balanced position error over normalized progress: three-seed mean ± sample standard deviation. The final bin is not episode-end FDE. Open the full-resolution plot.

Do object and target cues still play different roles?

Cumulative predicted-history/predicted-anchor replay, three-seed mean ± sample standard deviation. Both visual and coordinate prompts are changed by each shuffle.
Prompt conditionADE (cm) ↓Episode-end FDE (cm) ↓End S@10 (%) ↑End Combined@10 (%) ↑
BBox+Visual14.39 ± 0.4622.59 ± 0.7816.86 ± 1.375.45 ± 2.02
No Prompt19.87 ± 0.2541.30 ± 0.391.69 ± 0.980.39 ± 0.00
Shuffled Object18.95 ± 0.4824.09 ± 0.4511.67 ± 0.674.02 ± 0.22
Shuffled Target19.74 ± 0.4741.93 ± 1.611.17 ± 1.030.39 ± 0.39

Object shuffling mainly disrupts early acquisition; target shuffling harms transport and placement. Phases use recorded EE motion and gripper width for analysis only. Phase ADE averages waypoint errors within each episode and phase, not chunk-end errors.

Episode-balanced waypoint ADE (cm), three-seed mean ± sample standard deviation.
ConditionApproachPre-graspGrasp / occlusionTransportPlacement
BBox+Visual6.97 ± 0.2312.10 ± 0.4512.11 ± 0.5315.97 ± 0.5921.48 ± 0.71
No Prompt7.22 ± 0.1213.18 ± 0.5013.67 ± 0.6019.14 ± 0.1937.48 ± 0.29
Shuffled Object10.19 ± 0.2119.33 ± 0.5719.98 ± 0.7222.15 ± 0.9623.29 ± 0.34
Shuffled Target7.09 ± 0.2912.18 ± 0.1712.30 ± 0.0819.10 ± 0.5337.91 ± 1.41
Per-scene cumulative feedback
Episode-end FDE (cm), three-seed mean ± sample standard deviation. All three scenes are represented in training.
SceneSPOT (BBox+Visual)No Prompt
Scene 123.85 ± 2.0337.97 ± 1.47
Scene 222.64 ± 2.0348.39 ± 1.60
Scene 321.36 ± 0.2237.88 ± 2.48
Protocol, metric definitions, and downloadable results

Queries start at step 4 and advance by 16. Four past states, excluding the current state, are re-expressed at each new anchor. Waypoints share a chunk origin; they are not incremental transforms. Gripper width is absolute. Future GT actions are used only for scoring.

All conditions use the same 257 episodes, corrected alignment, query times, and paired noise. Each evaluation covers 986 queries (746 full chunks and 240 partial tails) and 13,906 future waypoints. First-chunk agreement, tail coverage, and no cumulative GT anchor resets were audited.

ADE averages waypoint position errors per episode, then across episodes; FDE uses each episode's final valid waypoint. End Combined@10 requires position error ≤10 cm, rotation error ≤20°, and matching gripper state (open above 3 cm). These are offline agreement criteria, not physical task success.

Rollout summaries and audits (JSON) · Per-seed / per-scene metrics (CSV) · Phase metrics (CSV)

Trajectory Visualizations

First chunk trajectory visualization
First-chunk prediction visualizes immediate local motion and horizon-wise position, rotation, and gripper errors.
Full trajectory visualization
Independent-window predictions stitched for visualization—not an autoregressive rollout. Predicted-state feedback is evaluated separately above.

More Scenes

Scene 1 trajectory
Scene 1. Stitched independent-window predictions; these images do not measure closed-loop execution.
Scene 2 trajectory
Scene 2. Stitched independent-window predictions; these images do not measure closed-loop execution.
Scene 3 trajectory
Scene 3. Stitched independent-window predictions; these images do not measure closed-loop execution.

Annotation Tools

EgoSPT uses a lightweight annotation workflow for first-frame object and target bounding boxes. An annotation tool records the two boxes, while a modification tool supports manual inspection and correction before training samples are built.

Annotation tool
Annotation modification tool

BibTeX (WACV 2027)

@inproceedings{li2027spvtp,
  title={Spatially Prompted Visual Trajectory Prediction for Egocentric Manipulation},
  author={Li, Yifan and Zhou, Xinyu and Ge, Yunhao and Tan, Xiaobo and Kong, Yu},
  booktitle={Winter Conference on Applications of Computer Vision (WACV)},
  year={2027}
}

Page numbers and DOI are omitted until verified.