DINO-WM Reproduction
Reproducing zero-shot planning with world models on frozen DINO features, and testing DINOv3.
Overview
DINO-WM learns task-agnostic visual dynamics in the latent space of a frozen DINOv2 encoder, which enables zero-shot planning without rewards or expert demonstrations. We set out to reproduce its planning results on PointMaze and Push-T under real compute constraints.
Training was first bottlenecked by I/O, not compute. We moved to precomputed float32 embeddings, decoder-free training and Flash Attention (which nearly halved VRAM), and ran on the DelftBlue and DAIC clusters. I owned the PointMaze runs and swapped in DINOv3 as the observation encoder, which gave about 17× lower visual loss thanks to smoother frame-to-frame features.
Highlights
- 01Profiled and removed data-loading bottlenecks across Kaggle, local GPUs and HPC clusters.
- 02DINOv3 vs. DINOv2 encoder comparison for latent world models.
- 03CEM planning in latent space evaluated on PointMaze success rate.