All projects
ResearchPhysical AI

DINO-WM Reproduction

Reproducing zero-shot planning with world models on frozen DINO features, and testing DINOv3.

Overview

DINO-WM learns task-agnostic visual dynamics in the latent space of a frozen DINOv2 encoder, which enables zero-shot planning without rewards or expert demonstrations. We set out to reproduce its planning results on PointMaze and Push-T under real compute constraints.

Training was first bottlenecked by I/O, not compute. We moved to precomputed float32 embeddings, decoder-free training and Flash Attention (which nearly halved VRAM), and ran on the DelftBlue and DAIC clusters. I owned the PointMaze runs and swapped in DINOv3 as the observation encoder, which gave about 17× lower visual loss thanks to smoother frame-to-frame features.

Highlights

  • 01Profiled and removed data-loading bottlenecks across Kaggle, local GPUs and HPC clusters.
  • 02DINOv3 vs. DINOv2 encoder comparison for latent world models.
  • 03CEM planning in latent space evaluated on PointMaze success rate.

Gallery

Imagined vs. actual rollouts while planning in latent space.
Validation losses for DINOv2, DINOv3 and DINOv3-Repeat4 world models.

Command menu

Jump to a section or project