All projects
Physical AI

Liminal

Long-horizon dual-arm manipulation with FLUX 3 Action, planned and checked by Claude.

Overview

Liminal explores collaboration between two arms: one arm changes or holds something in the scene so the other can do what it otherwise couldn't. FLUX 3 Action provides learned, text-prompted manipulation skills. Everything above that is our system: deciding why a skill is needed, when the other arm is ready, which conditions must stay true, and whether the goal really happened.

We trained one LoRA adapter per skill on our own VR-teleoperated demonstrations and chained them under a planner. After every step, Claude checks the cameras and replans when a step fails. A safety supervisor enforces joint limits, rate limits, a torque cap and the e-stop throughout.

My part was the research front-end: the public project page and a full-screen live lab that mirrors the real arms in a MuJoCo twin of the Haller, also embedded in the orchestrator HMI. I also produced the launch and teaser videos and helped integrate the end-to-end demo.

Results

€1,000
track prize, awarded by the ERH jury
2 / 3
full-chain runs finished 4/4 with no human touch
0 → 3/5
red-pen inserts: stock zero-shot FLUX vs. our LoRA
5
public datasets & 4 LoRA checkpoints

Highlights

  • 01Full 4-skill chain (open box → green pen → yellow cutter → red pen) on real hardware, with a camera check after each step.
  • 02Per-skill FLUX 3 Action LoRAs (rank 32) trained on our own LeRobot v3 datasets.
  • 03Behavior-tree executor with AFTER / WHILE / UNTIL conditions, retries and arm-ownership safety.
  • 04Live digital twin in the browser, driven by the real arms' joint telemetry.
  • 05All checkpoints and five teleoperation datasets released publicly on Hugging Face.

Gallery

The real station and its MuJoCo twin, which mirrors the arms' measured joints live.
Task plan → harness → per-arm FLUX 3 Action skills → safety supervisor, with perception closing the loop.
Real vs. simulated scene used to validate the twin.
Head-camera view used for Claude's after-step verification.

Command menu

Jump to a section or project