All projects
Physical AISoftware & Agents

NurseArm

An LLM agent that safely drives an SO-101 arm for medication tasks through bounded MCP skills.

Illustration: an SO-101 arm hands a green pill to an open, palm-up hand while the agent logs the MCP call

Overview

NurseArm connects an SO-101 robot arm to a browser interface and an LLM agent. The agent never produces raw motor commands. It discovers a bounded set of MCP tools, picks a registered skill, executes it and reports a structured result. Every decision lands in an append-only audit log.

Skills include direct Cartesian primitives and two trained ACT policies: sorting pills and handing over a green or black pill when MediaPipe detects an open, palm-up hand. A calendar can trigger the right skill at medication time. Text, voice (Faster-Whisper) and an optional Telegram bridge are all supported inputs.

Highlights

  • 01Bounded MCP skill server: the LLM selects tools and never improvises motor commands.
  • 02ACT policies (sort_pills, handover_pill) deployed through lerobot-rollout.
  • 03Hand openness and palm-up detection with MediaPipe on RealSense or webcam streams.
  • 04Local-first agent (Ollama + Qwen 3) with Claude as a drop-in backend.
  • 05An honesty document that separates what was built, what was borrowed and what was verified.

Command menu

Jump to a section or project