Vision-language-action (VLA) models excel at robot manipulation via imitation learning, but adapting them to new tasks often requires additional human demonstrations, which can be costly or infeasible. Meanwhile, vision-language models (VLMs) offer semantic task understanding but lack the physical grounding required for execution. To bridge this gap, we present InSight, a framework for self-guided skill acquisition that uses a VLM to identify primitives missing from a VLA's repertoire, grounds the VLM's proposals through robot execution, and distills new primitives from successful rollouts into the VLA. Primitive steerability, the ability to execute and terminate primitives on command, enables the robot to reuse known primitives while collecting training data for missing primitives without requiring full-task human demonstrations for each new task. InSight has two stages: (1) a VLM automatically segments existing demonstrations into primitive-labeled trajectories to fine-tune a primitive-steerable VLA, and (2) the VLM plans a sequence of known primitives executed by the VLA and new primitives attempted by VLM-parameterized low-level controllers. New-primitive segments from successful task rollouts are added to the training data, and the VLA is retrained. The adapted VLA can then reliably execute new skills using the acquired primitives, without per-primitive VLM calls. We evaluate InSight on six simulated and real-world tasks with no human demonstrations of target skills, including block flipping, drawer closing, sweeping, twisting, and pouring. On hardware, acquired twisting and pouring skills achieve 92% and 96% success, versus 32% and 16% for a zero-shot CaP-X baseline. Composing both skills into a 14-primitive task achieves 80% success with no combined-task demonstrations.
Turn sound on
Consider a robot on Mars, trained only to scoop rocks. When a dust storm coats its solar panels, it must sweep them clean, a behavior it was never shown. A VLA can only perform the skills in its demonstrations, and acquiring a new one, through more data or reinforcement learning, is costly to repeat for every task.
Yet new skills are rarely fully novel: they recombine primitives the policy already knows. Sweeping and scooping share approach and lowering, differing only in a lateral push; flipping a block reuses pick-and-place's grasp-and-lift and adds a rotation. A standard VLA already encodes these primitives, but entangles them in a single task instruction, so they cannot be steered individually.
InSight makes the primitives steerable and uses a VLM as an active agent, not just a test-time planner over a fixed skill set, but one that flags the primitive a task is missing, drives the robot to acquire it, and retrains it back into the policy. Acquired skills then persist and recombine for future tasks, enabling continual learning.
Human demonstrations are decomposed into labeled primitives by aligning a VLM-generated primitive plan with gripper-state transitions and dominant end-effector motion. Each primitive segment becomes a separate training episode, fine-tuning the VLA into a policy that can execute and terminate individual primitives on command.
Given a new task, the VLM builds a plan and flags any primitive missing from the VLA's vocabulary as a primitive gap. Known primitives are executed by the steerable VLA, while each gap is attempted by a low-level controller parameterized by a VLM-proposed motion axis and signed magnitude. A VLM oracle verifies success, and new-primitive segments from successful rollouts are distilled into the VLA, forming a data flywheel that grows the skill set.
Stage 1. A demonstration is split into labeled primitives using gripper-state and dominant-motion cues.
Automatic primitive segmentation matches manual labeling within ~0.3 s (≈6 frames), and the primitive-trained VLA stops on command in 16/16 trials versus 0/16 for an unsegmented policy.
Stage 2. The VLM identifies and parameterizes a missing primitive, the robot executes it, and a VLM oracle verifies success.
Implementation. InSight fine-tunes a π0.5 VLA with LoRA, and uses Gemini 3 Flash as the VLM across four roles: demonstration segmentation, task planning, primitive-gap proposal, and image-based success checking. The framework is agnostic to the underlying VLA.
Each skill below is acquired with no human demonstrations of that skill. Starting from demonstrations of a different task, InSight identifies the missing primitives, practices them, and folds them back into the policy. On the left is what the robot was trained on; on the right is the new skill it acquires.
Trained onpick-and-place
Acquired (no human demos)block flipping (rotate-block primitive)
With no human demonstrations of the flip, InSight practices the missing rotate-block primitive and climbs to 75% block-flip success over 479 rollouts. An RL baseline (SAC) given the same budget never completes a flip (0%).
Trained onopen drawer
Acquired (no human demos)close drawer, from an out-of-distribution open start
100% drawer closing over 25 trials from the out-of-distribution open start (70/82 acquired primitives) — while still opening the drawer.
Two real-world primitives acquired separately, then chained into a long-horizon task.
Twisting open a cap
Trained onpick-and-place (top grasp)
Acquired (no human demos)twist open the bottle cap (new primitive: rotate cap)
Pouring
Trained onpick-and-place (side grasp)
Acquired (no human demos)pouring (new primitives: tilt to pour, tilt upright)
Composing into a long-horizon task
The separately acquired twist and pour skills chain into a 14-primitive task with no end-to-end demonstration.
Twist-then-pour: 14 primitives chained from the separately acquired twist and pour skills.
High per-primitive reliability compounds into 92% success on twisting, 96% on pouring, and 80% on the 14-primitive twist-then-pour composition, versus 32% / 16% / 4% for CaP-X and 0% even for a π0.5 baseline fine-tuned on the same demonstrations.
Because acquired primitives run as known skills, InSight spends far less VLM thinking time than a zero-shot planner: it finishes twisting in 55 s vs 109 s and pouring in 124 s vs 207 s against CaP-X, roughly 2× faster per trial.
After adding the twist and pour primitives, the unified VLA retains its success on both original pick-and-place skills (86%→100% and 100%→100%), showing no degradation on the tested base skills as the primitive vocabulary expands.
The motivating scenario: a contact-rich, non-prehensile skill acquired with no sweeping demonstrations.
Trained onscooping
Acquired (no human demos)sweeping (new primitive: lateral push)
InSight flags the lateral push as the missing primitive, acquires it, and succeeds in 5/5 evaluation trials.
Teaching robots new manipulation skills is expensive. Adapting a VLA to a new task typically requires collecting more human demonstrations. InSight makes the individual primitives of a VLA steerable, which lets a VLM identify the primitives a new task needs, acquire the ones the policy is missing, and recombine them. A robot can extend itself to new tasks without demonstrations of the target skill, and the primitives it acquires can be reused on later tasks.
M. Wang is supported by the NASA NSTGRO Fellowship, and S. Tian was supported by the NSF GRFP.
@misc{wang2026insight,
title = {InSight: Self-Guided Skill Acquisition via Steerable VLAs},
author = {Wang, Maggie and Osterberg, Lars and Tian, Stephen and
Shorinwa, Ola and Wu, Jiajun and Schwager, Mac},
year = {2026},
eprint = {2606.24884},
archivePrefix = {arXiv}
}