Robotic Arm: Vision Language Grounding
A perception system that lets a robotic arm act on plain language. Say "pick the screwdriver" and it works out which pixels in the scene are the screwdriver, then hands that target to the arm.
The problem
A robotic arm can reach any coordinate you give it. The hard part is turning what a person says into that coordinate. Classic detection models only recognise the fixed set of classes they were trained on, so anything outside that list is invisible to the robot.
How it works
The system chains three models so that an open ended instruction ends up as a precise mask on a real object:
- spaCy parses the instruction and pulls out the target object and any qualifying context, so "take the controller device from the table" resolves to the controller rather than the table.
- CLIP scores regions of the camera image against that text, grounding the phrase in the scene without needing a predefined class list.
- SAM segments the winning region into a clean mask, giving the arm a precise boundary to plan a grasp against.
Because the grounding step is driven by language rather than a label set, the arm can be asked for objects it was never explicitly trained on.
Example commands
"pick the screwdriver"
"take the controller device from the table"
Why it matters
This is the layer that makes a manipulator usable by someone who is not a robotics engineer. It joins computer vision with language understanding so a robot can follow spoken or written instructions in a real, cluttered environment instead of a staged one.