How do robots see? A camera’s image sensor captures light as pixels, and software then cleans the image, finds objects, works out where they are and passes coordinates to the controller. Good lighting and calibration decide how reliable that is.
This walk-through follows a single frame from lens to decision, without going deep into any one technique. For the wider field, see our overview of robot vision systems.
Capturing the frame: camera, image sensor and light
The camera is the robot’s eye, and its image sensor records the scene as a grid of pixels. Every later step works from that grid, so the frame has to contain what the software needs. That makes lighting a design decision rather than an afterthought. Tech Briefs calls lighting an important element of any image processing system, and it has to suit the task to give usable images.
Hardware choice is a separate decision, covered in our guide to the best cameras for robots. What matters here is where the captured image gets analysed.
Where the image is analysed
Tech Briefs describes two arrangements. In a smart camera, the captured images are analysed directly in the camera housing. Industrial cameras instead transmit their images to a PC system for analysis, which usually gives higher precision and speed. A smart camera keeps the setup self-contained. The PC route suits tasks that demand more precision or speed.
Cleaning up and calibrating the image
Before a robot interprets a frame, software tidies it and ties it to the real world. Averroes lists preprocessing steps such as Gaussian smoothing to reduce noise. For segmentation it lists thresholding (such as Otsu), edge detection (Canny or Sobel) and learned segmentation, which separate the object of interest from its background.
Calibration is the quieter but equally important step. Averroes names intrinsic parameters and distortion correction as the core of camera calibration. A peer-reviewed review of machine vision in manufacturing states the purpose: calibration corrects lens distortions and converts image pixels to real-world coordinates. A pixel position means nothing to a wheel or a gripper until it is mapped to a position in the robot’s world.
Finding objects: classic techniques and neural networks
Rule-based methods
The same review notes that industry-level machine vision uses edge detection, template matching, color analysis, morphological operations and stereo imaging. These methods follow explicit rules, so it is easy to see why they succeeded or failed on a given frame.
Learned methods
Learned methods let the software work out the patterns itself. Averroes describes CNN backbones, meaning convolutional neural networks, that learn patterns from raw pixels. The review found the vision systems it covered use predominantly self-designed CNN architectures, YOLO variants or traditional machine-learning models. Robotiq summarises the payoff: computer vision allows robots to detect objects, recognize patterns and navigate complex environments.
From detection to decision
A detection is not yet an action. Averroes describes robot vision as a closed loop: capture, preprocess, detect a part, compute its 6D pose (position and orientation), send coordinates, pick, then verify. The table sets out what each stage does for the robot.
| Stage | What happens | Why it matters |
|---|---|---|
| Capture | The camera records a frame under suitable lighting | Poor light limits everything downstream |
| Preprocess | Noise is reduced and the image is segmented | Separates the target from its background |
| Detect | The target part or obstacle is located in the image | Turns pixels into a named object |
| Compute pose | The object’s 6D pose is estimated | The robot needs position and orientation, not just presence |
| Send coordinates | Corrected 2D or 3D gripping coordinates go to the robot’s control system | This is the hand-off from vision to motion |
| Pick and verify | The robot acts, then the result is checked | Closes the loop so mistakes are caught |
Stages and coordinate hand-off follow Averroes and Tech Briefs.
One more calibration links the camera to the arm. Hand-eye calibration determines the position and orientation of the camera sensor’s coordinate system relative to the robot’s tool, as the manufacturing review explains. Without it, a correct detection could still send the gripper to the wrong place.
Speed matters too. Averroes says real-time robot vision often has tight latency requirements, often under 200 ms and sometimes far lower. In return, Tech Briefs notes that vision gives robots considerably better decision-making capabilities to react flexibly to unexpected situations.
Machine vision and computer vision compared
The two terms overlap, and robot systems use both. Roughly, machine vision is the calibrated, measurement-oriented toolkit of industrial inspection and guidance. Computer vision is the broader field of understanding images.
- Machine vision: the review associates it with camera calibration, edge detection, template matching, color analysis, morphological operations and stereo imaging.
- Computer vision: Robotiq describes it as what lets robots detect objects, recognize patterns and navigate. The University of Michigan notes that traditional computer vision often relies on RGB cameras.
For a beginner, the useful takeaway is that a working robot pipeline blends them. Calibration and geometry come from the machine vision side, and learned detection from the computer vision side.
Seeing depth: can one camera give 3D?
A single standard camera records a flat image. The routes to depth that Averroes describes are two cameras, which estimate depth by triangulation, or active sensors that project light, such as time-of-flight and structured light. Time-of-flight is robust in varying light but has lower resolution, which is the trade-off to weigh for a robot that crosses bright and dim rooms.
A pair of cameras needs its own calibration. The manufacturing review notes that stereo calibration is used in 3D vision applications, finding the intrinsic parameters of each camera and the extrinsic parameters between them. A depth camera packages one of these approaches into a single unit.
Vision indoors: mapping and knowing where you are
Indoor robots use vision to build and use maps as well as spot objects. The University of Michigan describes robots combining perception with Simultaneous Localization and Mapping (SLAM), mapping an environment while categorizing what they see. Averroes adds that this enables navigation and mapping so robots can move through dynamic spaces. When the camera supplies the evidence, this is visual SLAM.
Recognising a spot the robot has visited before is a related but distinct task, covered in our piece on visual place recognition.
Why cameras are paired with other sensors
Cameras are powerful but not complete. Robotiq says vision will remain a fundamental sensing modality, alongside tactile and force sensing. The reasons for pairing it with other sensors are specific:
- Different physics. The University of Michigan’s perception research goes beyond RGB cameras to LiDAR, sonar, ground-penetrating radar, thermal cameras and touch sensors.
- No sense of touch. Robotiq points out that cameras cannot measure physical interaction, such as contact force, friction or slip.
- Fewer failure modes. Averroes lists sensor fusion of vision, lidar, IMU and tactile sensing as a way to reduce failure modes.
- Shared learning. Michigan researchers also transfer what one sensor, like a camera, has learned to improve another, such as radar.
The practical rule is to let the camera supply rich scene detail and add a sensor wherever the task depends on something an image cannot show.
Frequently asked questions
What does eye-in-hand mean?
Eye-in-hand is another name for vision-guided robotics, where the camera moves with the robot. The manufacturing review notes that such systems can adapt inspection paths, and hand-eye calibration ties the camera’s view to the tool.
How accurate is neural-network vision?
In the manufacturing studies reviewed in the same paper, defect detection and classification accuracy frequently exceeded 95%, reaching 98–100% in some controlled settings. Those figures describe factory inspection, so treat them as a ceiling for controlled conditions rather than a promise for any robot.
What skills does robot vision work involve?
Averroes lists optics, lighting, OpenCV and machine learning as the skills of a vision engineer, which mirrors the pipeline: the camera and light, then the software that interprets the image.
