Your indoor robot must cross a room without clipping a chair. Robot vision systems solve that by pairing a depth-capable camera with detection software, while a flat 2D camera suffices only for simple tasks.
- Choose depth first. If the robot must judge distance, pick a stereo pair, a time-of-flight sensor or another depth camera rather than a flat camera.
- Add detection second. Object detection, usually built on a convolutional neural network, tells the robot what is in the image and where.
- Prototype cheaply. Test your idea with a webcam and OpenCV before committing to expensive hardware.
If you want the basics of image capture before comparing hardware, begin with how do robots see. The rest of this guide treats vision as a perception toolkit and compares camera types and methods for depth sensing, detection and visual mapping, so you can choose where to start.
What computer vision does inside a robot
Computer vision in robotics is the capability that lets a robot react to changes in its task. According to an expert overview of robot vision, a robot without vision follows pre-programmed commands, whereas a robot with vision can respond to what its camera reports. That difference matters most in a variable environment, and an occupied indoor space is exactly that.
Robot vision became a practical tool in the 1980s and 1990s, when cameras began to image each piece a robot handled, as an industrial robotics overview describes. Those cameras were either mounted on the robot or positioned statically. The choice still shapes system design today. A camera fixed above a work area watches the scene from a known viewpoint. A camera on the robot moves with it and sees whatever the robot faces. A robot that travels through rooms needs the second arrangement, because a static camera cannot follow it everywhere.
When a robot does not need vision
Vision is not automatically worth its cost. The same expert overview notes that when a part is presented on a jig or another repeatable process, a camera gains you nothing, because the robot already knows where the part is. Vision earns its place when positions vary, when objects must be identified, or when the robot moves through space that changes. For an indoor autonomous robot, that is nearly always the case, so the practical question is which camera and which methods, not whether to use vision at all.
Pick your route through this guide
Readers arrive with different goals. Use the route that matches yours, then follow the linked guides for depth.
- You are choosing hardware. Read the camera section first, then the depth section. Together they tell you whether a flat camera is enough or whether you need distance information.
- You need the robot to recognise things. Go to the detection section, then check the calibration and conditions section, because lighting decides how well detection works in practice.
- You want the robot to map or locate itself. Read the mapping and localization section, which points to the guides on visual SLAM, odometry and place recognition.
- You are a developer who wants to try something today. Jump to the OpenCV section at the end and work outward from a webcam feed.
Choosing a camera for an indoor robot
No single camera is best for every indoor robot. The right choice follows from the task: a flat image is enough for some jobs, while others need distance, and the room’s lighting limits what any camera can do. For a fuller treatment of hardware, see the guide to the best cameras for robots.
Flat 2D cameras
A 2D camera captures flat images. A review of vision systems in modern robotics describes them as suited to basic tasks such as barcode reading and simple object detection. If your robot only has to recognise a label, read a code or spot an obvious object, a 2D camera keeps cost and processing demands low. Its limit is that a flat image does not by itself say how far away anything is. A robot that must steer around furniture will usually want more than a flat image gives.
Depth-capable cameras
A 3D camera provides depth information, which the same review calls crucial for tasks requiring spatial awareness, such as picking objects from a bin. Depth sensors measure the distance to objects, which supports 3D mapping and obstacle avoidance. The expert overview of robot vision adds that some industrial camera models are capable of 3D vision, which lets a robot work through layers of parts. For an indoor robot, the practical reading is simple. If obstacle avoidance or mapping is part of the job, plan for a depth camera from the start.
Resolution, field of view and lighting
Resolution and field of view trade against each other. The same overview points out that getting closer to a part can make up for poor resolution, but this can hurt the field of view. Economical camera configurations also have less resolution, which means a less crisp picture and less data for small details. A mobile robot cannot always move closer to what it needs to read, so decide early whether you need to see a wide area or fine detail, and budget for the camera that fits.
Lighting is the other deciding factor. The overview warns that poor or inconsistent lighting can affect a system’s ability to identify points of interest, and that supplemental lighting is often used to rectify these issues. The review of vision systems lists varying lighting and occlusions among the conditions in which a system must still perform accurately. If the robot will cross sunlit windows, dim corridors and bright offices, weigh lighting robustness as heavily as raw resolution.
Camera types side by side
The table compares the main camera approaches by what they measure and when each makes sense for an indoor robot.
| Camera approach | What it does | Planning use |
|---|---|---|
| 2D camera | Captures flat images for basic tasks such as barcode reading and simple object detection | Choose when the task is recognition without distance, and you want the lightest processing load |
| Stereo vision | Uses two cameras to simulate human binocular vision, calculating depth by comparing the two images | Choose when you want depth from ordinary camera hardware and can treat the pair as one calibrated system |
| Time-of-flight | Measures the time a light signal takes to bounce back from an object to determine its distance | Choose when you want a direct distance reading rather than depth inferred from image comparison |
| Structured light | Projects a known pattern, such as a grid or stripes, onto the object and measures how it deforms | Choose when a projected pattern suits your scene, and check it against the room’s lighting |
These principles come from a survey of vision systems in modern robotics.
Measuring depth with cameras
Robots measure depth with cameras in several ways, and each method suits a different mix of hardware, lighting and processing power. The question of how robots measure depth has its own guide at how do robots measure depth. This section helps you pick between the main principles.
Stereo vision
Stereo vision uses two cameras to simulate human binocular vision. It calculates depth by comparing the two images, as the robotics vision review explains. The appeal for robot builders is that depth comes from comparing images rather than from a separate emitter. The cost is that the comparison happens in software, so the quality of the result depends on how well the two views are matched. That is why calibration, covered below, matters so much for stereo. The guide to stereo vision robotics goes into the method in detail.
Time-of-flight
A time-of-flight sensor measures the time it takes for a light signal to bounce back from an object to determine its distance. Because the sensor reports distance directly, it is a natural fit when you want a depth reading without relying on comparing two views. See time of flight camera robotics for the full treatment.
Structured light
Structured light projects a known pattern, such as a grid or stripes, onto the object and measures how the pattern deforms. The sensor therefore supplies its own pattern as part of the measurement. When you evaluate such a sensor for an indoor robot, test it in the real lighting of the space, since lighting problems affect how reliably any vision system identifies what it needs.
Cameras or lidar?
Depth need not come from a camera at all. If you are weighing a laser scanner against camera-based depth, the comparison in lidar vs camera robot navigation sets the two approaches side by side. The decision usually turns on the same factors as before: what the task demands, what the room’s lighting allows and how much computing the robot can carry.
Detecting objects and people
Object detection identifies and locates objects within an image. The robotics review lists three families of technique for this: template matching, feature extraction and machine learning models. The broader guide to object detection for robots compares them for robot use. Here is how to choose among them.
Classical methods
Template matching and feature extraction suit situations where the target looks consistent. They are a sensible first experiment because they are simple to try on a live camera feed. Their weakness is the one that affects all vision: when lighting changes or something blocks the view, an approach tuned to one appearance can fail. The review names varying lighting and occlusions as challenges that systems must handle.
Convolutional neural networks
A convolutional neural network (CNN) is the standard machine learning tool here. The review states that neural networks, particularly CNNs, are used for image classification, object detection and image segmentation. Use a CNN when the objects vary too much for hand-built matching, or when you need to tell categories apart. People are no exception. A detector recognises whatever categories it has been built to recognise, so detecting people is a matter of what the model is trained on and how well the scene is lit and visible. For labelling every part of an image rather than boxing objects, see semantic segmentation robotics.
Reading codes and text
Some identification tasks need no learned model at all. The expert overview notes that barcode scanning or optical character recognition (OCR) can guide robot operations. If your indoor robot only needs to recognise labelled locations or items, reading a code is far simpler than training a detector.
From detection to action
Detection is only useful when its result reaches the robot’s controller. In bin picking, the camera localizes and orients parts in space, and this data is sent back to the robot controller. The same loop, in which camera output guides robot motion, is the idea behind visual servoing. If that interests you, read what is visual servoing.
Calibration and working conditions
Calibration is how you make a camera’s measurements trustworthy. In general terms, it relates what the camera records to real geometry so that positions and distances computed from images mean something. It matters most for depth: stereo depth comes from comparing two images, so the two cameras must be treated and checked as one system, and any shift in how they are mounted undermines the comparison.
How to approach calibrating a robot camera
Treat calibration as part of building the system, not a one-off chore. Calibrate after the camera is fixed in its final mounting position, and calibrate again if you move or replace it. Software libraries provide ready-made routines for this, so check the calibration tools in OpenCV’s documentation and follow the procedure your chosen camera’s manufacturer provides. Matching the procedure to the camera hardware is more reliable than improvising your own.
Controlling the lighting
Calibration cannot rescue a poorly lit scene. Poor or inconsistent lighting affects the ability to identify points of interest, and supplemental lighting is often used to rectify this. For an indoor robot, the practical approach is to test in the darkest and brightest spots of its route, and to add lighting where the image degrades, rather than assuming one good test is representative.
Mapping, localization and visual control
Beyond seeing single objects, robot vision systems help robots map spaces and know where they are within them. Depth sensors support this: they measure the distance to objects, which aids 3D mapping and obstacle avoidance. The remaining guides in this cluster each take one part of that task.
- Tracking motion. If your robot needs to estimate how it has moved from camera images, read what is visual odometry.
- Mapping while localizing. To understand how a robot builds a map and places itself in it, start with what is visual slam, then continue to loop closure slam for how a system recognises that it has returned somewhere familiar.
- Recognising places. For identifying where an image was taken, see visual place recognition, and for finding a robot’s position within a known space, visual localization for robots.
- Building 3D models. If the goal is a spatial model of the room itself, 3d mapping for robots using cameras covers the camera side of that job.
Choose among these by the question you need answered: how the robot moved, where it is, whether it has been here before, or what the space looks like. Depth hardware chosen earlier feeds all of them, which is why the camera decision comes first.
Compute and reliability limits
Capability costs processing power. The robotics review states that real-time processing of high-resolution images and 3D data requires significant computational power. A sharper camera or a depth stream is therefore not a free upgrade: it may demand a stronger onboard computer, which affects the robot’s size, cost and power budget.
Reliability is the second limit. Vision systems must perform accurately in diverse and challenging conditions, such as varying lighting and occlusions. A sensible design process pairs each capability with a stress test. Check depth in the room’s brightest and dimmest areas. Check detection when furniture or people partly block the view. Prefer the simplest approach that passes, because simpler pipelines are lighter on compute and have fewer ways to fail.
Starting with OpenCV and a webcam
A laptop, a webcam and OpenCV are enough to test most of the ideas above before you buy specialised hardware. Because a flat 2D camera suits basic tasks like barcode reading and simple object detection, a webcam is a fair first sensor for those experiments, as the vision systems review notes.
- Get a live feed. Display your webcam image on screen and confirm the picture is sharp and well lit before writing any detection logic.
- Try a classical method. Test template matching or feature extraction on an object that looks consistent, and note when changing light breaks it.
- Add a learned model. If classical methods fail on varied objects, move to a CNN-based detector, accepting the extra compute it needs.
- Add depth when distance matters. Once the robot must judge space, bring in a stereo pair, time-of-flight sensor or other depth camera.
- Test where the robot will work. Repeat your checks in the real room, with the lighting problems described earlier, and add supplemental lighting if the image degrades.
After this prototype you will know whether a flat camera is enough, which is the most valuable thing to learn before buying anything further. From there, the cluster’s guides on depth, detection and mapping each deepen one branch of the toolkit.
Frequently asked questions
| Question | Answer |
|---|---|
| Does every robot need computer vision? | No. A robot that receives parts on a jig or through another repeatable process gains nothing from a camera. Vision pays off where positions or surroundings vary. |
| Is a 2D camera enough for an indoor robot? | Only for basic tasks such as barcode reading or simple object detection. If the robot must gauge distance for mapping or obstacle avoidance, add depth sensing. |
| What reduces detection reliability? | Poor or inconsistent lighting and occlusions are the main named problems. Supplemental lighting is often used to correct lighting issues. |
| Which techniques find objects in an image? | Template matching, feature extraction and machine learning models. Convolutional neural networks are the machine learning approach most associated with classification, detection and segmentation. |
| Why can a better camera need a better computer? | Real-time processing of high-resolution images and 3D data requires significant computational power, so sensor upgrades can raise processing demands. |
