What is visual servoing? It is closed-loop robot control in which camera measurements steer the motion directly, so an arm can correct for a part that has shifted. Its main condition is that useful image features stay visible throughout the move.
Picture an arm told to grasp a part, but the part has been nudged since the last calibration. A pre-planned path misses it. A visual servoing loop keeps looking, compares what the camera sees with what it should see, and corrects until the two match. This sits within the wider field of robot vision systems, but it does a different job from mapping, because the image feeds the controller rather than a map.
Vision as a control signal, not a one-off measurement
Visual servoing is feedback control with a camera as the sensor. The robot does not look once, compute a target position and then move blindly. It measures, acts and measures again, so errors in the arm or the scene are corrected on the way. The common misreading is that this is simply “vision, then motion”. The defining feature is that the vision output is part of the loop.
The best-known form is image-based visual servoing (IBVS). According to a summary of IBVS research, IBVS regulates pose and trajectory using real-time image feature feedback. Task objectives are defined directly in terms of image-space features, such as where a point should appear in the camera frame. The same summary reports that stability analyses generally use Lyapunov-based techniques or exploit the exponential decay of error under proportional control, which is why a simple gain on the error is the usual starting point.
Common misunderstandings at a glance
| Common assumption | What is supported | Planning use |
|---|---|---|
| Visual servoing needs a full model of the scene | IBVS defines objectives in image space and regulates motion from image feature feedback, without requiring explicit reconstruction. | Start with features you can detect reliably rather than a scene model. |
| Any visual feature will do | Feature primitives include points, lines, SIFT keypoints and image moments, and effectiveness is tied to feature observability. | Choose features that stay visible and well separated during the whole motion. |
| The camera view can be ignored once the move starts | Performance is linked to latency and robustness to occlusion and outlier events. | Keep the camera’s view of the target clear and the image-to-command delay short. |
| The learned approach replaces the control loop | Two-stream CNNs can learn a mapping from image pairs to task-space pose errors and velocity commands, still feeding a controller. | Treat learning as a way to estimate the error, not a way to skip feedback. |
These points are drawn from the IBVS overview on regulation, feature types and performance factors.
Image-based versus position-based visual servoing
The difference lies in what the controller regulates. IBVS drives the error measured in the image itself. Position-based visual servoing (PBVS) instead works from an estimate of the target’s pose relative to the camera and regulates that pose. The practical consequence is where the risk sits.
- IBVS depends on feature quality. Because the objective lives in image space, the loop needs features that can be tracked frame after frame. It avoids an explicit reconstruction of the scene, which keeps the pipeline short.
- PBVS depends on the pose estimate. If the estimated pose of the target is wrong, the controller faithfully drives toward the wrong place. It suits setups where a good target model and calibration are available.
- The choice follows the task. For approaching a target visible in the image, with unknown or awkward depth, IBVS is the natural fit. Its summary reports that IBVS is pervasive in manipulation, including grasping under unobservable depths.
Neither approach is a universal upgrade over the other. The decision comes down to whether image features or a pose estimate is the more dependable quantity in the specific cell.
Eye-in-hand and eye-to-hand cameras
Eye-in-hand means the camera is mounted on the robot, typically at the wrist or gripper, so it moves with the end effector. Eye-to-hand means the camera is fixed in the workspace and watches the arm and target from outside. The IBVS summary notes that IBVS is widely used with eye-in-hand systems in manipulation.
Eye-in-hand gives a close, target-centred view that sharpens as the gripper approaches, and the image error maps naturally onto the motion the gripper needs. The cost is that the view changes with every move, so the target can leave the frame or be blocked by the gripper itself. Eye-to-hand gives a stable, wide view of the work area and keeps both the arm and the target in sight, but the view of fine detail near the gripper is limited and the arm can occlude the target.
Choose eye-in-hand when the task is a precise final approach to a target. Choose eye-to-hand when the whole workspace needs to stay in view or the arm should stay light. The fixed view below shows the kind of shared workspace an eye-to-hand camera would overlook.
What the image Jacobian does
The image Jacobian, also called the interaction matrix, links camera motion to feature motion. In the IBVS summary, feature motion under camera motion is governed by this matrix. It answers the question the controller needs: if the camera moves this way, how will each tracked feature shift in the image?
Control runs that relationship in reverse. The controller knows how far each feature is from its desired image position and needs a camera velocity that shrinks that error. Classical IBVS does this with a pseudoinverse-based control law using the Moore-Penrose pseudoinverse, which converts the image error into a velocity command.
Why depth enters the picture
The Jacobian depends on how far each feature is from the camera, and a single camera does not measure that directly. The summary reports that monocular IBVS handles depth heuristically, approximates it as constant, or pre-computes it from CAD models or stereo. A fixed depth is often sufficient for local convergence when the feature geometry is well conditioned. When more accuracy is needed, direct measurement of depth through stereo disparity allows a more accurate Jacobian computation.
This is why a rough depth assumption does not doom the method, but a badly conditioned feature layout can. Spread features sensibly across the target instead of clustering them.
A simple setup, step by step
A minimal eye-in-hand tutorial setup uses a fiducial marker on the target and a camera on the gripper. A fiducial marker is a printed pattern with easy-to-detect corners. The IBVS summary describes dual-arm cooperation work that uses AprilTag corners as visual features, which shows the idea works in practice.
- Pick the features. Use the marker’s corner points as the tracked features.
- Record the goal view. Place the gripper where the grasp should happen and store where each corner appears in the image. These are the desired feature positions.
- Measure the error. In every frame, compare the current corner positions with the desired ones.
- Compute the command. Apply the image Jacobian pseudoinverse to the error, with a depth assumption such as a constant value, to get a camera velocity.
- Move and repeat. Send the velocity to the arm, grab the next frame and loop until the error is small.
The robot ends at the stored goal view regardless of small errors in where the target started, because the loop corrects against what the camera actually sees. Knowing where the target sits in the world is a separate question, handled by visual localization for robots, and recognising a revisited place in a map belongs to loop closure slam rather than to servoing.
Where visual servoing goes wrong
Failures usually trace to the image side of the loop. The IBVS summary links effectiveness to feature observability, Jacobian conditioning, robustness to occlusion and outlier events, and latency. Each maps to a concrete check.
- Feature loss. If a feature leaves the frame or is occluded, the error is no longer measurable. Pick features that remain in view and add a fallback behaviour for dropouts.
- Poor conditioning. Features that are nearly collinear or tightly clustered give a Jacobian that responds weakly to some motions. Spread them out.
- Outliers. A single mis-detected corner can pull the whole command off course. Reject detections that jump implausibly between frames.
- Latency. A delay between image capture and motion command turns a stable loop into an oscillating one. Keep the image-to-command pipeline lean before raising the gain.
Learning-based variants are an option when an analytic Jacobian is hard to obtain. Two-stream CNNs can learn a direct mapping from image pairs to pose errors and velocity commands without analytic Jacobian estimation. That trades the explicit geometry for training data, so the conditioning checks above give way to questions about how well the training covers the task.
Frequently asked questions
| Question | Answer |
|---|---|
| What is visual servoing? | Visual servoing is feedback control in which camera data drives robot motion directly. The camera measurement is the error signal in the loop, not a separate planning step. |
| What is the difference between image-based and position-based visual servoing? | Image-based servoing regulates the error between current and desired image features. Position-based servoing regulates an estimated pose of the target, so it relies on that estimate being accurate. |
| What is the difference between eye-in-hand and eye-to-hand? | An eye-in-hand camera is mounted on the robot and moves with the gripper. An eye-to-hand camera is fixed in the workspace and watches the arm from outside. |
| What does the image Jacobian do? | It relates camera velocity to the rate of change of image features. Classical IBVS inverts it with a pseudoinverse to turn feature error into a velocity command. |
| Can a fiducial marker be used as the target? | Yes. Marker corners can serve as image features, and AprilTag corners have been used as features in dual-arm cooperation work. |
