Definition: Optical tracking is a method that determines the position and orientation of objects, people or devices with cameras and image analysis. In each image, the software looks for known features: markers, light points or natural structures. From these, it calculates where the camera and the object are relative to each other. It is the basis of most AR and VR systems.

In B2B mechanical engineering, optical tracking is used for: AR instructions in service, headset and controller tracking in VR training, ergonomic analyses with motion capture, worker guidance at assembly stations, and optical 3D metrology in quality assurance.

Advantage: Contactless, accurate and scalable, from the smartphone to the measurement hall with dozens of cameras.

English: Optical Tracking
German: Optisches Tracking, kamerabasiertes Tracking, optische Positionsbestimmung

How It Works

The optical principle

A camera projects three-dimensional space onto a two-dimensional image plane. When the system knows the optics of the camera (focal length, distortion) and the geometry of the features it looks for, it can reverse the calculation:

  1. Capture the image: One or more cameras capture the scene, often in the infrared range, to filter out stray light.
  2. Find features: Algorithms detect points, edges or patterns and determine their exact location in the image.
  3. Match: The system matches each detected feature to a known point: a marker, an LED or a previously stored image feature.
  4. Calculate the pose: From several matches, the system gets the pose with six degrees of freedom (position and rotation).
  5. Smooth and fuse: Filters and inertial sensors (IMU) compensate for noise and bridge moments when the view is blocked.

Triangulation with several cameras

When two or more calibrated cameras see the same point from different angles, the intersection of the lines of sight gives its 3D position. The larger the distance between the cameras (baseline) and the better the calibration, the more accurate the result.

Requirements

  • Line of sight: What no camera sees cannot be tracked.
  • Calibration: The camera parameters and the position of the cameras relative to each other must be known.
  • Contrast: Features must stand out clearly from the background.

Outside-In vs. Inside-Out

Feature Outside-in Inside-out
Camera position Fixed in the room On the headset, tablet or controller
What is tracked Objects with markers in the room The device itself, relative to the environment
Accuracy Very high, reproducible in the measurement room Good, depends on the environment
Setup Complex, fixed area Hardly any effort, mobile
Typical use Motion capture, CAVE, metrology Standalone VR, smartphone AR

Outside-in: Cameras on walls or trusses observe the room and capture objects with reflective or active markers. Such systems track many measurement points at the same time. They are used in research and in automotive and aerospace development.

Inside-out: The cameras are on the device. In this way, headsets capture their own movement in the room and also the infrared LEDs of their controllers. The method behind this is usually SLAM.

Distinction: Valve's Lighthouse system is also optical, but it works the other way around: base stations send out laser lines, and photodiodes on the device measure when the lines hit them. Cameras and image processing are not involved.

Markers or Natural Features

You can classify optical tracking by what the camera uses for orientation:

  • Passive markers: Retroreflective spheres reflect infrared light directly back to the camera and appear as bright points. Several spheres in a fixed arrangement form a body that the system can identify clearly.
  • Active markers: Infrared LEDs emit their own light. They need power, but they are visible at a greater distance.
  • Image markers: Printed patterns such as QR-like codes. Details are in the entry Marker Tracking.
  • Natural features: Corners, edges and textures of the environment or of a component serve as anchors. See Markerless Tracking and Image Tracking.

The choice is a compromise: markers give stable, accurate results, but you must attach and maintain them. Natural features need no preparation, but they are more sensitive to smooth, reflective or single-color surfaces, which are common on machine enclosures.

B2B Applications

Service: AR at the machine

A technician points the tablet at a machine tool. Optical tracking identifies the machine by a marker on the housing or by its shape, and shows maintenance steps in the correct position. A digital twin can also show live data from the IoT.

Training: VR training

Inside-out cameras in the headset capture the head and controllers of the participants. For training on large models or with several people in the same room, an outside-in system can be worth it, because it gives a fixed, shared frame of reference.

Design review and ergonomics

In projection rooms (CAVE) or in ergonomic studies, markers track the head, hands and body of test persons. This lets you check whether an operator can reach an emergency stop or whether an assembly position causes strain. To record complete motion sequences, see Motion Capture.

Metrology and quality assurance

Photogrammetry systems photograph a component with coded reference marks from many angles and calculate 3D coordinates from the images. These reference points then serve as a framework for structured light scanners. Hand-held probes and scanners are also tracked optically.

Sales and trade shows

At trade shows, a tablet or headset application places a virtual system on the booth floor. For kiosk installations, a permanently attached image marker gives reliable tracking even when the lighting changes.

Practice in the Factory Hall

The lab and the production environment are very different. Dust, oil mist, temperature changes, sunlight through skylights and vibrating hall floors affect optical systems. These measures have proven effective:

  • Enclosed cameras and robust markers that you can clean
  • Regular calibration, ideally with an automatic self-check
  • Redundancy: several cameras per area, so that people or forklifts that block the view do not cause a failure
  • Test runs under real conditions at different times of day, before you release a system for training or service

Data privacy: Cameras in the room also capture employees. For outside-in systems and camera access on headsets, you must clarify whether only marker positions or also images are stored (data privacy).

Frequently Asked Questions (FAQ)

How accurate is optical tracking?

This depends strongly on the system. Stationary measurement systems with calibrated cameras reach submillimeter accuracy under lab conditions. Inside-out tracking in headsets and smartphones is designed for visualization and training, not for measurement tasks.

What interferes with optical tracking?

Occlusion, backlight, direct sunlight, reflective surfaces, dust on cameras and markers, and very uniform surfaces. In factory halls, there are also vibrations that can decalibrate cameras.

Why do many systems use infrared?

Infrared light is invisible to people and is easy to filter. Because of this, reflective markers or LEDs stand out clearly from the background, and tracking depends less on the room lighting.

Do I need outside-in tracking for VR training?

In most cases, no. Standalone headsets with inside-out tracking are sufficient for training and presentations. Outside-in is worth it when you need very large areas, many objects tracked at the same time, or metrology-grade accuracy.

How is optical tracking different from inertial tracking?

Inertial sensors measure acceleration and rotation rate very quickly, but they drift over time. Optical tracking gives absolute positions without drift, but it needs a line of sight. Modern systems combine both.

Industry Standards and Technologies

Tracking methods: - Marker tracking – tracking with image markers - Markerless tracking – tracking without markers - Image tracking – detection of images - SLAM – simultaneous localization and mapping - Motion tracking – tracking of movement - Eye tracking – tracking of gaze

Sensors and capture: - Sensors – industrial sensor technology - LiDAR – laser-based depth measurement - Motion capture – recording of movement - Reverse engineering – from the component to the model

Applications: - Augmented reality – AR basics - Virtual reality – VR basics - Digital twin – digital replica - ARKit and ARCore – mobile AR platforms