Speckle pattern depth

A projector next to the camera throws a fixed pattern of random dots onto the scene. Every small window of the pattern looks different, so a single image is enough to find where each window came from. Matching it against a reference image of the dots on a plane at a known depth gives a shift along the row, and the shift gives the depth.

Camera image of the scene click to select a window
Reference image: the dots on a plane at z0z_0 best match true position
Estimated depth z^\hat z bright = near no reliable match off by more than 2 cm
True depth zz same gray scale not lit by the projector
NCC of the selected window with the reference, over the shift ss in px best shift true shift

Texture where there is none

Window matching in stereo needs texture, and many surfaces have none: in this scene, every surface has a single color. A speckle projector solves this by adding texture: a laser shines through a diffuser and throws a fixed pattern of thousands of random dots, usually in the near infrared, so that people do not see it. The pattern is random, so every window of it is different from every other window along the row, and a single camera image is enough. This is how the first Kinect worked.

A reference image instead of a second camera

The projector sits a baseline bb beside the camera, with a parallel optical axis. Its dots therefore move only along the rows, just like the pixels of a rectified stereo pair. Instead of a second image, the device stores one picture of the dots on a plane at a known depth z0z_0, taken during calibration. A point at depth zz shows the dots shifted by

s=f b(1z−1z0)⇒z=11/z0+s/(f b)s = f\,b \left( \frac{1}{z} - \frac{1}{z_0} \right) \quad\Rightarrow\quad z = \frac{1}{1/z_0 + s/(f\,b)}

compared to the reference image: to the right for points in front of the plane, to the left for points behind it. The relation is the stereo formula z=f b/dz = f\,b / d, measured relative to the reference plane.

Finding the shift

For each pixel, a window around it is compared with windows of the reference image along the same row, for every shift in the search range. The comparison uses the normalized cross-correlation

NCC=∑(I−Iˉ)(R−Rˉ)∑(I−Iˉ)2∑(R−Rˉ)2\text{NCC} = \frac{\sum (I - \bar I)(R - \bar R)}{\sqrt{\sum (I - \bar I)^2 \sum (R - \bar R)^2}}

which ignores the gain and bias between the two windows: a dark surface or one at a slant reflects the dots more weakly, but in the same arrangement. A parabola through the best cost and its two neighbors gives the shift to a fraction of a pixel, and pixels whose best NCC is too low get no depth. These include the shadows, which the projector cannot reach, and windows that cross a depth edge, where half the window belongs to another shift.

Try this