Single Camera Catadioptric Stereo System

Abstract

In this paper, we present a framework for novel

catadioptric stereo camera system that uses a single camera and a single lens with conic mirrors. Various possible designs of the catadioptric stereo system with single-viewpoint constraint have been developed in this framework. The proposed systems are compact wide-baseline stereo systems with a panoramic view. The simple structures of the systems reduce the problems of misalignment between the camera and mirrors, which make the system free of complex search procedure for epipolar line. The wide baseline enables very accurate 3D-reconstruction of the environment. Additionally, the system has all the advantages of the single camera stereo system that stem from the same physical characteristics of the camera. The feasibility of the system as a practical stereo sensor has been demonstrated with experiments in an indoor environment. 1. Introduction

With stereo vision, depth perception is possible by establishing stereo disparity between the two images from two distinct viewpoints. For a long time, it has been important tool for machine vision applications.

There have been many possible types of stereo vision systems proposed. Among them, an attractive approach is the single-camera stereo. Generally, the characteristics of the stereo cameras are slightly different. For stereo images acquired by the two cameras, the focal lengths of the cameras are not exactly the same and the alignment of the imaging sensors is not accurate. Moreover, differences in the characteristics of the two imaging sensors cause intensity differences between corresponding points in stereo images. Using the single-camera stereo system, these unwanted geometric and chromatic differences can be eliminated to increase the ability to find correspondences reliably. A single-camera stereo system that uses a couple of planar mirror has been suggested [1]. The rectified stereo system was developed so that the corresponding features lie on the

same scan line automatically [2]. The system complexity was reduced by using an image-splitting prism which is called biprism [3]. However, the system suffers from a short baseline, a narrow field of view and chromatic aberration.

Another attractive approach is the catadioptric stereo. A catadioptric vision system using mirrors has been a popular means to get panoramic images [4], which contains a full horizontal field of view (FOV). Generally, it consists of coaxially aligned conic mirrors and cameras. A stereo system coaxially combining a couple of catadioptric vision systems was proposed [5]. However, it is bulky and always suffers from the different physical characteristics of the cameras and a slight misalignment between the systems. One of the partial solutions is using double mirrors positioned in front of the single camera [6]. This system reduces the number of required camera, but lead to the problems such as the difficult calibration and complex stereo matching process. Another partial solution is the catadioptric stereo sensor using a conical mirror and a beam splitter [7]. This system reduce the number of used mirror by using a beam splitter, but the orthogonally positioned two cameras still make the system suffer from bulky size, camera misalignment, and different characteristics of the cameras. A novel and effective solution for these problems is using a single camera and a double lobed mirror [8][9][10], which is a combined coaxial mirror pair. However, the critical deficiency of the system is the short baseline.

In this paper, we propose a compact, wide-baseline catadioptric stereo system, with which the camera alignment procedure and the complex search procedure for the epipolar line are not necessary. Additionally, the system has all the advantages of the single-camera stereo system that stem from the same physical characteristics of the cameras. 2. Single-viewpoint (SVP) catadioptric image formation

In various computer vision applications, a wide field of view is necessary. To obtain a wide field of

view, catadioptric vision sensors are useful. They are arrangements of cameras and specially configured mirrors. One important design factor is that the shape and the positioning of the mirror should ensure the constraint of single effective viewpoint. If )(RZ is the profile of the mirror shape, the complete class of solutions is given by [4]

( )

( ) )0( 2221

)2( 212

2222

222

>⎟⎟⎠

⎞⎜⎜⎝

⎛ +=⎟⎟

⎠

⎞⎜⎜⎝

⎛++−

≥⎟⎠⎞

⎜⎝⎛ −

=⎟⎠⎞

⎜⎝⎛ −−−

kckkcRcZ

kk

kckRcZ

(1)

where c is half the distance between the desired virtual viewpoint and the effective pinhole of the imaging lens, and k is a constant of integration. The combination of c and k in the solution determines the mirror configuration [4]. The effective mirror shapes for the catadioptric camera are plane, ellipsoid, hyperboloid and paraboloid. They are used as the elements for the design of the proposed systems.

These systems have the characteristics of rotational symmetry, therefore considerations on the radial cross section of the system are sufficient for the design.

With the SVP optics, we can convert the projected image to a panoramic image seen from the effective viewpoint by establishing the relationship between the world point and its projection onto the image plane.

Hyperboloidal mirrors and ellipsoidal mirrors, which have two foci, satisfy the SVP constraint. As shown in figure 1 (a, b), the light ray from the world point going to the first focus ( F ), which is also an effective viewpoint, is reflected to the second focus ( F ′ ), which is also an effective pinhole, and projected onto the image plane.

Given the parameters of the mirror ( a , b , and c ) and the focal length of the camera ( f ), the relationship between the world point and its projection onto the image plane is

M

M

ZR

fr = (2)

where cmRZ MM 2+= (3)

and

⎟⎟⎠

⎞⎜⎜⎝

⎛−

++=

22

2

21

mba

mamcRM (for hyperboloid)

⎟⎟⎠

⎞⎜⎜⎝

⎛+

+−−=

22

2

21

mba

mamcRM (for ellipsoid). (4)

Here, m is the slope of the line from the world point to the focus F )2,0( c :

w

w

RcZ

m2−

= )0( >wR (5)

where the range of m means the vertical FOV from the virtual viewpoint, which is determined by the mirror configuration.

(a)

(b)

(c)

Figure 1: Schematic of the catadioptric imaging system using a hyperboloidal mirror (a), an ellipsoidal mirror (b), and a couple of paraboloidal mirrors (c). The foci of the mirror are denoted by F and F ′ in (a) and (b), 1F and 2F in (c).

Paraboloidal mirrors satisfy the SVP constraint.

However, these mirrors are used as a coupled form, because the entire light ray directed to the effective viewpoint reflects to the direction parallel to the axis of rotation and consequently does not converge at a single point. As shown in figure 1(c), the light ray from the world point going to the focus of the first paraboloidal mirror ( 1F ), which is also an effective viewpoint, is vertically reflected, then folded by the second paraboloidal mirror to its focus ( 2F ), which is also an effective pinhole, and projected onto the image plane.

Given the parameters of the paraboloids ( 1p and

2p ), the focal length of the camera ( f ), and the offset of the first paraboloidal mirror ( e ), the relationship between the world point and its projection onto the image plane is:

M

M

ZR

fr = (6)

where

22

22

2 4p

pR

Z MM +−= (7)

and )1(2 2121 mmpRR MM −+== . (8)

Here, m is the slope of the line from the world point to the focus of the first paraboloidal mirror 1F ),0( e :

w

w

ReZ

m−

= )0( >wR (9)

3. Single camera catadioptric stereo system

Generally, the catadioptric stereo vision system is an extension of conventional catadioptric vision system, which is a coaxial alignment of perspective cameras and conic mirrors. The coaxial configuration of the cameras and mirrors makes the epipolar line radially collinear, which makes the system free of the time consuming search process for complex epipolar curve in stereo matching.

The panoramic stereo based on a double lobed mirror was first suggested in [8] and subsequently improved in [9]. The system is composed of a single camera and a coaxially aligned double lobed mirror. However, the non-SVP optics makes the depth analysis complex and the short baseline results in extremely low depth resolution. Recently, a slight modification of the system was suggested in [10]. The sophisticated scheme of shifting the effective viewpoint of outer lobe succeeded in widening the baseline without breaking the SVP constraint. However, the

modification did not actually widen the vertical baseline, which is the most important parameter for 3D reconstruction of the horizontal panoramic view.

To widen the vertical baseline of the coaxial single camera catadioptric stereo system, a modification of the double lobed mirror system is possible. Figure 2 shows a schematic of the wide-baseline catadioptric stereo system. There are two different kinds of topology for the design as shown. One is projecting the upper view to the outer rim of the image plane and lower view to the center region (a), the other is exchanging the imaging region for each view by projecting the upper view to the center region of the image plane through the center hole of the lower mirror (b).

(a) (b)

Figure 2: Schematic of the catadioptric stereo system using hyperboloidal mirrors: (1) and (2) denote The primary and secondary mirrors, respectively. F and F’ denotes the foci of the mirrors. P denotes the effective pinhole of the camera.

In these stereo configurations comprising two pieces of mirrors and a single camera, mirrors that independently satisfy the SVP constraint are required. The possible types are hyperboloid and ellipsoid. By selectively adapting the two kinds of mirrors at the position (1) and (2) in figure 2 (a, b), a total of eight stereo configurations are possible. 4. Folded catadioptric stereo system

The systems introduced in the previous section have the advantages of long baseline and single camera configuration. However, the problem is the overall size. Generally, the length of the system should be longer than twice the length of baseline.

A more compact catadioptric stereo system can be obtained by folding the upper mirror to the level of imaging lens by reflecting with another supplementary mirror.

Figure 3 shows a general form of single-viewpoint folded system that uses two conic mirrors (1) and (2). This system has an equivalent single mirror system

with the same relation between the directions of scene point )(φ and their image coordinates that is determined by β [11]. The light rays from the scene going to the near focus 1F are reflected on the mirror (1) to the direction of its far focus 1F ′ (in case of paraboloidal mirror, the direction is vertical). The system is folded by placing another conic mirror (2) between the near and far foci such that the ray is reflected to the point P , which is the effective pinhole.

Figure 3: General form of single-viewpoint folded catadioptric camera system that uses two conic mirrors.

With this design scheme, there are nine different

possible mirror pairs to construct folded imaging systems with SVPs [11].

In the stereo configurations comprising three pieces of mirrors and a single camera, a folded catadioptric system and a mirror that independently satisfy the SVP constraint are required. Then, for each imaging topology shown in Figure 2, there are two choices for the selection of lower mirror (1) and nine different choices of the upper mirror (2) folding. In other words, for a single folding scheme, there are two choices for the selection of lower mirror (1) and two choices of topology between (a) and (b) in Figure 2. As a result, we can say that there are totally thirty six possible folded catadioptric stereo configurations.

For example, the folding configuration can be a primary hyperboloidal mirror and a secondary planar mirror in Figure 3. Then, we have four different types of design as shown in Figure 4. Fixing the mirror (1) and (2) we can choose a conic mirror for the upper view (3) among hyperboloid (a, c) and ellipsoid (b, d). The choice of topology for the image formation can be upper view-center configuration (a, b) or lower view-center configuration (c, d).

In these stereo generation scheme, the mirror (3) can be positioned anywhere over the camera along the axis of rotation. In particular, the location of the mirror can be separated and raised over the mirror (2) to widen the baseline.

(a) (b)

(c) (d)

Figure 4: An example of possible stereo configurations with a single selection of folded catadioptric system out of nine choices. Therefore, totally 36 configurations are possible. 5. Panoramic image generation and depth computation

All the single-camera catadioptric stereo systems, which have coaxial camera-mirror configuration, obtain stereo inputs with inner zone and outer rim as shown in Figure 5(a). One of the major advantages of these systems is the simple epipolar geometry. All the epipolar lines are radial in raw catadioptric image, and they changes to be parallel with the coordinate conversion to panoramic image as shown in Figure 5(b). This fact simplifies the search process for the stereo matching by letting the corresponding features lie on the same scan line

On the other hand, these types of stereo system suffer from the difference of resolution between each view. It reduces the stereo matching accuracy if the resolution of the input image is not so high. The resolution for the circular direction linearly decreases as the sampled point moves from outer rim to the center of the imaging circle. Moreover the radial resolutions are different for each view according to the mirror profiles. There was an attempt to achieve resolution invariant panoramic view by designing

mirror profile so that all the single pixels occupy the same size of solid angle [12]. However this approach breaks the SVP constraint. Moreover, the radial and circular resolutions for each view are different in stereo configuration.

To solve this problem, we adopt the scale space theory. Different levels of resolution are created by the convolution with the different Gaussian kernel [13]: )()(),( xx IsGsL ∗= σ . Here, I is the image with ),( yx=x , G is Gaussian kernel, s is scale factor and σ is reference standard deviation of the Gaussian kernel (σ is determined to have a minimum value to achieve anti-aliasing and noise reduction). It means that if s is 2, the resulting image is equivalent to the half image of the image with s is 1.

For each view, the scale factor s is defined for radial and circular direction. The value is determined by the ratio of resolution between the two corresponding projected locations for the given vertical angle φ :

⎟⎟⎠

⎞⎜⎜⎝

⎛=

φφ ddr

ddrs oir

i ,1max

⎟⎟⎠

⎞⎜⎜⎝

⎛=

φφ ddr

ddrs ior

o ,1max

1=cis

i

oco r

rs = (10)

where the subscript ‘i’ and ‘o’ represent inner and outer region, and superscript ‘r’ and ‘c’ represent radial and circular direction, respectively. The applied Gaussian kernel is depicted in Figure 5(a).

The coordinate conversion from the catadioptric raw image to the panoramic images is possible both for the spherical and cylindrical coordinate. All the sampling scheme is similar except the difference of vertical axis between vertical angle φ and height Z .

For the stereo matching between the two converted panoramic views, any conventional algorithms are applicable. Once correspondence between image points has been established, depth computation in both spherical and cylindrical panorama is straightforward by simple triangulation [5].

Figure 6 shows a sampling of the depth resolution in vertical cross section. The sampling is obtained by computing depth for every possible pair of image correspondences. The depth resolution mainly depends on camera resolution, the length of baseline and camera-mirror configuration.

Figure 5: Panoramic image generation scheme. (a) shows captured image which contains two different views with inner and outer region. The sampling is conducted by applying Gaussian kernel. The kernel size is determined by the image resolution. (b) is the converted panoramic images for each view. The resulting resolutions between the two are the same.

Figure 6: Depth resolution. Each point represents the estimated position calculated by all the possible pair of image correspondence in a single epipolar plane. This instance is generated with the configuration of Figure 4(c), the baseline is 100mm, and the diameter of the imaging circle is 1200 pixels. ‘ ∗ ’s on z axis represent the virtual viewpoints of the stereo system.

6. Experiment

To demonstrate the feasibility of the system, an experimental system setup was built as shown in Figure 7, which is configured as shown in Figure 4(c) wherein one planar and two hyperboloidal mirrors are used. Generally, hyperboloidal mirrors are recommendable in that they simplify the system structure and do not permit horizontal panoramic view. The length of the baseline was set to 100mm which is adequate for indoor usage. The overall size of the mirror system was designed to have 140mm tall and 80mm wide (the width of mirror is 60mm). The mirror system is mounted in front of the camera. The system is designed to have high resolution horizontal view which is most important in general indoor applications. The vertical field of view was set to more than o16 for both upward and downward views. A 8.11 inch CCD camera with 1200 horizontal scan lines (Point grey - Scorpion) and a lens with 12mm focal length were used to capture the image.

Figure 8 shows a catadioptric stereo image of our laboratory captured by the given experimental setup. All the corresponding points between each view are radially collinear.

The problem of this system is defocusing. Because the system is the combination of two different catadioptric systems, the focusing range is largely different between them. Even in a single view, the focusing range varies radially. A partial solution is to reduce the aperture size. It increases the depth of field at the cost of shutter speed.

To convert the input image to stereo panoramic views, we compute the correspondence between the light ray directions from the virtual viewpoint and the position in the acquired image using the projective geometry explained in section 2. Then, to speed up the algorithm, these data are saved in a lookup table and referred at each iteration. Gaussian smoothing scheme that is described in section 5 is optionally applicable for better matching accuracy. If the resolution of input image is very high or real-time performance is required, simple bilinear interpolation is sufficient. Figure 9(b, c) shows generated panoramic stereo pair which is defined in spherical coordinate. The size of images are 1800x160 which indicates the sampling resolution is

o2.0 . Stereo matching was conducted with simple

window-based correlation search - sum of absolute difference (SAD). The correspondence was searched for the same scan line because the epipolar lines are collinear. The size of window is 9x9. Additionally, reverse verification [14] is applied to cancel out

probable false matching which prone to generate unwanted clutter. Figure 9(a) shows the disparity map, in which nearby objects such as tables and a robot appear brighter than distant objects such as bookshelves and racks.

In figure 10, panorama and its disparity map from two different sampling methods are zoomed to compare. (a) and (b) are results from linear interpolation and resolution equalization for each. We can observe the aliasing effect is suppressed and resolutions for each view are equalized in (b). Accordingly, in stereo matching, the number of filtered points (black dots in disparity map) from the reverse verification becomes smaller in case of resolution equalization.

Figure 11(a-e) are results of 3D reconstruction with the panoramic images and disparity map in figure 7(a-c). We can observe the 3D sensation with the change of the viewing direction from o16− to o16 . Figure 12 is the vertical projection of 3D reconstruction with a height of 10cm near the level of tabletop. We can observe the linear edges of nearby tables and the objects on them. Even with a simple stereo matching algorithm, the disparity map and the depth map show that the reconstruction accuracy is quite good except the self-occlusion regions and the ambiguous regions (specular region, textureless region, and a region with repetitive pattern).

Figure 7: Single camera catadioptric stereo system for panoramic view. The system is 140mm tall and 80mm wide. The length of the baseline is 100mm. The mirror system is mounted in front of the camera.

Figure 8: A catadioptric stereo image captured by the proposed system shown in Figure 7. The resolution of the image is 1600x1200. The epipolar lines in two different views are radially collinear.

(a) (b)

Figure 10: Comparison of sampling method for panoramic image between (a) linear interpolation (b) resolution equalization. Top: disparity map based on top view, middle: top view, bottom: bottom view.

(a)

(b)

(c)

(d)

(e) Figure 11: Results of 3D reconstruction: 3D sensation is observable with the change of the viewing direction (a) o16+ (b) o8+ (c) o0+ (d) o8− (e) o16−

(a)

(b)

(c) Figure 9: Panoramic stereo images produced by the raw image in Figure 8, and corresponding disparity map. The disparity map (a) was computed from the two panoramic images – top view (b) and bottom view (c). Disparity map was computed by simple block matching algorithm (SAD). The size of window is 9x9. The sizes of images are 1800x160. It covers field of view of

oo 32360 ×

Figure 12: Outline of the environment. Obtained 3-D reconstruction in height -10cm to -20cm was vertically projected. 7. Conclusion

A framework is developed for the design of single camera catadioptric stereo system. A total of 44 possible stereo configurations with single-viewpoint constraint are suggested.

Single camera configuration makes the whole system simple. In addition, the same physical characteristics of the camera make the stereo matching simpler with better accuracy. The mirror-folding scheme makes it easy to design a more compact system with a wide baseline.

The single camera stereo scheme and folding scheme caused the problem of resolution difference between stereo pair. We solved it by applying the scale space theory.

The result demonstrates that the proposed system provide sufficient accuracy for various machine vision applications in which the panoramic range measurements are important. It can be widely used for robotic applications (environment recognition, path planning, and obstacle avoidance), virtual reality, surveillance systems, and military applications.

References [1] J. Gluckman and S. K. Nayar, "Planar Catadioptric Stereo: Geometry and Calibration", Proc. Computer Vision and Pattern Recognition, 1999. [2] J. Gluckman and S. K. Nayar, "Rectified Catadioptric Stereo Sensors", Proc. Computer Vision and Pattern Recognition, 2000. [3] D. Lee and I. Kweon, "A Novel Stereo Camera System by a Biprism", IEEE Trans. Robotics & Automation, vol. 16, no. 5, pp. 528-541, Oct. 2000. [4] S. Baker and S. Nayar, "A Theory of Single-Viewpoint Catadioptric Image Formation", Int. Journal of Computer Vision, 35(2), pp. 175-196, 1999. [5] J. Gluckman, S. K. Nayar and K. J. Thoresz, "Real-Time Omnidirectional and Panoramic Stereo", Proc. Image Understanding Workshop, 1998. [6] S. A. Nene and S. Nayar, "Stereo with Mirrors", Proc. Int. Conference on Computer Vision, 1998. [7] S. Lin and R. Bajcsy, "High Resolution Catadioptric Omni-Directional Stereo Sensor for Robot Vision", Proc. Int. Conference on Robotics & Automation, 2003. [8] D. Southwell, A. Basu, M. Fiala and J. Reyda, "Panoramic Stereo", Proc. Int. Conference on Pattern Recognition, 1996. [9] M. Fiala and A. Basu, "Panoramic stereo reconstruction using non-SVP optics", Proc. Int. Conference on Pattern Recognition, 2002. [10] E. L. L. Cabral, J. C. de Souza Junior and M. C. Hunold, "Omnidirectional Stereo Vision with a Hyperbolic Double Lobed Mirror", Proc. Int. Conference on Pattern Recognition, 2004. [11] S. Nayar and V. Peri, "Folded Catadioptric Cameras", Proc. Computer Vision and Pattern Recognition, 1999. [12] T. L. Conroy and J. B. Moore, "Resolution Invariant Surfaces for Panoramic Vision Systems", Proc. Int. Conference on Computer Vision, 1999. [13] T. Lindeberg, "Scale-space theory: A Basic Tool for Analysing Structures at Different Scales", Journal of Applied Statistics, 20, 2, pp. 224-270, 1994. [14] P. Werth and S. Scherer, "A Novel Bidirectional Framework for Control and Refinement of Area Based Correlation Techniques", Proc. Int. Conference on Pattern Recognition, 2000.