A Framework of View-Dependent Planar Scene Active Camouflage

Huei-Yung Lin, Wen-Nung Lie, and Min-Liang WangDepartment of Electrical Engineering

National Chung Cheng University168 University Rd., Min-Hsiung, Chia-Yi 621, Taiwan{lin, wnlie}@ee.ccu.edu.tw, poolllz.wang@msa.hinet.net

Abstract

Active camouflage is a technique for occluding ob-jects disappear from the observer. Current implemen-tations make assumptions on the viewpoints of both theobserver and the camouflaged target. In this work, wepresent a framework of view-dependent planar scenecamouflage. The cameras representing the observerand the occluding object are placed in general positions.The images captured by the object camera are trans-formed to the observer viewpoint to generate globallyconsistent visual data for transparent camouflage. Ex-perimental results are presented for real scene imagesusing a projector-camera system.

1. Introduction

Camouflage originally is a common scheme for ani-mals to help avoid detection by predators or prey. Ex-amples include the special apparent patterns of floun-der, frogs, and some insects. This concept is lateradopted by modern military tactics to disguise the en-emies. The objective of camouflage is usually to hideobjects by making them look like the natural back-ground to the observer. Thus, some synthetic patternsfor the object surface have to be generated based onthe surrounding environment. In addition to blendingin with the surroundings, there are also other types ofimplementations for visual camouflage. For example,one can make the object disappear in front of the ob-server by creating the illusion of its transparency. Thisapproach has recently attracted much attention sinceit can be applied to multimedia, virtual reality, andaugmented reality applications [5, 6, 8].

In the past few decades, related work can be clas-sified into three categories: camouflage assessment,strategies for active motion camouflage, and camou-flage breaking. Camouflage assessment is to evaluate

the quality of camouflaged targets, in terms of humanor machine perception [10]. Active motion camouflageattempts to find an optimal path or trajectory for adynamic observer or target [1, 12]. The purpose ofcamouflage breaking, on the other hand, is to detectthe camouflaged targets in the visible spectrum [13, 9].Although a great number of algorithms have been pro-posed for the trajectory planning of motion camouflage,and the assessment and breaking of existing camouflageapproaches, the progress in camouflage techniques isstill fairly limited. Most existing methods simply usemanually or randomly generated surrounding patternas the object’s camouflage texture.

Recently, a new methodology of camouflage tech-nologies named active camouflage or adaptive camou-flage is proposed by many researchers. It is aimed toblend the object into its surroundings by use of panelsor coatings capable of altering their appearance, color,luminance and reflective properties. Thus, active cam-ouflage has the potential to achieve perfect conceal-ment from visual detection. Based on the similar con-cept, Tachi et al. developed an optical camouflage sys-tem using retro-reflective projection technology (RPT)[7]. They put a video camera behind the object tocapture the invisible region, and then project the oc-cluded background onto the object’s surface. Specialcloak with retro-reflective material is used as projectionsurface to ensure the light only reflects in the directionof projection. However, nothing related to the cam-era and projector positions and scene geometry wasaddressed in their work.

It is well-known in 3-D vision that the occluded re-gion depends on both the occluding object and theobserver’s viewpoint. To achieve visually consistentforeground and background alignment, the camouflagepattern should be generated based on the position ofthe observer. In this paper, we propose a frameworkof view-dependent active camouflage. Two camerasare used to illustrate this general case: one camera

Canadian Conference on Computer and Robot Vision

978-0-7695-3153-3/08 $25.00 © 2008 IEEEDOI 10.1109/CRV.2008.12

319

is placed behind the occluding object to capture thebackground scene (called the back camera), and theother camera is served as the observer (called the frontcamera). The intrinsic parameters and relative posi-tion and orientation between these two cameras areobtained from camera calibration. The information isthen used to generate consistent camouflage pattern forthe observer’s viewpoint. Consequently, visually con-sistent camouflage from different viewing directions canbe achieved.

2. Image Transformation between TwoCameras

The relationship between a 3-D point and the corre-sponding 2-D image point for the commonly used pin-hole camera model can be written as

x = PX (1)

where X and x are the 3-D and image points repre-sented by homogeneous 4-vector and 3-vector, respec-tively. The 3 × 4 homogeneous matrix P, which isunique up to a scale factor, is called the perspectiveprojection matrix of the camera. It can be further de-composed into the intrinsic camera parameter matrixand the relative pose of the camera:

P = K[R t] (2)

The 3 × 3 matrix R and 3× 1 vector t are the relativeorientation and translation with respect to the worldcoordinate system, respectively. The intrinsic parame-ter matrix K of the camera is a 3×3 matrix and usuallymodeled as

K =

fx γ u0

0 fy v0

0 0 1

(3)

where (u0, v0) is the principal point of the camera, γis a skew parameter related to the characteristic of theCCD array, and fx and fy are scale factors in the xand y directions [14].

If we consider two cameras in a general position, theprojections of a given 3-D scene point onto the two im-age planes have to be calculated with both the cameraparameters and the pixel correspondences. Thus, itis not possible to establish the relationship of the im-age coordinates between these two images if the 3-Dstructure of the scene is not known a priori [4]. Con-sequently, a unique transformation between the imagepair may not always exist for any arbitrary scene. Fromprojective geometry, however, image points of one cam-era are related to corresponding image points of the

other camera by a homography if the recorded scene isa planar surface. Moreover, the homography inducedby the planar scene can be written as

x2 = Hx1 (4)

where x1 and x2 are the corresponding points of theimage pair represented by homogeneous 3-vector, andH is a 3 × 3 matrix with an arbitrary scale factor.

Suppose that P1 and P2 are the projection matricesof the first and second camera, respectively, then wehave

x1 = P1X and x2 = P2X (5)

for any given 3-D scene point X. Now, if both the cam-eras are fully calibrated, and the intrinsic parametermatrices are represented by K1 and K2. The projectionmatrices can then be written as

P1 = K1[I | 0] and P2 = K2[R | t] (6)

if the first camera coordinate is set as the world coor-dinate, and the translation and rotation of the secondcamera are given by t and R, respectively. For the im-age point x1 of the first camera, it is easy to verifythat

K [I | 0][K−1x1

λ

]= x1 (7)

where λ is a scale factor. Thus, any 3-D point on theray

X =[K−1x1

λ

](8)

projects to the image point x1.For the homography induced by the plane nT X+d =

0, any 3-D point X on the plane is given by

X =[

x1

− 1dn

T x1

](9)

since [nd

]T [x1

1dn

T x1

]= 0 (10)

From Eqs. (5) and (6),

x2 = P2X = K2(R− 1dtnT )K−1

1 x1 (11)

Thus, the homography for Eq. (4) is given by

H = K2(R− 1dtnT )K−1

1 (12)

From Eqs. (4) and (12), the pixel correspondences be-tween the two views can be obtained provided that theintrinsic parameter matrices K1 and K2, the relative po-sition between the cameras, and the planar scene equa-tion are known.

320

X

x1 = P1X x2 = P2X

C1 C2x2 = Hx1

Figure 1. Plane induced homography.

3. Camouflage System and Implementa-tion

The objective of transparent camouflage for a givenviewpoint is to observe the background scene withoutseeing the occluding objects in between. Based on theprinciple of active camouflage, a general configurationof the proposed optical camouflage system is illustratedin Figure 2. It consists of two cameras, one occludingobject, a planar background scene, and a data projec-tor. The camera Cf , called the front camera, is placedfurther away from the background scene and served asthe observer. The other camera Cb, called the backcamera, is placed behind the occluding object to cap-ture the occluded background scene (with respect tothe observer’s viewpoint). Our goal is to generate glob-ally consistent visual information for the front camerausing the image captured by the back camera, and havethem displayed on the surface of the occluding objectusing the data projector.

It should be noted that the field of view of the backcamera should be large enough to cover the occludedscene with respect to the front camera. This can usu-ally be achieved by using a fish-eye lens or the lens withsmall focal length. Although the shape of the occlud-ing object can be arbitrary in principle, it is assumedto be planar for simplicity. The point correspondencesbetween different apparent surfaces can always be ob-tained since there is a one-to-one mapping along theline of sight of the front camera. Under this assump-tion, the derivation of camouflage pattern for the oc-cluding object involves two transformations:

(i) Transform the back camera image to the front

camera image plane.

(ii) Transform the resulting image to the occluding ob-ject plane.

In this work, we first demonstrate that the occludingregion is assigned and synthesized on the front cameraimage. In this case, only the first transformation hasto be carried out. The synthesized front camera imageis then prewarped and displayed on the object plane toillustrate the view-dependent active camouflage.

For the scene transformation of the back and frontcameras, the image can be generated using homogra-phy given by Eq. (4), since the background is assumedto be planar. Suppose the cameras Cb and Cf in Fig-ure 2 correspond to the cameras C1 and C2 in Figure 1,respectively. The homography can be computed usingEq. (12) with intrinsic parameter matrices, relativeposition and orientation of the front and back cam-eras, and the background scene equation. Thus, thetransformation requires the calibration of both cam-eras, computation of the background scene equation,and image synthesis for the front camera.

To derive the intrinsic and extrinsic parameter ma-trices (i.e., Kb, Kf , and [Rb | tb], [Rf | tf ]), a 3-D calibra-tion object is placed in front of both cameras for cali-bration. The relative pose between the front and backcameras are then computed from their relative posewith respect to the calibration object. More specifi-cally, the rotation matrix R and translation vector tare given by

R = RfR−1b and t = −RfR

−1b tb + tf (13)

if the back camera coordinate is set as the world coor-dinate.

321

Projector

Cb

Cf

O

R, t

nT X + d = 0

Figure 2. Schematic diagram of the proposed optical camouflage system.

For the derivation of the planar scene equation, acheckerboard pattern with known 2-D metric informa-tion is placed on the background scene. Perspectiveprojection and coplanar constraint of the 3-D scenepoints are used to determine the best fitting plane rep-resented by nT X + d = 0. Suppose the 3-D pointsare denoted by (λxi, λyi, λf), for i = 1, 2, ..., N , where(xi, yi) is the image point, f is the focal length of thecamera and λ is a scale factor. Then the parameters nand d can be solved by the overdetermined system

λAn + d = 0 (14)

using SVD (Singular Value Decomposition), where

A =

X1 Y1 Z1

X2 Y2 Z2

......

...XN YN ZN

and

d =

dd...d

are N × 3 matrix and N × 1 vector, respectively.Image synthesis for the front camera is to generate a

globally consistent visual data for the occluded region.

It involves image transformation using the homography

H = Kf (R− 1dtnT )K−1

b (15)

and bilinear interpolation for pixel resampling. Exceptfor different viewable areas, the synthesized image andthe original front camera image should be perfectly reg-istered for the planar background region. Thus, it canbe used to demonstrate the principle of optical cam-ouflage conceptually, and verify the correctness of thefirst transformation. If we consider a special case thatthe background scene is far away from the cameras,for example, the occluding object is in front of a dis-tant landscape. Then the 3-D structure of the scenebecomes insignificant, and Eq. (15) can be rewrittenas

H = KfRK−1b (16)

since d goes to infinity. In this case, the homographycan be computed using only the camera parameter ma-trices and the relative orientation, without the trans-lation information between the cameras.

Given the resulting synthesized front camera imagefrom planar homography, the second transformation isto generate an equivalent image for the occluding re-gion and display it on the object’s surface. This in-volves a projective transformation from the front cam-era image plane to the object’s surface. If the locationof the occluding object is known, then four point corre-spondences can be used to determine the transforma-

322

Figure 3. Experimental setup.

tion for image rectification. The rectified image can beeither directly displayed on the planar object (using anLCD display, for instance) or projected on the objectsurface using a data projector. In the implementation,the latter approach is adopted since it is more flexiblefor the experimental setup. In this case, an additionalplanar homography similar to Eq. (16) is applied. Thisis due to the position and pose differences between theprojector and the front camera.

4. Experimental Results

The proposed technique for active camouflage hasbeen tested on several real scenes. Figure 3 shows theexperimental setup of the optical camouflage system.Two video cameras (SONY DFW-X710) are placed infront of a planar scene, with one closer to the scene(back camera) and the other further away from thescene (front camera). The intrinsic parameters and rel-ative pose of the cameras are derived from Tsai’s cal-ibration method [15], and the background scene equa-tion is obtained as described in the previous sectionusing a planar calibration pattern.

For the first experiment, a motion picture is pro-jected on the planar background scene using a dataprojector. The image frames captured by the frontand back cameras are shown in Figures 4(a) and 4(b),respectively. It is assumed that there is an occludingobject behind the back camera so that the image regioninside the red bounding box of Figure 4(a) is not avail-able to the front camera. Figure 4(c) illustrates theactive camouflage by displaying the transformed backcamera image on the occluded region (blue boundingbox). Misalignment on the boundary is primary due tothe physical difference between the planar calibrationpattern and the projection wall.

(a) Front camera image.

(b) Back camera image.

(c) Observer image with camouflaged occluded region.

Figure 4. Experimental results of a projecteddynamic scene.

323

(a) Front camera image.

(b) Back camera image.

(c) Observer image.

Figure 5. Experimental results with an objectin motion.

(a) Back camera image.

(b) Prewarped image for projection.

(c) Active camouflage result.

Figure 6. Experimental results with projectedcamouflage pattern.

324

For the second experiment, a person is walking infront of the background scene. The image frames cap-tured by the front and back cameras are shown in Fig-ures 5(a) and 5(b), respectively. The image of trans-parent camouflage is illustrated in Figure 5(c). Thedistortion is more severe for the object region furtheraway from the background scene since planar homog-raphy cannot be applied correctly.

For the last experiment, a planar object placed be-hind the back camera is used to partially block thefront camera’s view. The occluded background scenecaptured by the back camera is projected onto the ob-ject’s surface using a data projector. The back cameraimage, prewarped image for projection, and the trans-parent camouflage image taken from the front cameraare shown in Figures 6(a), 6(b), and 6(c), respectively.It is seen that the projected image is blurry and thecolor is not consistent with the background scene. Thismight be due to the material property of the objectsurface. One way to correct the distortion is to employradiometric calibration or photometric compensation[16, 2, 3]. Furthermore, if the geometry of the projec-tion surface is non-planar but known, it is also possibleto project the image in an undistorted way [11].

5. Conclusions

The objective of active camouflage is to make theoccluding object disappear from the observer. It hasmany applications on multimedia, virtual reality, aug-mented reality, etc. Current camouflage technologiesassume that the viewing directions of the observer andthe camouflaged object are nearly identical. In this pa-per, we have proposed a framework for view-dependentplanar scene active camouflage. Two video camerasplaced in general positions are used to represent the oc-cluding object and the observer. Experimental resultsof real scene images have demonstrated the feasibilityof the proposed method.

Acknowledgments.

The support of this research in part by the NationalScience Council of Taiwan, R.O.C. under Grant NSC-95-2221-E-194-075, and Chungshan Institute of Scienceand Technology, R.O.C. under contract XD95175P isgratefully acknowledged.

References

[1] A. Anderson and P. McOwan. Model of a predatorystealth behaviour camouflaging motion. Proceedingsof the Royal Society B, 270:489–495, 2003.

[2] M. Ashdown, T. Okabe, I. Sato, and Y. Sato. Robustcontent-dependent photometric projector compensa-tion. In CVPRW ’06: Proceedings of the 2006 Con-ference on Computer Vision and Pattern RecognitionWorkshop, page 6, 2006.

[3] K. Fujii, M. Grossberg, and S. Nayar. A Projector-Camera System with Real-Time Photometric Adap-tation for Dynamic Environments. In IEEE Con-ference on Computer Vision and Pattern Recognition(CVPR), volume 1, pages 814–821, Jun 2005.

[4] R. I. Hartley and A. Zisserman. Multiple View Geome-try in Computer Vision. Cambridge University Press,ISBN: 0521540518, second edition, 2004.

[5] M. Inami, N. Kawakami, D. Sekiguchi, Y. Yanagida,T. Maeda, and S. Tachi. Visuo-haptic display usinghead-mounted projector. In VR, pages 233–, 2000.

[6] M. Inami, N. Kawakami, and S. Tachi. Optical cam-ouflage using retro-reflective projection technology.In ISMAR, pages 348–349. IEEE Computer Society,2003.

[7] M. Inami, N. Kawakami, and S. Tachi. Optical cam-ouflage using retro-reflective projection technology. InISMAR ’03: Proceedings of the The 2nd IEEE andACM International Symposium on Mixed and Aug-mented Reality, page 348, 2003.

[8] D. Iwai, S. Hanatani, C. Horii, and K. Sato. Limpiddesk: Transparentizing documents on real desk inprojection-based mixed reality. In VR ’06: Proceedingsof the IEEE Virtual Reality Conference (VR 2006),page 65, 2006.

[9] T. E. B. R. Micheals, X. Gao, P. Lewis, C. Power,W. Yin, and A. Erkan. Frame-rate omnidirectionalsurveillance & tracking of camouflaged and occludedtargets. In VS ’99: Proceedings of the Second IEEEWorkshop on Visual Surveillance, page 48, 1999.

[10] S. Nyberg and L. Bohman. Assessing camouflagemethods using textural features. Optical Engineering,40(9):1869–1876, Sep. 2001.

[11] R. Raskar, M. S. Brown, R. Yang, W.-C. Chen,G. Welch, H. Towles, B. Seales, and H. Fuchs. Multi-projector displays using camera-based registration. InProceedings of the 10th IEEE Visualization 1999 Con-ference (VIS ’99), 1999.

[12] P. V. Reddy, E. W. Justh, and P. S. Krishnaprasad.Motion camouflage in three dimensions. ArXiv Math-ematics e-prints, Mar. 2006.

[13] A. Tankus and Y. Yeshurun. Convexity-based visualcamouflage breaking. Computer Vision and Image Un-derstanding, 82(3):208–237, 2001.

[14] E. Trucco and A. Verri. Introductory Techniques for3-D Computer Vision. Prentice Hall, 1998.

[15] R. Tsai. A versatile camera calibration technique forhigh-accuracy 3d machine vision metrology using off-the-shelf tv cameras and lenses. IEEE Trans. Roboticsand Automation, 3(4):323–344, 1987.

[16] G. Wetzstein and O. Bimber. Radiometric compensa-tion through inverse light transport. In PG ’07: Pro-ceedings of the 15th Pacific Conference on ComputerGraphics and Applications, pages 391–399, 2007.

325