CAMERA-BASED CALIBRATION OF MULTIPLE

LASER PROJECTORS FOR COLLABORATIVE PROJECTION

MENENDEZ Francisco J., HALABI Osama, FUJIMOTO Tadahiro, CHIBA Norishige

Department of Computer Science Graduate School of Engineering

Iwate University E-mail: fcomenendez@cg.cis.iwate-u.ac.jp, [ohalabi | fujimoto | nchiba] @cis.iwate-u.ac.jp

ABSTRACT

In this paper we propose a method to automatically

calibrate and combine multiple laser projectors for

collaborative projection.

One of the main caveats of today’s laser projectors is the

relationship between the speed in which the galvanometers

can move and the complexity of the data we are trying to

display. As image complexity grows, we are forced to

increase the scanning speed in order to avoid flickering.

However, the inertia effect of the rotating mirrors, which

manifests as curved deformations in sharp angles, becomes

more evident as we force the speeds up.

To improve the way these projectors behave, we propose

combining two or more projectors to display one single

image. In this way, the amount of lines that can be drawn

without flickering would increase, without forcing the

galvanometers to move at greater speeds or jumping

greater angles. Furthermore, to obtain an accurate

calibration, we use a digital camera to capture the drawing

area of each singular projector and produce the necessary

transformations to allow for a full overlap of these areas.

1. INTRODUCTION

Using laser light to construct shapes and images is hardly

new, however, it has not been until recently that laser

display technology has become relatively cheaper and

more available.

The growing availability of this technology opens a new

door for Computer Graphics researchers, as laser projectors

have their very own characteristics that separate them from

raster display and even other vector-based devices.

Some of the particular problems associated with laser

projection deal mainly with the limitations imposed by the

hardware itself. To understand these problems we must first

understand how a laser projector works.

A laser projection system is basically comprised of three

basic parts:

1) A computer or some other electronic source that

generates signals that control the projector.

2) A hardware interface that serves as a communication

bridge between the computer and the projector. Depending

on the technology, this interface may exist in the computer

as a connected board, inside the projector, as part of its

hardware; or as a third, separated component (e.g.: a

network-based interface).

3) The projector itself, that usually consists of one laser

beam (three beams in the case of RGB projectors), and a

couple of high-speed scanners (mirrors attached to their

corresponding galvanometers), that rotate according to the

signals sent by the computer, in order to make the laser

beam move and display the desired images.

Some of the problems derive from the physical nature of

the method used to move the beam. Limitations in rotation

speed lead to problems related with image complexity and

flickering. The weight of the mirrors and its inherent

rotational inertia causes deformations when drawing sharp

angles.

Some other problems are directly related with direct eye

exposure to the beam. Safety measure should be taken

regarding where to place the projector and selecting the

appropriate laser power for each occasion.

In this paper we aim to solve two of these problems. To

keep the beams away from the audience, we must

sometimes place the projector in a position that does not

guarantee a correct projection. With the help of a camera,

we will show how to overcome this problem. Later on, we

will extend this method to combine multiple projectors to

increase the refresh rate, and reduce flickering.

1.1 Previous Work

There is some previous work regarding projector

calibration [BRO05] [RAS98], studying the alignment of

various raster projectors, in order to increase the display

area. While our objective is to create a single, collaborative

projection area (instead of increasing the size), some basic

concepts are similar (such as the use of grids and cameras

to calibrate). Also, in our case, the correct alignment of the

projection areas is critical, since even the smallest

differences can become quite evident.

Furthermore, an attempt to reduce flickering was proposed

by Abderyim et al. [ABD07], by removing unnecessary

jumps (blanking lines) as much as possible. In this paper

we will not consider the blanking lines problem, but a

combination of both methods would theoretically allow

creating more complex imagery without the undesired

flickering effect that usually follows.

2. SINGLE PROJECTOR CALIBRATION

In a previous work [MEN07] we discussed the need for a

simple and efficient way to display images regardless of

the location of the projector. This was because of the

concerns that rose regarding the possibility of direct eye

exposure to the laser beams.

As we will explain later, we first need to calibrate each of

the involved projectors individually using the this method,

before we can proceed to the multiple-projector calibration.

This method uses a camera, placed where the audience

would be, and automatically captures the deformations

caused by the projection angle and the irregularities in

projection surface, processes the data and creates a

calibration grid used to interpolate the coordinates before

displaying.

2.1. Capture and image processing

To detect the deformations caused by the angle between the

projector and the projection surface, as well as its

irregularities, we project a pre-defined grid on the surface

and then calculate the difference between what we project

and what we obtain.

In order to recognize the projected grid from the surface,

we must first take a picture of the surface without anything

projected on it. Since we want this process to be fully

automatic, we take advantage of a common feature of most

late digital cameras called “PTP” (Picture Transfer

Protocol) [ISO05]. This protocol is what allows cameras to

directly send images to inkjet printers to produce prints

without the need of a computer. Surprisingly, this protocol

also allows for more than just data transfer, but to control

many of the camera functions, including taking snapshots.

Once we have placed the camera where our audience

would be, it should not be moved until the calibration

process is complete. Using PTP, the computer

communicates with the camera and commands it to take

the necessary images at the appropriate times, thus

eliminating any possible errors caused by pressing the

shutter button manually.

After we have taken the first picture of our projection

screen, we proceed to take a second picture with a dot grid

pattern projected. For calibration purposes, we need this

grid to be of an odd number of dots (e.g.: 3x3, 5x5, 7x7,

etc). The size of the grid will be determined by the

maximum and minimum laser coordinates, and the distance

between the projection surface and the projector. An

example of this grid can be found in figure 1.

Figure 1: (a) Ideal grid and (b) resulting projection

The next step is to detect the grid from the images. To do

so, we calculate the difference between the first and second

images, using equation (1), where diffimg(x,y) will store

the color difference of the pixels from imgbase(x,y), base

image and, imggrid(x,y) projected grid, at the x-y

coordinate, thus removing the background and leaving

only the dots from the projected grid.

( , ) max( ( , ) ( , ), 0)diffimg x y imgbase x y imggrid x y= − (1)

This difference image might have some noise product of

changes in luminosity or undesired beam reflections. To

eliminate these noises, we binarize the difference image,

changing the threshold according to the number of dots

found, versus the number of dots expected. In other words,

if we find more dots than we expect, it is reasonable to

suppose those extra dots are caused by noise and then we

raise the threshold to filter that noise. This is shown in

equation (2), where diffbin(x,y) will hold the binary pixels

and we use the threshold θ to filter noise. A result of this

binarization is shown in figure 2

1 if ( , )( , )

0 otherwise

diffimg x ydiffbin x y

θ>

(2)

Figure 2: Binary difference image. In this step we have

isolated the grid from the background

Next, we estimate the coordinates, in pixels, of the detected

grid points from the binarized image. We accomplish this

by choosing the center of each block of pixels that

represent one grid dot and store it in memory for further

processing.

(a) (b)

2.2 Grid detection

Now we have a set of uncorrelated x-y coordinates, in pixel

units, representing the dots. However, before we can use

this data, we must restore the detected dots into their

original grid arrangement.

To accomplish this, we first find the corner points, that is,

the four points that have the highest and lowest x-y values.

Taking these four points, we take a first guess at the

arrangement, by placing a grid which corners match

exactly these four points. The rest of the detected points are

matched to the remaining grid points by using a weight

function, shown in (3), where pt holds the x-y coordinate in

pixels of the point we want to match, gridx,y holds the x-y

coordinate in pixels of the possible grid point we are

evaluating; always preferring that grid point that returns

the highest weight value.

, ,

,

, 1, 1,

, 1 , 1

,

if has been matched( , )

0 otherwise

( , ) ( , ) ( , )

( , ) ( , )

2

x y x y

x y

x y x y x y

x y x y

x y

grid pt gridwMatch pt grid

w pt grid wMatch pt grid wMatch pt grid

wMatch pt grid wMatch pt grid

grid pt

+ −

+ −

−=

= +

+ +

+ ⋅ −

(3)

In plain words, this means that for each grid point we

evaluate, we will not only prefer the closest one point, but

also consider those grid points that have already been

matched. In this way, if two points are equidistant from the

grid point being considered, the grid point will prefer that

one that has the closest matched neighbors.

We repeat this process until all grid points have been

assigned a point detected from the binary image. Now this

grid holds all the detected points in the correct arrangement

(see figure 3) However, since the data was taken from an

image, these points are expressed in pixel units instead of

the original laser projector units. In order to obtain a grid

we can actually use, we need to transform these pixel

coordinates into laser coordinates.

Figure 3: The observed points arranged in a grid fashion

The first step is to correlate the middle point of the grid

with the center of the laser projection zone. Namely, in the

detected grid, the center point will be a given pair of x-y

numbers in pixels. However, since the center of projection

for a laser projector is always (0, 0), we must translate the

grid so the center points are aligned, as shown in equation

(4), where optimalcenterx,centery is the center, in laser

coordinates of the ideal grid we first projected (usually 0,0),

and the observedcenterx,centery is the center of the observed

grid, in pixel units.

, ,centerx centery centerx centerytranslate optimal observed= − (4)

The correlation between laser units and pixels

(pixel-to-laser scale) is estimated by an iterating binary

search algorithm. We try to minimize the equation shown

in (5), where optimal holds the ideal grid in laser units and

observed is the visually obtained grid in pixel units.

Applying different scale values, we approximate the pixel

grid to be as close as possible to the original maximum and

minimum values of the projected gird, until we get to the

maximum level of iterations or an acceptable error value is

reached. The error is estimated as the absolute distance

between the corners of both grids (originally projected and

visually obtained).

0 ,0 0 ,0

0 , 0 ,

,0 ,0

, ,

( ) ( )

( )

( )

(

size size

size size

size size size size

distance scale optimal scale observed translate

optimal scale observed translate

optimal scale observed translate

optimal scale observed transla

= − ⋅ +

+ − ⋅ +

+ − ⋅ +

+ − ⋅ + )te

(5)

After applying the translation and scaling to the observed

grid, we obtain the final calibration grid as the difference

between the ideal grid and observed deformation, as shown

in equation (6). We can consider this final, scaled grid as

the visual deformation caused by the projection angle and

surface irregularities, expressed in laser units. However,

this is not the correction grid we need. This “calibration”

grid can be thought of an inverted grid that counters the

aforementioned deformations, and can be obtained as the

difference between the visually obtained deformation grid

and the original projected (not-deformed) grid.

( ), , ,2x y x y x y

calibrated optimal observed translate scale= ⋅ − + ⋅ (6)

2.3. Calibration

Now that we have a calibration grid we can correct the

coordinates of our image and overcome the observed

deformations.

The calibration process starts by taking each vector in the

original image and finding which cells it intersects from

the initial, unchanged gird. The vector is then broken down

into smaller parts, each part fully contained in its cell. Each

part is then transformed, using bilinear interpolation, so it

matches the deformations in the calibration grid. These

new set of vectors is what we finally display with the laser

projector.

An example of a single-projector calibration can be seen in

figure 4.

Figure 4: (a) Test pattern prior to calibration, (b)

compensation due to calibration and (c) final, corrected

image.

3. MULTIPLE PROJECTORS CALIBRATION

Since our goal is to combine the projection areas for each

single projector to work as a single unit, initially we

manually aim the projectors to be as overlapped as possible,

regardless of the projection deformations that may arouse

from this. An example of this setting can be seen in figure

5

Figure 5: Calibration grids for both projectors

The process for calibrating multiple projectors can be

considered an extension of the calibration for a single

projector. First, for each projector involved we perform the

single calibration method as previously explained. The next

step is a matter of fine-aligning the projection areas and

resizing the areas accordingly.

To align the projection areas we take the middle point of

the grids in pixels, as we observed them in (4), and find the

average middle point. Since this middle point is in pixels,

we multiply by the pixel-to-laser scale value, for each

projector and move the center of the projection area for

each projector to the common averaged center by applying

a pre-translation to the entire grid, as shown in (7), where

translatei is the translate value obtained in (4), for the

projector i, and scalei is the scale value obtained from

minimizing (5).

i

i i

translatecenter

nProjectors

preTranslate center scale

=

= ⋅

∑ (7)

The scaling problem is solved in a similar way. We find the

average pixel-to-laser scale value of each projector and

then apply a pre-scale to the calibration grid prior to make

the necessary corrections.

To perform the final calibration, we simply use the new

centered, resized grid, instead of the one obtained with the

single-calibration method.

4. RESULTS

For our experiments we used two ILDA compliant laser

projectors. Two interface boards were used. A Pangolin

QM2000 internal PCI interface and a QM2000.NET,

network based interface. In order to perform the

communication, we used the SDK provided by the

manufacturer.

While, for best results, we should place both projectors to

have their projection zones as overlapped as possible, for

our experiment we separated the projectors on purpose.

Figure 6 shows each projection grid for both projectors,

prior calibration. We can see that both size and perspective

differ from each other

Once we have calibrated each projector individually, we

performed the necessary calculations to overlap the two

regions.

An example of the dual-calibration in action can be seen on

figure 7. In this example, we took our university logo,

transformed it into vectors and split it into two parts with

similar amount of vectors. In the first image, we see the

two parts being projected prior to the calibration takes

place. We can see that there is an evident gap between the

two parts. After calibration, however, we can see that the

original image is restored and correctly displayed

To test the effectiveness of our method, we created frames

Projector 1 Projector 2

Audience (digital camera)

(a) (b)

(c)

with increasing complexity and displayed them with both

methods (single and dual projection), and changed the scan

rate until a noticeable flickering effect appeared. To

measure the flickering effect, we set up a camera to take

snapshots with a shutter speed of 1/30th

of a second. If the

picture taken showed the entire drawing as it was intended,

the scan rate was adequate and we can assume that no

flicker appears. However, if the picture was incomplete,

this meant that the projector failed to refresh at an

appropriate speed. The results are shown in figure 8

Figure 6: Calibration grids for both projectors

We can see from this result that using two projectors

instead of one, allows us to draw more points per frame

using almost exactly half the scan-rate. In other words, by

using the same regular scan-rate than a single projector, we

can achieve images that are twice as complex.

5. CONCLUSIONS

We have shown a method that automatically corrects the

deformations caused by projection angles and surface

irregularities for single projector. In this way, we can safely

project onto a surface without risking eye injuries on the

viewers.

We also extended this method by combining two projectors

to fully overlap, creating a single projection area. This new

area can help reduce the flickering effect for complex

images with large numbers of vectors, while keeping the

same scan-rate.

Figure 7: (a) University logo before and (b) after

calibration

0500010000150002000025000300003500040000200 300 400 500 600PointsScan Rate Single ProjectorDual Projector

Figure 8: Points per frame versus required scan rate to

avoid flickering

(a)

(b)

6. FUTURE WORK

While we have shown a consistent method for combining

multiple laser projectors for collaborative projection, the

problem of obtaining an adequate break-down of the image

and assigning the responsibilities to each projector

according to their capabilities remains to be done.

This would be helpful, for example, in a heterogeneous

environment, where faster projectors can compensate for

slower ones.

Another interesting point of study is to extend this method

to handle not only still frames, but animations as well. In

this case, the speed of the projectors must be taken into

account to avoid the slower projector lagging behind.

Issues such as synchronization and frame-rate will become

areas of research.

ACKNOWLEDGEMENTS

This work was partially supported by the Japanese

Ministry of Education, Science, Sports and Culture,

Grant-in-Aid for Exploratory Research 19650014.

REFERENCES

[ABD07] An Efficient Drawing Algorithm for Laser

Graphics, Purkhat ABDERYIM, Osama HALABI and

Norishige CHIBA, p9, Tohoku-Section Joint Convention of

Institutes of Electrical and Information Engineers 2007

[BRO05] Camera-Based Calibration Techniques for

Seamless Multi-Projector Displays. BROWN Michael,

MAJUMDER Aditi, YANG Ruigang, IEEE Transactions

on Visualization and Computer Graphics, March/April

2005 (Vol. 11, No. 2) pp. 193-206

[ISO05] Picture transfer protocol (PTP) for digital still

photography devices. ISO Standard 15740:2005

[MEN07] Camera-Based Calibration for Laser Projectors,

Francisco MENENDEZ, Osama HALABI and Norishige

CHIBA, p.3, Tohoku-Section Joint Convention of Institutes

of Electrical and Information Engineers 2007

[RAS98] Seamless Projection Overlaps Using Image

Warping and Intensity Blending, RASKAR R., WELCH G.,

and FUCHS H, Fourth International Conference on Virtual

Systems and Multimedia, (Gifu, Japan), Nov. 1998