5
IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 56,NO. 1, FEBRUARY 2007 199 CCD Color Camera Characterization for Image Measurements Paul J. Withagen, Frans C. A. Groen, Fellow, IEEE, and Klamer Schutte Abstract—In this article, we will analyze a range of different types of cameras for its use in measurements. We verify a general model of a charged coupled device camera using experiments. This model includes gain and offset, additive and multiplicative noise, and gamma correction. It is shown that for several cameras, ex- cept a typical consumer webcam, the general model holds. The associated model parameters are estimated. It is shown that for most cameras the model can be simplified under normal oper- ating conditions by neglecting the dark current. We further show that the amount of additive noise is exceeded by the amount of mul- tiplicative noise at intensity values larger than 10%–30% of the in- tensity range. Index Terms—Charge-coupled device (CCD), charge-coupled image sensors, digital imaging sensors, gain measurement, image processing, noise measurement, video cameras. I. INTRODUCTION C HARGE-COUPLED device (CCD) cameras are widely used for computer vision and image processing applica- tions. For many applications it is important to have a model of the imaging process and temporal image noise. For example, a model is used for the correction of temporal changes in inten- sity [1], illumination-invariant optical flow computation [2], or object detection [3], [4]. Often a model is used that fits the applications needs. How- ever, little work is reported on the accuracy of these models for a specific camera, and the importance of the different compo- nents of the model. In this article we evaluate a general camera model for different camera types, ranging from a high end 10-bit digital camera to a consumer webcam. The cameras used have been selected based on their availability. To model all possible effects of the CCD, we use the gen- eral model of a CCD camera introduced in [5]. In addition to that model, most modern cameras use some kind of gamma ad- justment to map the image in the available quantization range for obtaining a better looking image. Therefore, we add gamma correction to the model of Healey. We describe experiments to evaluate this model. The experiments presented verify the model and determine the model parameters for all cameras. The ex- Manuscript received June 15, 2005; revised July 24, 2006. An earlier version of this work was presented at IMTC 2005. P. J. Withagen is with Philips Medical Systems, 5680 DA Best, The Nether- lands (e-mail: [email protected]). F. C. A. Groen is with the Informatics Institute, University of Amsterdam, Amsterdam 1098 SJ, The Netherlands (e-mail: [email protected]). K. Schutte is with the Electro-Optics Group, TNO Defence, Security and Safety, 2509 JG The Hague, The Netherlands (e-mail: [email protected]). Color versions of one or more figures are available online at http://ieeexplore. ieee.org. Digital Object Identifier 10.1109/TIM.2006.887667 periments are repeated for different scene contents to verify the model at different settings of camera gain and shutter time. Besides verification of the model we show the effect of the in- dividual contributions in the model to allow for simplifications. In particular, we evaluate: whether the gamma function in the camera is sufficiently accurately described by our model; what the effect of the dark current is, what the noise distri- bution is. In the remainder of this article, first, a (theoretical) model of a CCD camera is discussed in Section II. In Section III we de- scribe the measurement setup and give the results. These results lead to simplifications of the model given in Section IV. Finally, conclusions are presented in Section V. II. THEORETICAL MODEL OF A CCD CAMERA Healey [5] describes the following model for a single pixel recorded at time using a CCD camera (1) with the camera gain, the true scene intensity, and the measured image intensity for a given color band. The following noise contributions are present: 1 the dark current is an offset, constant over time. The shot noise has a Poisson distribution with and depending on . The readout noise has a Gaussian distribution: , constant. The quantization noise has a uniform distribution with the smallest step in pixel value. There are three ways to control the global image intensity; we will use the term apparent gain for their joint effect. It can be controlled using the camera gain, camera shutter time or lens iris. All can be fixed (manual control) or automatically adapted to the scene (automatic gain/shutter/iris control). For the CCD, iris or shutter control causes a lower image intensity in (1). We model this by parameter with a value in the range [0,1] (2) Additionally, most cameras apply a gamma adjustment to map the range of intensity values from the CCD to the avail- able output range. Assuming it is implemented in the camera electronics just before digitization, (2) changes to (3) with the gamma value which is assumed to be time constant and equal for all pixels. 1 We use for the mean of and for its standard deviation. 0018-9456/$25.00 © 2007 IEEE

CCD Color Camera Characterization for Image Measurements

  • Upload
    k

  • View
    212

  • Download
    0

Embed Size (px)

Citation preview

IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 56, NO. 1, FEBRUARY 2007 199

CCD Color Camera Characterizationfor Image Measurements

Paul J. Withagen, Frans C. A. Groen, Fellow, IEEE, and Klamer Schutte

Abstract—In this article, we will analyze a range of differenttypes of cameras for its use in measurements. We verify a generalmodel of a charged coupled device camera using experiments. Thismodel includes gain and offset, additive and multiplicative noise,and gamma correction. It is shown that for several cameras, ex-cept a typical consumer webcam, the general model holds.

The associated model parameters are estimated. It is shown thatfor most cameras the model can be simplified under normal oper-ating conditions by neglecting the dark current. We further showthat the amount of additive noise is exceeded by the amount of mul-tiplicative noise at intensity values larger than 10%–30% of the in-tensity range.

Index Terms—Charge-coupled device (CCD), charge-coupledimage sensors, digital imaging sensors, gain measurement, imageprocessing, noise measurement, video cameras.

I. INTRODUCTION

CHARGE-COUPLED device (CCD) cameras are widelyused for computer vision and image processing applica-

tions. For many applications it is important to have a model ofthe imaging process and temporal image noise. For example, amodel is used for the correction of temporal changes in inten-sity [1], illumination-invariant optical flow computation [2], orobject detection [3], [4].

Often a model is used that fits the applications needs. How-ever, little work is reported on the accuracy of these models fora specific camera, and the importance of the different compo-nents of the model. In this article we evaluate a general cameramodel for different camera types, ranging from a high end 10-bitdigital camera to a consumer webcam. The cameras used havebeen selected based on their availability.

To model all possible effects of the CCD, we use the gen-eral model of a CCD camera introduced in [5]. In addition tothat model, most modern cameras use some kind of gamma ad-justment to map the image in the available quantization rangefor obtaining a better looking image. Therefore, we add gammacorrection to the model of Healey. We describe experiments toevaluate this model. The experiments presented verify the modeland determine the model parameters for all cameras. The ex-

Manuscript received June 15, 2005; revised July 24, 2006. An earlier versionof this work was presented at IMTC 2005.

P. J. Withagen is with Philips Medical Systems, 5680 DA Best, The Nether-lands (e-mail: [email protected]).

F. C. A. Groen is with the Informatics Institute, University of Amsterdam,Amsterdam 1098 SJ, The Netherlands (e-mail: [email protected]).

K. Schutte is with the Electro-Optics Group, TNO Defence, Security andSafety, 2509 JG The Hague, The Netherlands (e-mail: [email protected]).

Color versions of one or more figures are available online at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TIM.2006.887667

periments are repeated for different scene contents to verify themodel at different settings of camera gain and shutter time.

Besides verification of the model we show the effect of the in-dividual contributions in the model to allow for simplifications.In particular, we evaluate:

• whether the gamma function in the camera is sufficientlyaccurately described by our model;

• what the effect of the dark current is, what the noise distri-bution is.

In the remainder of this article, first, a (theoretical) model of aCCD camera is discussed in Section II. In Section III we de-scribe the measurement setup and give the results. These resultslead to simplifications of the model given in Section IV. Finally,conclusions are presented in Section V.

II. THEORETICAL MODEL OF A CCD CAMERA

Healey [5] describes the following model for a single pixelrecorded at time using a CCD camera

(1)

with the camera gain, the true scene intensity, andthe measured image intensity for a given color band. Thefollowing noise contributions are present:1 the dark current

is an offset, constant over time. The shot noise hasa Poisson distribution with and depending on .The readout noise has a Gaussian distribution: ,constant. The quantization noise has a uniform distribution

with the smallest step in pixel value.There are three ways to control the global image intensity;

we will use the term apparent gain for their joint effect. It canbe controlled using the camera gain, camera shutter time or lensiris. All can be fixed (manual control) or automatically adaptedto the scene (automatic gain/shutter/iris control). For the CCD,iris or shutter control causes a lower image intensity in (1).We model this by parameter with a value in the range [0,1]

(2)

Additionally, most cameras apply a gamma adjustment tomap the range of intensity values from the CCD to the avail-able output range. Assuming it is implemented in the cameraelectronics just before digitization, (2) changes to

(3)

with the gamma value which is assumed to be time constantand equal for all pixels.

1We use � for the mean of x and � for its standard deviation.

0018-9456/$25.00 © 2007 IEEE

200 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 56, NO. 1, FEBRUARY 2007

TABLE ICAMERAS USED TO VERIFY THE CCD MODEL

Fig. 1. Different sets of data are created by covering part the measurement object. Shown are the measuring object and a frame from each data set from theSiemens camera. The sets are: Full: the entire object is visible; Top5: the five darkest sections of the middle row and the entire bottom row are visible; Top2: thetwo darkest sections of the middle row and the entire bottom row are visible; Low6: only the six sections at the bottom are visible; Low3: only the darkest threesections are visible. (a) Black-illumination off. (b) Full. (c) Top5. (d) Top2. (e) Low6. (f) Low3.

III. MEASUREMENTS

For verification of the model given above, measurements wereperformed with a range of different camera types. The camerasused are listed in Table I. The following measurements will bedescribed. For each camera the gamma will be estimated using alog-log plot of the pixel value against the true (photon-counted)intensity value. We measure the dark current by plotting thegamma-corrected pixel intensity against the true intensity. Thisgives a straight line that intersects the pixel intensity axes at thedark current. The additive and multiplicative noise are estimatedfrom a plot of the per-pixel variance against its mean. This gives

a straight line of which the offset and the slope give the additiveand multiplicative noise respectively. Difference in the slope ofthis line for different illumination conditions shows the kind ofapparent gain correction used by the camera.

As measurement object we used a half-transparent plate whichis homogeneously back-illuminated. In front of this plate arelayers of gray and brown filters, see Fig. 1(a). There are eighteendifferent sections, zero to five layers of the gray filter on the toprow and zero to thirteen layers of the brown filter on the centerand bottom row. The illumination from each of the thirteenbrown sections decreases according to , with the numberof layers. This has been verified using a photon counter.

WITHAGEN et al.: CCD COLOR CAMERA CHARACTERIZATION FOR IMAGE MEASUREMENTS 201

Fig. 2. Results from the CCD verification experiments. The left shows a log-log plot of the image intensity per section against the true intensity of the section.The center graphs show a plot of gamma-corrected image intensity per section against the true section intensity. The right graphs show the standard deviationover time plot against the average over time. The different sets of data are recorded with different parts of the measurement object, see the legend in of Fig. 1.(a) JVC, gamma. (b) JVC, gamma-corrected. (c) JVC, noise. (d) Siemens, gamma. (e) Siemens, gamma-corrected. (f) Siemens, noise. (g) JAI, gamma. (h) JAI,gamma-corrected. (i) JAI, noise. (j) Philips, gamma. (k) Philips, gamma-corrected. (l) Philips, noise.

Imagery depicting this object was recorded with all cameras.To estimate the temporal mean and variance, sequences of 100images were recorded. For investigating the automatic apparentgain control, we recorded the measurement object with differentbrighter parts of the object covered. This changed the coveredpart of the scene to black, while leaving the remainder of thescene unchanged, see Fig. 1. This will activate the automatic ap-parent gain control while leaving some sections to measure on.

A. Gamma Estimation

For each sequence of 100 images the standard deviation andaverage of one (the red color) channel were calculated, both perpixel, and for the sections with equal intensity as a whole. Toignore saturation effects, pixel with a value in the top 10% andbottom 10% of the intensity range are not taken into account.

The gamma is estimated from a log-log plot of the averageintensity per section against the true (photon counter measured)

202 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 56, NO. 1, FEBRUARY 2007

TABLE IIRESULTS OF THE CAMERA CHARACTERIZATION FOR EACH OF THE CAMERAS. ALL ESTIMATES ARE LEAST SQUARES ESTIMATES OF THE DATA AVAILABLE WITH

THEIR STANDARD DEVIATIONS. THE AMOUNT OF MULTIPLICATIVE NOISE IS GIVEN FOR THE HIGHEST PIXEL VALUE (ONE IN OUR CASE)

intensity of that section using a least-squares linear fit in Matlab.All data points should lie on a straight line (at least for pointswith intensity much greater than the dark current) with a slopeequal to gamma. For the actual results see the figures on the leftin Fig. 2. The estimated gammas for all cameras are given inTable II. As expected, all cameras have a gamma smaller thanone and for most cameras the individual data points are closeto a straight line. Therefore, we conclude that our model of thegamma is sufficient. However, this does not hold for the we-bcam. The error in the fit is larger as can be explained by themeasurement error. Additionally, Fig. 2(j) shows that the datais not accurately modeled by the exponential model and thegamma differs between measurements. The estimated value ofgamma differs with intensity between 0.53 and 0.72. The stan-dard deviation of the JVC camera is higher than expected.

B. Dark Current Estimation

Using the gamma estimated above, we correct the image se-quences and recalculate the average intensity for each section.Plotting these against the true intensity of the sections nowshould give a straight line which intersects the section-intensityaxes at a value related to the dark current, see the figures in thecenter of Fig. 2.

The dark current is independent of apparent gain. Before fit-ting a line, the data from all sequences is scaled such that theirapparent gain is equal to the lowest of the sequences for thatcamera. Then one line is fitted through all data, using a linearleast squares fit in Matlab. The estimated dark current and theerror of the estimate are given in Table II. For all cameras thedark current we estimated is smaller then the standard deviationof the estimate, so we cannot conclude that it is unequal to zero.The dark current is for all cameras lower than the additive noise,so we conclude that for pixels with a sufficiently large intensitywe can neglect the dark current.

C. Noise Estimation

The distribution of the noise in the CCD model contains bothadditive and multiplicative noise. We plot the standard devia-tion over the gamma-corrected images against its average fora number of pixels to see the effect of both contributions, seethe figures on the right in Fig. 2. The intersection of a straightline fitted to this data with the standard deviation axes gives thecontribution of the additive noise and the slope of the line givesthe amount of multiplicative noise. Averages over all sequencesof the amount of additive and multiplicative noise are given inTable II. We conclude that the amount of multiplicative noiseexceeds the amount of additive noise at intensities greater than10 to 30% of the intensity range.

For the webcam the additive part of the noise is more impor-tant than the multiplicative part over the entire intensity range,see Fig. 2(l).

We observe that for most cameras, the fitted lines throughthe noise are almost parallel. This suggests equal camera gain,indicating that these cameras control the intensity by changingtheir iris opening or shutter time. For the Siemens camera thisdoes not hold, see Fig. 2(f). For this camera the noise changeswith a changing image intensity, but not with the same amountas the slope of the intensity in Fig. 2(e). This is caused by thecombination of automatic gain and automatic shutter control.

D. Artifacts

The webcam seems to have some artifacts, see Fig. 2 bottomrow. Among others its gamma changes with intensity and is in-sufficiently modeled by the exponential. Therefore, this cameraviolates the general model given in (3).

IV. SIMPLIFICATIONS TO THE CCD MODEL

Our experimental validation concludes that we can neglectthe contribution of the dark current as the measured value lieswithin one standard deviation from zero and is smaller than theadditive noise. This simplifies (3) to

(4)

The remaining noise terms are all zero-mean. The shot noiseis multiplicative and both the readout noise and the

quantization noise are additive.Our experiments also show that the additive noise contri-

butions ( and ) are equal to or smaller than the multi-plicative noise contribution for sufficiently large intensity values(larger than 10–30% of the intensity range). For sufficientlylarge intensity this simplifies the equation above to

(5)

V. CONCLUSION

A model of a CCD model was introduced and experimentallyevaluated using a range of different cameras. Experiments showthat the model of the CCD is sufficient for all cameras, exceptthe low-end webcam. The gamma correction in these camerasis sufficiently accurately described by an exponential gammamodel.

Experiments further demonstrate that for sufficiently largepixel intensities the model can be simplified. Specifically, thedark current can be neglected and the additive noise is exceeded

WITHAGEN et al.: CCD COLOR CAMERA CHARACTERIZATION FOR IMAGE MEASUREMENTS 203

by the multiplicative noise at intensities over 10 to 30% of theintensity range.

Using this model, all cameras except for the webcam can beused for accurate measurements. Using the webcam for com-puter vision can be expected to give problems as its responsecannot be predicted using a common CCD model. The gammais not accurately modeled by the general model and depends onthe image intensity.

Results on the webcam show that it is dangerous to pick ageneral camera model and assume its validity. It is importantto validate the model for the specific camera used. As only onecamera has been tested for each type, the values of the modelparameters are not necessarily valid for other cameras of thattype.

Automatic correction to intensity changes is performed dif-ferently by the different cameras. The JAI camera has no au-tomatic intensity adjustment; the JVC camera changes shuttertime; the Siemens camera changes both shutter time and gain;and the webcam seems to change the value of gamma. This hasan effect on the image noise. For the JAI and JVC camera theamount of noise for a certain pixel intensity is independent ofthe apparent gain, whereas for the webcam and Siemens camera,this is not true.

Our experiments show that when using cameras for measure-ment tasks, it is important to verify the model of the sensor.

REFERENCES

[1] K. Kamikura, H. Watanabe, H. Jozawa, H. Kotera, and S. Ichinose,“Global brightness—variation compensation for video coding,” IEEETrans. Circuits Syst. Video Technol., vol. 8, no. 8, pp. 988–1000, Aug.1998.

[2] Y. Altunbasak, R. M. Mersereau, and A. J. Patti, “A fast parametricmotion estimation algorithm with illumination and lens distortion cor-rection,” IEEE Trans. Image Process., vol. 12, no. 4, pp. 395–408, Apr.2003.

[3] B. Xie, V. Ramesh, and T. E. Boult, “Sudden illumination change de-tection using order consistency,” Image Vis. Comput., vol. 22, no. 2, pp.117–125, 2004.

[4] N. Ohta, “A statistical approach to background subtraction for surveil-lance systems,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2001,pp. II:751–II:767.

[5] G. E. Healey and R. Kondepudy, “Radiometric CCD camera calibra-tion and noise estimation,” IEEE Trans. Image Process., vol. 16, no. 3,pp. 267–276, Mar. 1994.

Paul J. Withagen received the M.Sc. degree inapplied physics from the Pattern RecognitionGroup, Delft University of Technology, Delft, TheNetherlands, in 1998. The subject of his M.Sc.thesis was automatic classification of ships usingforward looking infrared (FLIR) images. In 2006,he received the Ph.D degree from the University ofAmsterdam, Amsterdam, The Netherlands, for hisdissertation “Object detection and segmentation forvisual surveillance.”

He then started working as a Researcher at theTNO Physics and Electronics Laboratory (now called TNO Defence, Securityand Safety), The Hague, The Netherlands. There he conducted research in acollaboration with the Intelligent Systems Group, University of Amsterdam.The subject of the research was object detection and segmentation for visualsurveillance. He is currently a System Designer on image processing/imagequality at Philips Medical Systems, Best, The Netherlands.

Frans C. A. Groen (F’05) became a scientific staffmember of the Pattern Recognition Group at theApplied Physics Department, Delft University ofTechnology, Delft, The Netherlands, in 1970, wherehe was responsible for the Robotics Research. In1988, he became Full Professor at the ComputerScience Department, University of Amsterdam, Am-sterdam, The Netherlands, heading the research inintelligent autonomous systems. He served part-timeas Full Professor at the Free University of Ams-terdam (from 1986 to 1996) were he set up a group

in physics applied computer science. He stayed as Fullbright Research Scientistin 1984 at the Robotics Institute at Carnegie Mellon University, Pittsburgh, PA,and in 1996 as Visiting Professor at the University of Utah, Salt Lake City.He has served in international and national advisory boards, published over180 papers and supervised 27 Ph.D. theses. He is currently the Director of theInformatics Institute, University of Amsterdam. His current research interest inperception for intelligent autonomous systems and cooperative robotics.

Klamer Schutte received the Ph.D. from the Uni-versity Twente, Enchede, The Netherlands, in 1994for his dissertation “Knowledge-based recognition ofman-made objects.”

After a two-year stay as a Postdoctoral researcherat the Pattern Recognition Group, Delft Universityof Technology, Delft, The Netherlands, in 1996, hejoined TNO Physics and Electronics Laboratory,since 2005 merged into TNO Defence, Securityand Safety. His current position is Chief ScientistElectro-Optics.