WHITE PAPERwww.baslerweb.com

1

Better Image Quality with Basler PGITo achieve very high image quality using digital cameras, the different camera components and functions must interact in perfect harmony.

Our innovative PGI technology delivers a harmonized combination of 5 × 5 Debayering, Color-Anti-Aliasing, Improved Sharpness and Denoising. It produces images with significantly improved brilliance, fidelity of detail and image sharpness while simultaneously reducing noise. Because it is integrated compactly into the camera‘s FPGA, PGI is fully real-time compatible, almost latency free and frees up memory in the PC for image editing software.

Content

1. The Bayer Matrix as the Basis for Optimization of Color in Images .......................................................... 01

2.1 Debayering ........................................................................ 01

2.2 PGI Debayering ................................................................02

3.1 Color-Anti-Aliasing ....................................................... 03

3.2 PGI Color-Anti-Aliasing ............................................... 04

4.1 Image Sharpness ............................................................ 04

4.2 PGI Improved Sharpness ............................................ 04

5.1 Noise ................................................................................... 05

5.2 PGI Denoising ................................................................. 05

6. Computational Power ...................................................07

7. Summary ............................................................................07

1. The Bayer Matrix as the Basis for Optimization of Color in Images

In the world of image processing, color cameras are increasingly replacing monochrome ones. Color cameras deliver color images, which contain significantly more information than monochrome images. In a color image, each pixel is comprised of multiple color values, such as the values for the colors Red (R), Green (G) and Blue (B). These images are thus referred to as ‘RGB color’ images.

Color cameras with real three-color image sensors are highly complicated and expensive. One good and afford-able alternative comes through a common type of color camera featuring image sensors using the so-called Bayer matrix or Bayer pattern. The pattern was invented in 1975 by Bryce E. Bayer, an employee at Eastman Kodak, and patented under US patent number 3,971,065.

In a Bayer matrix, each pixel works with its own color screen that resembles a chess board. 50 % of those masks are colored green, with 25 % of each of the remaining ones in red or blue (see Fig. 1).

There is no mandatory speci-fication for which color must be placed in the upper left corner of the Bayer matrix. For this reason there are four different options, as seen in Fig. 1, that vary from sensor

to sensor. Depending on which colors occupy the first two pixels of the first row, they are referred to as RG, BG, GB or GR (see Fig. 2).

Fig. 3 shows a detail of the raw image taken by a Bayer sensor, with the respective pixel colors clearly visible:

2.1 Debayering

On a Bayer sensor, each pixel only sees one color. A proper RGB color image requires three colors per pixel, namely red, green and blue. The missing colors are inserted using a technique called ‘interpolation.’ The spe-cific interpolation method here is known as Debayering, demosaicing or simply as ‘color filter array interpolation.’

GR G

B

GR G

B

GR G

B

GR G

B

GR G

B

GR G

B

GR G

B

GR G

B

GR G

B

GR G

B

GR G

B

GR G

B

GR G

BBG

GR

GB G

RG BR G

Fig. 1: The Bayer matrix

Fig. 2: The four color possibilities for the Bayer matrix: RG, BG, GB and GR

Fig. 3: Raw image with BG Bayer matrix

Please note: The original sizes of Fig. 3, 5, 6, 7, 8, and 10 can be viewed in your browser. Please click on the respective figure.

2

Debayering is based on interpolation. For example, since only 25 % of the pixels are red, a red value must be interpo-lated for the remaining 75 % of the pixels. A variety of inter-polation algorithms are available, and choosing between them has a massive impact on the conceptual approach, concrete implementation, resource requirements and results.

Debayering algorithms work by drawing on the color values from neigh-boring pixels to esti-mate missing colors. The dimensions for this analysis, measured in number of surrounding pixels reviewed for each

pixel, are denominated as 2 × 2, 3 × 3, 4 × 4, 5 × 5, etc. Fig. 4 shows a 2 × 2 and a 5 × 5 environment.

2 × 2 debayering is based on a technique called ‘nearest neighbor interpolation’ - in this white paper referred to as ‘simple debayering’ - and is known for being less resource-intensive. It often produces serious color errors, as can be clearly seen in the following practical sample image (Fig. 5). The most common manifestations are orange and sky blue color errors, especially where black and white meet at edges, such as within the numbers and letters. On diagonal lines, the orange and sky blue alternate con-stantly, which is especially distracting to the human eye. The image also appears to be noticeably unfocused, with diagonal edges appearing to be zippered.

A better interpolation process can improve those results. Of particular benefit is the ability to draw data from a larger surrounding area.

2.2 PGI Debayering

PGI debayering is a new, significantly improved debayering algorithm developed by Basler to work with a 5 × 5 pixel radius.

Compare the previous image with Fig. 6, which uses the same raw data run through the PGI algorithm to produce an RGB image. It is immediately evident that the image quality is better in every regard. No color flaws can be identified, the script is clearly legible and the diagonal edges appear to be smooth and sharp.

The PGI algorithm was designed to produce a significantly better color image quality for the human eye. It has been shown that multiple calculation steps are required for this. Beyond the debayering, these include:

� Removal of color flaws (Color-Anti-Aliasing),

� Improved Sharpness and

� Denoising.

Each of these calculation steps are explained in detail below.

The 5 × 5 debayering in PGI is based on a deep mathematic understanding of the interpolation problem and its solu-tion, combined with an understanding of the properties of the human eye. In complex algorithms of this kind, there are typically options for selecting operating points. These are each selected in such a way that they produce a result especially suitable for viewing by humans.

There are two type of vision cells present in the human retina: The rod cells for seeing in black and white and cones for seeing in color. The rod cells are more common and more densely packed than the cones. As a result, we can see black and white contrasts in much greater resolution.

PGI takes advantage of this characteristic by reproducing sharp black-and-white contrasts in high brilliance and detail fidelity, similar to the image quality in a black-and-white camera. This ensures that contours are highly visible and that images and letters, for example, are particularly easy to read.

Beyond this, the rods for black-and-white vision are also especially sensitive to green colors. PGI accounts for this by giving special weight to the green pixels when recon-structing black-and-white contrasts, thereby producing a more natural image impression.

Fig. 4: A 2 × 2 environment to the left, a 5 × 5 environment to the right.

Fig. 5: Color error due to 2 × 2 debayering, as can be clearly observed in the barcode, for example.

Fig. 6: The raw image from Fig. 3 processed using PGI debayering.

3

Ultimately the human eye is highly perceptive of color noise and individual pixels with color errors. For this reason, a strong emphasis was placed during the devel-opment of PGI on selecting the color reconstruction method with an especially low level of color noise.

3.1 Color-Anti-Aliasing

Color errors, especially on sharp edges, are a common side effect of less effective debayering algorithms. These flaws are especially evident when a raw image like the one below (Fig. 7) is used. The image depicts cosine waves of linearly increasing frequency from left to right and top to bottom.

If you apply a 2 × 2 simple debayering algorithm to this image, for example, then the resulting image looks like this (Fig. 8):

As you can see, the color flaws are in the corners. Based on these corners, the causes are well explained in Fig. 9.

Let's examine this image starting in the upper left corner. It's easiest to correctly reconstruct colors in areas with low spatial frequencies, i.e. areas of the image such as in the upper left where large sections are light or dark and where the brightness changes only occur gradually. We observe no color flaws.

Color errors become much more frequent and prominent in sections where the image has high spatial frequencies. These arise for example when the distances between black and white lines are so tight that they are placed in neighboring pixels. High spatial frequencies also occur at sharp edges with one light and one dark side.

Where vertical structures are involved, the situation may arise where they are depicted as shown in the upper right of Fig. 9. In the first case, the columns with red and green colors receive more light than the columns with the colors green and blue. For this reason the red pixels are afforded an above-average level of brightness, while the blue pixels are assigned a below-average brightness. Too much red combined with too little blue produces an orange color error. In the reverse case, i.e. too little red and too much blue, produces a sky-blue color error.

For horizontal bars, the situation is rotated by 90°. Here the rows with red and green pixels receive more light than those with green and blue pixels, producing orange. If it were reversed. With more light for the rows of green and blue pixels, then a sky blue would appear.

In theory, the situation that appears in the lower right of Fig. 9 is also possible, where diagonal bars produce a magenta and green discoloration. In practice, however, these are typically less frequent, as the optical quality of the lens is insufficient to depict these very tightly packed structures with proper contrast.

Nyquist

Fig. 7: Raw image with cosine waves of increasing frequency from top left to bottom right.

Fig. 8: Raw image from Fig. 7 with 2 × 2 simple debayering.

Fig. 9: The color flaws are especially visible in the corners.

4

3.2 PGI Color-Anti-Aliasing

With PGI, the discolorations occurring through these effects are analyzed and corrected for all potential fre-quencies below the theoretical limit. This theoretic limit is produced from the digital system theory and is called the Nyquist frequency.

Fig. 10 shows the same image after processing with PGI — the discolorations have been eliminated up to the Nyquist frequency for the green pixels. The Nyquist fre-quency for the green pixels runs diagonally from the upper right to the lower left, and can be identified clearly in the image as the threshold for the discolorations.

The effect can also be seen clearly in real images (Fig. 11 and 12).

4.1 Sharpness

As shown above, conventional debayering algorithms are often unable to reproduce fine, sharp structures in good sharpness.

4.2 PGI Improved Sharpness

PGI by contrast delivers a qualitatively exceptional repro-duction of black-and-white structures.

When depicting high-resolution structures, color cameras working with Bayer matrices often produce not just the aforementioned color aliasing effects, but also problems with image sharpness. There are fundamental limits to the pure linear interpolation algorithm. PGI adjusts the linear interpolation algorithm to the image structure. This pro-duces improved image sharpness similar to that found in monochrome cameras.

In this way PGI always delivers strong image sharpness. When faced with especially challenging tasks, there is also an adjustable sharpness factor available. It allows for the image to be re-sharpened in a certain area, such as to com-pensate for the impacts of sub-optimal optics. Through all of this, the image sharpness always remains oriented toward the perceptive characteristics of the human eye.

It should be noted that image sharpness is closely tied to noise, and that linear re-sharpening would fundamentally lead to increases in noise. Because the human viewer per-ceives gray noise as significantly less disruptive than colored noise, PGI works with a sharpening process that leads solely to a moderate increase in gray noise but has no impact on color noise.

In the neutral position for the sharpness factor, the re-sharpening function is switched off. Figures 13 and 14 provide images for comparison:

Fig. 10: PGI Color Anti-Aliasing eliminates false colors for the green pixels.

Fig. 11: With 2 × 2 simple debayering

Fig. 12: The same detail with PGI debayering

Fig. 13: A lack of image sharpness based on standard procedures without sharpening

Fig. 14: Sharpening using PGI

5

Good image sharpness is of major or even critical impor-tance for many applications. This is especially the case for applications in which color cameras must correctly iden-tify numbers or letters. Good legibility is ultimately helpful whether the image analysis is being made by humans or machines. PGI helps especially where precise detection of color and text is needed.

For example, traffic monitoring relies on the ability to cor-rectly record the color of a vehicle and its license plate. For retailers, it is important that the color of the merchandise being scanned is properly detected and that the barcode be properly read.

A crystal-clear image ultimately allows for image pro-cessing tasks to be handled at lower resolutions, i.e. with fewer pixels. This can translate into more affordable equip-ment, from the lighting setup and camera to computing power, and for quicker cycles whereby the camera records more images during the same cycle.

5.1 Noise

Noise is an unavoidable part of any image. The majority of noise results from the fact that light is comprised of many photons. Photons are quantum mechanical parti-cles whose behavior is dictated by a certain randomness. That means that one single pixel, even under constant brightness, sometimes receives more and sometimes fewer photons. The resulting noise is of a temporal nature and is described as Photon Shot Noise.

Another source of noise is the image sensors and their electronic circuits. The image sensor noise is most notice-able in low light and short exposure situations, as well when strong analog amplification is used. Both noise types come together in the image sensor and together produce the temporal noise of the raw image.

Beyond this, local noise is the second important source of noise. It is primarily produced by the fact that light sensi-tivity varies slightly between different pixels. Even minor differences are visible in the image. This effect is referred to as Photo Response Non Uniformity (PRNU).

Particularly in low light or very short exposure time situa-tions, as well as where strong analog amplification is used, visible differences in brightness between pixels in absolutely dark areas can be observed. This effect is known as Fixed Pattern Noise or Dark Signal Non Unifor-mity (DSNU), and varies in impact depending on the specific sensor. It is not typically an issue for high-quality sensors.

The aforementioned noise effects occur in both mono-chrome and color cameras.

There are typically many layers of calculation between the raw image and the finished color image. Each of these computational steps can reinforce or reduce the noise based on the laws of error propagation. It should be noted practically speaking that in many more cases the noise is amplified instead of reduced. Of especially critical importance in this regard is the impact of sequences of mathematical operations, which can exponentially increase the noise from computational step to computa-tional step. In particularly unfortunate cases, this ends up strongly magnifying the overall impact of the noise.

5.2 PGI Denoising

PGI switches the configuration of these operations, exe-cuting them parallel to one another instead of sequen-tially. This avoids constantly compounding noise magnifi-cation. The design of the individual computational steps puts significant priority of maintaining low noise levels. The PGI algorithms thus always produce a pleasantly low-noise image.

Beyond this, PGI features an active noise filtering option with configurable parameters. Because noise filtering for industrial applications is designed for a rapid sequence of individual images, it focuses solely on individual images and avoids the mutual impact felt when sequential image content is evaluated. Noise filtering quality is strongly affected by the scope of available input data. The larger the environment for the noise filtering, the better the potential results — although the computational costs rise rapidly as well.

5 × 5 noise filtering was thus selected for PGI as a reason-able compromise. It is particularly effective for noise fil-tering in the high-quality image sensors used in Basler cameras, as they are already engineered for low to mod-erate noise levels. At the same time, it keeps implementa-tion highly affordable and efficient.

Fig. 15 shows the impact of debayering on noise. In the process, the homogeneous gray areas in (a) are based on the Gaussian noise. (b) shows the image produced using simple debayering. The image looks unfocused on the one hand, yet on the other hand color noise of the type that most disturbs the human eye can clearly be detected. (c) and (d) show the associated distribution of the RGB color vectors in the RGB color spectrum. What is of par-ticular interest is the stretching of the scatter plot, as this represents an intuitive measure for the strength of the noise in the resulting image. It can be seen that the RGB color vectors are significantly stretched, which coincides with the impression of the image in (b).

Fig. 15 (e) shows the results with PGI. Here too significant noise can be detected, although it is combined with sig-nificantly better sharpness of detail. When compared with (b), it can be seen that the color noise is significantly

6

reduced, which is considerably less irritating to the human eye. Here too however a certain level of non-colored noise is also clearly identifiable. This can also be seen in the distribution of RGB color vectors shown in (f) and (g). It can be seen here that the noise scatter plot is signifi-cantly narrower in distribution than the one shown above. This is particularly clear in (f), where the lateral expansion of the scatter plot has been significantly reduced.

Fig. 15 (h) shows how the noise can be further reduced through the use of PGI noise suppression. Only a small amount of noise can still be seen in the image. The distri-butions of RGB color vectors shown in (i) and (j) are also significantly less stretched. A comparison of (i) against (f) shows that the vertical expansion of the noise scatterplate is now in particular significantly smaller.

The noise filter works by interpreting minor deviations in the brightness values as noise, and larger deviations as actual image content. The threshold values for differenti-ating between the two can be parameterized.

It is important to understand that setting that threshold value high may eliminate more than just noise; In some cases it can also filter out the fine structures in the image

that were at risk of being lost in the noise. For this reason a good deal of touch is required in getting those threshold values right. Of prime importance however is that key textures remain unaffected for the interpretation of the image.

The noise filter is deactivated in the neutral setting. When noise filtering is activated, it is often a good idea to acti-vate slight sharpening as well.

6. Computational Power

Beyond the aforementioned aspects, PGI has also been engineered in all aspects to be very lean and low in resource demands. All computational operations are con-ducted parallel and in optimal harmony on a 5 × 5 environ-ment as part of an FPGA (Field Programmable Gate Array) implementation, which thus requires an especially low level of FPGA-internal memory and exceptionally fewer calculation operations. This is the only way to make it possible to execute these complex computations within the camera without measures that lead to higher costs, such as larger FPGAs or other computational units.

Fig. 15: A comparison of the impact of debayering on noise. It shows (a) a raw image with Gaussian noise, (b) a color image produced using simple debayering, (c) and (d) the associated distri-bution of the RGB vectors in the RGB color spectrum, (e) the color image produced using PGI, (f) and (g) the RGB vector distribution for that image, (h) the color image with PGI and noise suppression and (i) and (j) their RGB vector distribution.

a)

g)

d)c)

e)

i)

b)

f)

h) j)

7

Debayering can always be conducted internally or exter-nally for the camera. External debayering, such as by the computer, framegrabber or graphics card, would appear to give the advantage of requiring significantly less band-width to transmit the color image. In theory this would seem to leave more bandwidth available to transmit more images per second.

Yet when high-quality debayering involving extensive computation is performed by the PC, that PC quickly experiences a significant jump in computational load, even when optimized code is used. Because the PC can only process the incoming images at low frames per second rates, the benefit of high transmission bandwidths is negated and the entire process is moot. High-quality debayering will also further slow down the PCs if image editing software is to be used, as is typically the case. On the whole, this produces an unpleasant situation.

For this reason, debayering should be understood as an important, but not the sole, component in the image pro-cessing chain. For good image quality it is important that the image preparation steps be conducted in the proper order. Debayering is at the heart of this, and anyone seeking to move debayering to the computer must also ensure that other upstream and downstream image pro-cessing steps are performed correctly.

For this reason it makes sense to execute the PGI algo-rithm directly in the camera's FPGA. PGI in the camera's FPGA is fully real-time compatible, is positioned at the right spot in the image processing chain and works flaw-lessly with it. Latency times are massively shortened, the connected PC can give its full power to the image editing application without sacrificing on high-quality images directly from the camera.

7. Summary

On the whole it can be said the PGI raises the already exceptional image quality from Basler cameras to new heights. It is designed to optimize the experience for the human eye, delivers exceptionally good debayering with sharply reduced color errors, excellent image detail and low noise levels, while also permitting optional post-sharpening of the image and noise filtering.

Basler AGGermany, HeadquartersTel. +49 4102 463 500

Fax +49 4102 463 599

sales.europe@baslerweb.com

www.baslerweb.com

Basler, Inc.USATel. +1 610 280 0171

Fax +1 610 280 7608

sales.usa@baslerweb.com

Basler Asia Pte Ltd.SingaporeTel. +65 6367 1355

Fax +65 6367 1255

sales.asia@baslerweb.com

©Basler AG, 12/2015

For information on Disclaimer of Liability & Privacy Statement please see www.baslerweb.com/disclaimer

Basler AG

Basler is a leading global manufacturer of digital cameras for industrial and retail applications, medical devices, and traffic systems. Product designs are driven by industry requirements and offer easy integration, compact size, excellent image quality, and a very strong price/perfor-mance ratio. Founded in 1988, Basler has more than 25 years of experience in vision technologies. The company employs 500 people at its headquarters in Ahrensburg, Germany, as well as in international subsidiaries and offices in Europe, Asia, and the Americas.

Authors

Dr. Jörg KunzeTeamleader Predevelopment

Dr. rer. nat. Jörg Kunze joined Basler AG in 1998. As the Team Leader for Predevelopment, he is responsible for new technologies. His areas of exper-tise include image sensors, camera hardware, noise, color fidelity, image quality and computational imaging.

He has developed numerous new algorithms for image signal processing, including several involving color filter array interpolation. His name appears on numerous inven-tions and patents.

Contact

Dr. Jörg Kunze – Teamleader Predevelopment

Tel. +49 4102 463 206Fax +49 4102 463 46206Email: joerg.kunze@baslerweb.com

Basler AGAn der Strusbek 60-6222926 AhrensburgGermany

Sören BögeProduct Manager

Sören Böge has been a Product Manager at Basler since 2015. As part of his work, he is responsible for the new Basler ace CMOS camera models with IMX sensors from Sony's Pregius series, as well as the PYTHON sensors from ON Semiconductor.

After graduating with a degree in industrial engineering, he worked for many years in the automotive industry as a Project Director and Product Manager for a measurement system.

Contact

Sören Böge – Product Manager

Tel. +49 4102 463 693Fax +49 4102 463 46693Email: soeren.boege@baslerweb.com