1

Digital CamerasEngineering Math Physics (EMP)

Jennifer Rexford

http://www.cs.princeton.edu/~jrex

2

Image Transmission Over Wireless Networks

• Image capture and compression– Inner-workings of a digital camera– Manipulating & transforming a matrix of pixels– Implementing a variant of JPEG compression

• Wireless networks– Wireless technology– Acoustic waves and electrical signals– Radios

• Video over wireless networks– Video compression and quality– Transmitting video over wireless– Controlling a car over a radio link

3

Traditional Photography

• A chemical process, little changed from 1826

• Taken in France on a pewter plate

• … with 8-hour exposure The world's first photograph

4

Image Formation

Digital Camera

Eye

Film

5

Image Formation in a Pinhole Camera

• Light enters a darkened chamber through pinhole opening and forms an image on the further surface

6

Aperture

• Hole or opening where light enters–Or, the diameter of that hole or opening

• Pupil of the human eye–Bright light: 1.5 mm diameter–Average light: 3-4 mm diameter–Dim light: 8 mm diameter

• Camera–Wider aperture admits more light–Though leads to blurriness in the

objects away from point of focus

7

Shutter Speed

• Time for light to enter camera–Longer times lead to more light–… though blurs moving subjects

• Exposure–Total light entering

the camera–Depends on aperture

and shutter speed

8

Digital Photography

• Digital photography is an electronic process

• Only widely available in the last ten years

• Digital cameras now surpass film cameras in sales

• Polaroid closing its film plants this year…

9

Image Formation in a Digital Camera

• Array of sensors– Light-sensitive diodes convert photons to electrons– Buckets that collect charge in proportion to light– Each bucket corresponds to a picture element (pixel)

+10V+10V

+ + + + + ++ + + + + +

PhotonPhoton

++

A sensor converts one kind of energy to another

10

CCD: Charge Coupled Device

• Common sensor array used in digital cameras– Each capacitor accumulates charge in response to light

• Responds to about 70% of the incident light– In contrast, photographic film captures only about 2%

• Also widely used in astronomy telescopes

CCD sensor

11

Sensor Array: Image Sampling

Pixel (Picture Element): single point in a graphic image

12

Sensor Array: Reading Out the Pixels

• Transfer the charge from one row to the next

• Transfer charge in the serial register one cell at a time

• Perform digital to analog conversion one cell at a time

• Store digital representation

Digital-to-analog conversion

13

Sensor Array: Reading Out the Pixels

14

More Pixels Mean More Detail

1280 x 960

640 x 480

1600 x 1400

15

The 320 x 240

hand

The 2272 x 1704

hand

16

Representing Color

• Light receptors in the human eye– Rods: sensitive in low light, mostly at periphery of eye– Cones: only at higher light levels, provide color vision– Different types of cones for red, green, and blue

• RGB color model– A color is some combination of red, green, and blue– Intensity value for each color

0 for no intensity 1 for high intensity

– Examples Red: 1, 0, 0 Green: 0, 1, 0 Yellow: 1, 1, 0

17

Representing Image as a 3D Matrix

• In the lab this week…– Matlab experiments with digital images

• Matrix storing color intensities per pixel– Row: from top to bottom– Column: from left to right– Color: red, green, blue

• Examples– M(3,2,1): third row, second column, red intensity– M(4,3,2): fourth row, third column, green intensity

123

21

18

Limited Granularity of Color

• Three intensities, one per color– Any value between 0 and 1

• Storing all possible values take a lot of bits– E.g., storing 0.368491029692069439604504560106– Can a person really differentiate from 0.36849?

• Limiting the number of intensity settings– Eight bits for each color– From 00000000 to 11111111– With 28 = 256 values

• Leading to 24 bits per pixel Red: 255, 0, 0 Green: 0, 255, 0 Yellow: 255, 255, 0

19

Number of Bits Per Pixel

• Number of bits per pixel– More bits can represent a wider range of colors– 24 bits can capture 224 = 16,777,216 colors– Most humans can distinguish around 10 million colors

8 bits / pixel / color8 bits / pixel / color 4 bits / pixel / color4 bits / pixel / color

20

Separate Sensors Per Color

• Expensive cameras– A prism to split the light into three colors– Three CCD arrays, one per RGB color

21

Practical Color Sensing: Bayer Grid

• Place a small color filter over each sensor

• Each cell captures intensity of a single color

• More green pixels, since human eye is better at resolving green

22

Practical Color Sensing: Interpolating

• Challenge: inferring what we can’t see– Estimating pixels we do not know

• Solution: estimate based on neighboring pixels– E.g., red for non-red cell averaged from red neighbors– E.g., blue for non-blue cell averaged from blue neighbors

Estimate “R” and “B” at the “G” cells from

neighboring values

23

Interpolation

• Examples of interpolation

• Accuracy of interpolation– Good in low-contrast areas (neighbors mostly the same)– Poor with sharp edges (e.g., text)

and makes

and makes

and makes

24

Are More Pixels Always Better?

• Generally more is better– Better resolution of the picture– Though at some point humans can’t tell the difference

• But, other factors matter as well– Sensor size– Lens quality– Whether Bayer grid is used

• Problem with too many pixels– Very small sensors catch fewer photons– Much higher signal-to-noise ratio

• Plus, more pixels means more storage…

25

Digital Images Require a Lot of Storage

• Three dimensional object– Width (e.g., 640 pixels)– Height (e.g., 480 pixels)– Bits per pixel (e.g., 24-bit color)

• Storage is the product– Pixel width * pixel height * bits/pixel– Divided by 8 to convert from bits to bytes

• Example sizes– 640 x 480: 1 Megabyte– 800 x 600: 1.5 Megabytes– 1600 x 1200: 6 Megabytes

26

Compression

• Benefits of reducing the size– Consume less storage space and network bandwidth– Reduce the time to load, store, and transmit the image

• Redundancy in the image– Neighboring pixels often the

same, or at least similar– E.g., the blue sky

• Human perception factors– Human eye is not sensitive

to high frequencies

2727

Compression Pipeline

• Sender and receiver must agree– Sender/writer compresses the raw data– Receiver/reader un-compresses the compressed data

• Example: digital photography

compress uncompress

compress uncompress

2828

Two Kinds of Compression

• Lossless– Only exploits redundancy in the data– So, the data can be reconstructed exactly– Necessary for most text documents (e.g., legal

documents, computer programs, and books)

• Lossy– Exploits both data redundancy and human perception– So, some of the information is lost forever– Acceptable for digital audio, images, and video

2929

Lossless: Huffman Encoding

• Normal encoding of text– Fixed number of bits for each character

• ASCII with seven bits for each character– Allows representation of 27=128 characters– Use 97 for ‘a’, 98 for ‘b’, …, 122 for ‘z’

• But, some characters occur more often than others– Letter ‘a’ occurs much more often than ‘x’

• Idea: assign fewer bits to more-popular symbols – Encode ‘a’ as “000”– Encode ‘x’ as “11010111”

3030

Lossless: Huffman Encoding

• Challenge: generating an efficient encoding– Smaller codes for popular characters– Longer codes for unpopular characters

English Text: frequency distribution

Morse code

3131

Lossless: Run-Length Encoding

• Sometimes the same symbol repeats– Such as “eeeeeee” or “eeeeetnnnnnn”– That is, a run of “e” symbols or a run of “n” symbols

• Idea: capture the symbol only once– Count the number of times the symbol occurs– Record the symbol and the number of occurrences

• Examples– So, “eeeeeee” becomes “@e7”– So, “eeeeetnnnnnn” becomes “@e5t@n6”

• Useful for fax machines– Lots of white, separate by occasional black

32

Joint Photographic Experts Group

• Starts with an array of pixels in RGB format– With one number per pixel for each of the three colors– And outputs a smaller file with some loss in quality

• Exploits both redundancy and human perception– Transforms data to identify parts that humans notice less– More about transforming the data in Wednesday’s class

Uncompressed: 167 KB Good quality: 46 KB Poor quality: 9 KB

3333

108 KB 34 KB

8 KB

Joint Photographic Experts Group (JPEG)

Lossy compression

34

New Era of Computational Photography

• Beyond manual editing of photographs– E.g., to crop, lighten/darken, sharpen edges, etc.

• Post-processing of a single photography– De-blur photos marred by camera shake– Changing the depth of field or vantage point

• Combining multiple pictures– Creating three-dimensional representations– Stitching together photos for a larger view– Creating a higher-resolution picture of a single scene

• <Insert your idea here>

http://www.news.com/8301-13580_3-9882019-39.html

35

Conclusion

• Conversion of information– Light (photons) and a optical lens– Charge (electrons) and electronic devices– Bits (0s and 1s) and a digital computer

• Combines many disciplines– Physics: lenses and light– Electrical engineering: charge coupled device– Computer science: manipulating digital representations– Mathematics: compression algorithms– Psychology/biology: human perception

• Next class: compression algorithms