Simple LCD Transmitter Camera Receiver Data LinkGrace Woo, Ankit Mohan, Ramesh Raskar, Dina Katabi

LCD Display

Camera

Figure 1: Simple setup with a commodity cam-era and standard LCD screen.

ABSTRACT

We demonstrate a freespace optical system using aconsumer camera and projector in indoor environmentsusing available devices for visual computing. Through de-sign, prototype and experimentation with this commodityhardware, we analyze a practical optical solution as wellas the drawbacks for current wireless challenges unmet byclassic RF wireless communication. We summarize andintroduce some new applications enabled by such similarsetups.

1. Introduction

We consider a simple hardware setup using a con-sumer camera and a standard LCD screen such asthe one shown in Figure 1. There are currently manyvisible light communication (VLC) broadcast config-urations exploring the potentials of a data link estab-lished using custom freespace optical hardware equip-ment. This design differs in it’s use of a consumerhardware setup and aims to achieve high bit rate asa result.

1.1 Related Work

Komine et. al presented prototypes and a funda-mental framework for transmission of white LED lightin [4, 5]. More recently, Little et. al [6] build an indoorwireless lighting system also employing OOK timemodulation. This class of works represent an effort

to demonstrate visible light data transfer systems us-ing classic temporal techniques. QR Codes, Data Ma-trix Codes, Shot Codes and EZ Codes are examplesof popular 2D barcodes which encode information tobe transmitted visually. These modes of 2D visual in-formation transmission already conform to ISO stan-dards developed [2, 3, 1] to utilize 2D encoding anddecoding of visual information.

1.2 Contributions and LimitationsOur implementation of a display and camera pro-

vides high bit rate transfer of up to 30Mbps usinga single link where the transmitting LCD screen is astandard Dell 24 inch plasma screen and the receivingcamera is a Canon Rebel SLR with standard featuresettings. Here we summarize the fundamental contri-butions and limitations of this work:

Contributions:

• We build an end-to-end transmit and receiveprototype system using a standard Dell 24” LCDdisplay as the transmitter, and a commodityCannon SLR camera as the receiver where thetotal oversampling rate is 1 pix to 4 pix x 4pix.The bitrate per frame demonstrated in this workis 4.96Mbits per frame with an uncoded averageBER of 10%.

• This is an assertion of a channel model for apixel-to-pixel communication system based onan actual implemented system

• Based on the observed channel properties, wepropose several application scenarios that makeuse of the visible light channel.

The limitations of this approach are the same asany existing visible light link approach. One primarygoal of this work is to characterize key benefits tousing a visible light transmitter to receiver pair.

Limitations:

• Line of sight is required between the transmitterand the receiver.

• End-to-end visible communication poses newproblems, such as geometric angular effects andresampling issues, which potentially call for newalgorithms and protocols.

• Performance of visible light systems is ultimatelydetermined by hardware. Key performance en-hancements are determined by hardware which

1

CameraLCD

...0101101...

...0101101...

Figure 2: An LCD and a camera used for com-munication in our Netpix system. The LCDdisplays images corresponding to the input bi-nary data; the camera captures a photo of thedisplay and decodes the images to recover thedata.

...0101... Display Camera

BERk,l,m

...0101...

Figure 3: A feedback system allows for betteradjustment and fine tuning of parameters be-tween the transmitter and receiver. The cor-rectly exposed parameters result in a systemwhich may be adjusted depending on the num-ber of users.

we limit ourselves to by using off-the-shelf equip-ment.

2. System Overview

We build on a simple understanding of a channelmodel and propose a communication system that usesa camera and a LCD display to communicate usingvisible light. We describe the main components of ourtransmitter and receiver chain.

2.1 Transmitter

The transmitter proposed considers a transmitterscenario such as the one depicted in Figure 2. We areinterested in the bitrates received at each one of thecameras, with respect to angle and distance as relatedto users in a physical space.

We consider a scenario where incoming bits arefirst compressed. These bits may be split into severalframes to guarantee independence. Next, forward er-ror correction is placed to protect the bits through alossy channel. An additional block in this chain cre-ates a feedback system to determine how these pro-tection bits must be placed in physical space.

Figure 4 shows a block diagram for a system withmultiple receive cameras, and a single transmitting

display. We consider the details of the exposed pa-rameters and how they are related to the results.

Parameter k determines how bits are placed intime depending on the temporal coherence of thechannel. Parameter m determines how bits are placeddepending on where users are in physical space. Con-sidering the need for these two parameters, we sendtwo calibration frames before placing bits sequentiallyin the data frames. A system with feedback such asthat in Figure 3 determines how often calibrationframes are sent. The evaluation from the completereceive chain determines how these parameters k andm are set depending on the environment.

The first calibration frame sent is a framecornercal frame consisting of four corner mark-ers. The second calibration frame sent is a framesampoints frame consisting of a grid containing allsampling points in the next few data frames. Follow-ing these two frames are data frames containing allinformation.

2.2 ReceiverThe high level receive system block diagram is

shown in Figure 5. It is split into two sections wheremany of the preprocessing elements are reminiscientof traditional image processing. The postprocessingelements are more reminiscient of RF design blocks.Here we explain the role of each component.

Spatial Tracking: The purpose of spatial track-ing is to determine where in the scene the data is lo-cated. We search for the corners using a modified fastcorner algorithm [7]. As there might be several cor-ners in the scene, each of the corner candidates usingthe fast corner detector is compared to a large quadri-lateral generated from the second frame. This is donein the system using a function find corners whichtakes as input the incoming frame imin and the sam-pled frames cornercal frame and sampoints frame.

The quadrilateral from the second frame is ob-tained by repeatedly blurring and lowpassing the sec-ond sampling frame and then converting to a high-contrast black and white image. The square is foundby discovering all white regions in the scene and cal-culating a “square score” given by:

min(√

area, perimeter4

)max

(√area, perimeter

4

)2

All distances are calculated from the corner can-didates in the first frame to the centroid of the squarefound in the next frame. The four corners with dis-tances closest to each other are considered the cornercalibration points for this block. The brightest pixelpoint associated with these corner calibration pointsare considered the corners for this round.

2

...0100101010001101... Compress Frame SplitGeometrically

Placed Symbols LCD or Projector

k m

Figure 4: The transmit chain is designed for a scenario where a single screen might be transmittinginformation with many onlooking receivers. The transmit chain carries properties similar tothat of a RF transmit chain with additional parameters for adjusting frame split, forward errorcorrection and awareness of users in physical space.

Homography: Once the scene is found, the per-spective scene may be restored using well known tech-niques for recovering perspective projections. This isdone in the system using a function frame recover.A transfer function is formed using the corners fromfind corners and a homogeneous inversion may beperformed from the formed matrix C.

All the following frames are cropped using cornersfound from find corners and the transform coordi-nates fround from frame recover. These are passedthrough the rest of the receive chain until anotherdark calibration frame is found.

Timing Recovery: Timing recovery is doneby detecting all dots from sampoints frame.sampoints frame recovers the image by adaptivelyequalizing the entiring image and then passing itthrough a high-contrast filter wth contrast limits of0.2. This is then converted to a high-contrast blackand white image. The centroids of each one of thesepoints are found using standard logic functions.

These points are assumed to be contained on anear perfect grid. The points of this mesh are re-ordered as such. This mesh is interpolated by an up-sampling factor of 4 to allow for localized timing off-set. Here, a match filter from the grid length is usedas a kernel for the image. The convolved image is usedto find the optimal sampling points.

Sampling: Sampling is done from the result ofthe timing recovery frame where all maximum val-ues corresponding to a region within a found gridpoint is sampled. Sampling takes as input an incom-ing color image and slices the image into three slicesand considers each slice with grayscale levels. As aresult of the interference discussed, each slice is adap-tively equalized followed by high contrast adjustment.The result is sampled with sampling points found inthe timing recovery.

In reporting the BER of the system, the recoveredframes are compared to the random frames sent.

3. Hardware Overview

In this work, we use only commodity hardware toachieve a high-bit rate link and make no modifica-tions. As discussed in future work, there are many

1 1.1 1.2 1.3 1.4 1.510

−8

10−6

10−4

10−2

100

Distance (meters)

Bit

Err

or R

ate

Red channelGreen channelBlue channelOverall

Figure 6: BER with respect to distance be-tween the camera and display for encoding inall three color channels for a 1599x1035 pixeltransmit block.

prototypes in development out of the scope of thiswork which would allow a much more powerful plat-form. We provide in detail the specifications of thetruly off-the-shelf equipment we choose.

In these experiments, we choose a standardDell Ultra Sharp 24” wide screen flat display with1920x1200 resolution. The screen contains an anti-glare coating. These screens cost approximately $350today. A n x n array of these standard displays simu-late a higher resolution “transmitter” displays avail-able in the future.

We use the “Canon Rebel XS” camera to capturetransmitted images. These cameras likewise cost ap-proximately $500. The sensor resolution is approxi-mately 10 megapixels. The lens is a standard 50mmlens with f/8 aperture size.

4. Results

3

BER

Spatial tracking Homography Timing Recovery Sampling Decoding ...0100101010001101...

Figure 5: The receive chain design is the most computationally heavy portion of the design. Thisblock diagram gives a high level overview, the details are discussed in the text.

Using the transmit chain and receive chain de-scribed in Section 2, we carry out several experimentswhich demonstrate the performance of the Netpix sys-tem. While the system design is based on ideal as-sumptions about the nature of the optical channel,the actual performance of the system is stressed un-der many parameters which are exposed to the highestlayer. There are two goals in these experiments. Thefirst is to establish the performance of the system.The second is to develop a channel model thus givingfundamental insight into the nature of a pixel to pixelsystem.

Figure 6 reports BER with respect to dis-tance for encoding in all color channels for a4.96Mbits/frame/screen in a n x n array of camerasand screens where n = 3. With a shutter rate of 1/30seconds, this gives an aggregate throughput of 1.34Gb/s. Here, the total transmit size of each frame is1599 x 1035 pixels. In this experiment, the array setupis such that each camera is exactly in front of eachdisplay.

Here, we determine when the goodput is 1.34Gb/s- BER*1.34Gb/s. Despite large bit errors, the overallgoodput for a 3 x 3 camera/display array is still veryhigh (≈ 1Gb/s). These transmit frames were gener-ated with 1 bit of information contained in each sliceof red, green and blue.

While it is interesting to consider the circum-stances under which a very high speed link works, wewould also like to understand how these pixels mightbehave under other constraints. Figure 7 is a 1599 x1035 pixel board with bits of symbol size 4. The sym-bol size is the number of pixels on the display thateach bit occupies. Here, we consider the results withno color and how this might eliminate errors. Here, at1m, there are 0 errors and a sharp jump at 1.1m.

Figure 7 reports BER when using only theblack and white channels. This reports only 1/3 thethroughput. These results decoded using the same re-ceive chain however result in much lower BER. Thissuggests a considerable effect from neighboring colorbands.

Angle evaluation is done using a square blackand white checkerboard carrying a throughput of7.86Mbits. Figure 8 shows BER as a function of dif-ferent viewing angles. A viewing angle of 30 degreesmay cover significant area within a room.

1 1.1 1.2 1.3 1.4 1.5

10−4

10−3

10−2

10−1

100

Distance (meters)B

it E

rror

Rat

e

Figure 7: BER with respect to distance be-tween the camera and display for encoding inblack and white. This eliminates the effects ofcolor interference and reports BER purely asa result of what is observed at the sensor.

0 10 20 30 40

10−4

10−3

10−2

Angle (degrees)

Bit

Err

or R

ate

Figure 8: BER with respect to angle betweenthe camera and display for encoding in blackand white. Angle increases in the photos fromleft to right (0, 10, 20, and 30 degrees).

4

0 20 40 60 80 1001.7

1.8

1.9

2

2.1

2.2

2.3x 10

−3

Observation Number

Bit

Err

or R

ate

Figure 9: Coherence plots showing how BERchanges with respect to time. This experimen-tal plot suggests the presence of a stochasticchannel in practice. We further explain the na-ture of this channel in the text.

Figure 9 shows that although the channel modelfor these results are analytical, the resulting bit errorsstill have randomness. We can see this effect in bothangle and time.

Figure 10 shows BER as a function of the sym-bol size. As the symbol size increases, the chancefor neighbors interfering decreases thus reducing theBER.

5. Conclusion

We present a wireless optical system built withcommodity hardware which may be used for manynetwork scenarios. Depending on the application andneeds of the network, there are many modifications tobe made which will ultimately result in significantlyhigher bit rates. We are currently in the process ofdeveloping such a hardware platform. Even with cur-rent commodity off-the shelf hardware, we obtain veryhigh bit-rates. As the consumer world strives to de-velop the newest form factors for projectors and cam-eras, visual communication systems such as the oneproposed in this work will allow users to use a new setof networking components capable of delivering highbit-rate wireless links.

6. References[1] ISO. International symbology specification –

maxicode. ISO/IEC 16023:2000, 2000.[2] ISO. Automatic identification and data capture

techniques – QR code 2005 bar code symbologyspecification. ISO/IEC 18004:2006, 2006.

[3] ISO. Automatic identification and data capturetechniques - Data Matrix bar code symbologyspecification. ISO/IEC 16022:2006, 2006.

1 2 4 80

0.05

0.1

Symbol size

Bit

Err

or R

ate

0°10°20°30°

Figure 10: BER vs. symbol size viewed as re-ported for various angles between the cameraand the display. The symbol size is the numberof pixels on the display that each bit occupies.The BER falls down significantly as the symbolsize goes up. Performance is similar for differ-ent angles.

[4] T. Komine and M. Nakagawa. Integrated systemof white led visible-light communication andpower-line communication. IEEE Trans. onConsumer Electronics, Feb 2003.

[5] T. Komine and M. Nakagawa. Fundamentalanalysis for visible-light communication systemusing led lights. IEEE Trans. on ConsumerElectronics, Feb 2004.

[6] T. D. C. Little, P. Dib, K. Shah, N. Barraford,and B. Gallagher. Using led lighting forubiquitous indoor wireless networking. IEEEIntl. Conf. on Wireless and Mobile Computing,Networking and Communications, October 2008.

[7] E. Rosten and T. Drummond. Machine learningfor high-speed corner detection. In EuropeanConference on Computer Vision, May 2006.

5