How to come up with new Ideas Raskar Feb09

Ramesh Raskar, MIT Media Lab

After X, what is neXt

Coming up with New Ideas in Imaging



Xd

X++

X X+Y

X

X

neXt


Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Camera CultureMIT Media Lab http://raskar.info

http://cameraculture.info

Ramesh RaskarAssociate Professor

Create tools to better capture and share visual information

The goal is to create an entirely new class of imaging platforms

that have an understanding of the world that far exceeds human ability

and produce meaningful abstractions that are well within human comprehensibility


Camera CultureCamera Culture

Course WebPage :

http://cameraculture.info/courses/


After X, what is neXt

Coming up with New Ideas in Imaging



Xd

X++

X X+Y

X

X

neXt



Simple Exercise .. Simple Exercise ..

What is neXt


Strategy #1: XStrategy #1: Xdd

• Extend it to next dimension (or some other) dimensionExtend it to next dimension (or some other) dimension

• Context aware resizing Context aware resizing – VideoVideo– Instead of square resizing-> CD cover (with a hole in center) resizingInstead of square resizing-> CD cover (with a hole in center) resizing

• Text, Audio (Speech), Image, Video .. Whats next ?Text, Audio (Speech), Image, Video .. Whats next ?

• Video, 3D meshes, 4D lightfieldsVideo, 3D meshes, 4D lightfields• Images to infrared, sound, ultrasoundImages to infrared, sound, ultrasound• Macro scale to microscale (Levoy, Lightfield to Microscope)Macro scale to microscale (Levoy, Lightfield to Microscope)• Time to space to angle to idTime to space to angle to id• (coded exposure <- coded aperture)(coded exposure <- coded aperture)

Coded-Aperture ImagingCoded-Aperture Imaging

• Lens-free imaging!Lens-free imaging!• Pinhole-camera Pinhole-camera

sharpness,sharpness,without massive light without massive light loss.loss.

• No ray bending (OK for No ray bending (OK for X-ray, gamma ray, etc.)X-ray, gamma ray, etc.)

• Two elementsTwo elements– Code Mask: binary Code Mask: binary

(opaque/transparent)(opaque/transparent)– Sensor gridSensor grid

• Mask autocorrelation is Mask autocorrelation is delta function (impulse)delta function (impulse)

• Similar to MotionSensor ?Similar to MotionSensor ?

Flutter Shutter CameraFlutter Shutter CameraRaskar, Agrawal, Tumblin [Siggraph2006]

LCD opacity switched in coded sequence

Figure 2 results

Input Image

Problem: Motion Deblurring

Ramesh Raskar, Camera Culture, MIT Media Lab

Image Deblurred by solving a linear system. No post-processing

Blurred Taxi

Ramesh Raskar, Camera Culture, MIT Media Lab

Flutter Shutter: Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo Blurred PhotoPSF == Broadband Function

Fourier Transform

Coded Aperture CameraCoded Aperture Camera

The aperture of a 100 mm lens is modified

Rest of the camera is unmodifiedInsert a coded mask with chosen binary pattern

Out of Focus Photo: Coded Aperture

Captured Blurred Photo

Refocused on Person

Larval Trematode WormLarval Trematode Worm


Strategy #2: X+YStrategy #2: X+Y• Fusion of the dissimilarFusion of the dissimilar

– More dissimilar, more spectacular the outputMore dissimilar, more spectacular the output

• ExampleExample– Scientific imaging + PhotographyScientific imaging + Photography

• Coded apertureCoded aperture• TomographyTomography

• Lightfields + User interfacesLightfields + User interfaces• Projector = cameraProjector = camera

– Spatial Augmented RealitySpatial Augmented Reality


Imaging in Sciences: Imaging in Sciences: Computer TomographyComputer Tomography

• http://info.med.yale.edu/intmed/cardio/imaging/techniques/http://info.med.yale.edu/intmed/cardio/imaging/techniques/ct_imaging/ct_imaging/


Borehole tomographyBorehole tomography

• receivers measure end-to-end travel timereceivers measure end-to-end travel time• reconstruct to find velocities in intervening cellsreconstruct to find velocities in intervening cells• must use limited-angle reconstruction method (like must use limited-angle reconstruction method (like

ART)ART)

(from Reynolds)


Prototype cameraPrototype camera

4000 4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens× 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens

Contax medium format camera Kodak 16-megapixel sensor

Adaptive Optics microlens array 125μ square-sided microlenses


C:\Apps\Photodex\CompuPicPro\compupic.exe C:\levoy\talks\lightfields\1016-wbal-rgb-rebal-crot16.jpg


Example of digital refocusingExample of digital refocusing

Coded-Aperture ImagingCoded-Aperture Imaging

• Lens-free imaging!Lens-free imaging!• Pinhole-camera Pinhole-camera

sharpness,sharpness,without massive light without massive light loss.loss.

• No ray bending (OK for No ray bending (OK for X-ray, gamma ray, etc.)X-ray, gamma ray, etc.)

• Two elementsTwo elements– Code Mask: binary Code Mask: binary

(opaque/transparent)(opaque/transparent)– Sensor gridSensor grid

• Mask autocorrelation is Mask autocorrelation is delta function (impulse)delta function (impulse)

• Similar to MotionSensor ?Similar to MotionSensor ?

Mask in a Camera

Mask

Aperture

Canon EF 100 mm 1:1.28 Lens, Canon SLR Rebel XT camera


Strategy #3: X Strategy #3: X Do exactly the oppositeDo exactly the opposite

• Processing, Memory, BandwidthProcessing, Memory, Bandwidth– In Computing world, in any era, one of this is a bottleneckIn Computing world, in any era, one of this is a bottleneck– But overtime, they change. You can often take an older idea and do But overtime, they change. You can often take an older idea and do

exactly the opposite.exactly the opposite.– E.g. bandwidth is now considered virtually limitlessE.g. bandwidth is now considered virtually limitless

• In imaging:In imaging:– Larger sensors?Larger sensors?

• Everyone is thinking about building cheaper, smaller pixel sensors and THEN Everyone is thinking about building cheaper, smaller pixel sensors and THEN improving SNR .. Maybe just build larger sensors?improving SNR .. Maybe just build larger sensors?

– SLR: Faster mirror flip or no mirror flipSLR: Faster mirror flip or no mirror flip• Companies spent years improving mirror flip speedCompanies spent years improving mirror flip speed• Why not just remove it?Why not just remove it?

• More computationMore computation• Less lightLess light


• e.g. Reverse Auctione.g. Reverse Auction

Less is MoreLess is MoreBlocking Light == More InformationBlocking Light == More Information

Coding in Time Coding in Time Coding in SpaceCoding in Space

Larval Trematode WormLarval Trematode Worm

Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006

Vicon Motion Capture

High-speed IR Camera

Medical Rehabilitation Athlete Analysis

Performance Capture Biomechanical Analysis

Towards ‘on-set’ motion capture

• 500 Hz with Id for each Marker Tag• Visually imperceptible tags + Natural lighting• Unlimited Number of Tags• Base station and tags only a few 10’s $

Traditional: High-speed IR Camera + Body markers

Second Skin: High-speed LED emitters+ Photosensing Body markers


R Raskar, H Nii, B de Decker, Y Hashimoto, J Summet, D Moore, Y Zhao, J Westhues, P Dietz, M Inami, S Nayar, J

Barnwell, M Noland, P Bekaert, V Branzoi, E Bruns

Siggraph 2007

Prakash: Lighting-Aware Motion Capture UsingPhotosensing Markers and Multiplexed Illuminators


Imperceptible Tags under clothing, tracked under ambient light

Hidden Marker Tags

Outdoors

Unique Id

Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006Labeling Space(Indoor GPS)

Each location receives a

unique temporal code

But 60Hz video projector

is too slow

Projector

Tags

Pos=0

Pos=255

Time


Pattern

MSB

Pattern

MSB-1

Pattern

LSB

For each taga. From light sequence, decode x and y

coordinateb. Transmit back to RF reader (Id, x, y)

0 1 1 0 0 X=12


Inside of Multi-LED Emitter


Tag


• When life gives you lemon, make lemonadeWhen life gives you lemon, make lemonade

Ramesh Raskar, Karhan Tan, Rogerio Feris, Ramesh Raskar, Karhan Tan, Rogerio Feris, Jingyi Yu, Matthew TurkJingyi Yu, Matthew Turk

Mitsubishi Electric Research Labs (MERL), Cambridge, MAMitsubishi Electric Research Labs (MERL), Cambridge, MAU of California at Santa BarbaraU of California at Santa Barbara

U of North Carolina at Chapel HillU of North Carolina at Chapel Hill

Non-photorealistic Camera: Non-photorealistic Camera: Depth Edge Detection Depth Edge Detection andand Stylized Rendering Stylized Rendering

usingusing Multi-Flash ImagingMulti-Flash Imaging

Depth Discontinuities

Internal and externalShape boundaries, Occluding contour, Silhouettes

Depth Edges

Our MethodCanny

Canny Intensity Edge Detection

Our Method

Photo Result

Car Manuals

What are the problems with ‘real’ photo in conveying information ?

Why do we hire artists to draw what can be photographed ?

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Shadows

Clutter

Many Colors

Highlight Edges

Mark moving parts

Basic colors

A New Problem


Strategy #4: X Strategy #4: X • Given a Hammer ..Given a Hammer ..

– Find all the nailsFind all the nails– Sometimes even screws and boltsSometimes even screws and bolts

• Given a cool solution/technique, Given a cool solution/technique, – find other problemsfind other problems

• Good recent examplesGood recent examples– Gradient domain techniquesGradient domain techniques

• Introduced in Graphics for High dynamic range tone Introduced in Graphics for High dynamic range tone mapping [Fattal Lischinski 2002]mapping [Fattal Lischinski 2002]

• Now a major hammerNow a major hammer– Image editing, compositing, fusion, alpha matting, reflection layer recoveryImage editing, compositing, fusion, alpha matting, reflection layer recovery

A Night Time Scene: Objects are Difficult to Understand due to Lack of Context

Dark Bldgs

Reflections on bldgs

Unknown shapes

Enhanced Context :All features from night scene are preserved, but background in clear

‘Well-lit’ Bldgs

Reflections in bldgs windows

Tree, Street shapes

Background is captured from day-time scene using the same fixed camera

Night Image

Day Image

Result: Enhanced Image

Flash Result Reflection LayerAmbient

Flash and Ambient ImagesFlash and Ambient Images[ Agrawal, Raskar, Nayar, Li Siggraph05 ][ Agrawal, Raskar, Nayar, Li Siggraph05 ]

Agrawala et al, Digital Photomontage, Siggraph 2004

Agrawala et al, Digital Photomontage, Siggraph 2004

actualphotomontageset of originals

perceived

Source images Brush strokes Computed labeling

Composite


Strategy #5: X Strategy #5: X • Given a problem, find other solutionsGiven a problem, find other solutions

– Given a nail, find all hammersGiven a nail, find all hammers– Sometimes even screwdrivers and pliers may workSometimes even screwdrivers and pliers may work

• High Dynamic Range Tone MappingHigh Dynamic Range Tone Mapping– Started with Jack Tumblin’s LCISStarted with Jack Tumblin’s LCIS– Gradient domainGradient domain– Bilateral filterBilateral filter– Filter banks etc .. Filter banks etc .. – About 6 years of heavy machineryAbout 6 years of heavy machinery– Btw, the topic is done to death but continues to enthuseBtw, the topic is done to death but continues to enthuse


Strategy #6: X++Strategy #6: X++• Pick your adjective ..Pick your adjective ..• Making it faster, better, cheaperMaking it faster, better, cheaper

neXt = adjective + XneXt = adjective + X


X++ : Add your favorite adjectiveX++ : Add your favorite adjective• Context aware, Context aware, • AdaptiveAdaptive• (temporally) Coherent, (temporally) Coherent, • Hierarchical, Hierarchical, • ProgressiveProgressive• EfficientEfficient• ParallelizedParallelized• DistributedDistributed

• Good example: Image or video compression schemesGood example: Image or video compression schemes• But X++ is a bad signBut X++ is a bad sign

– The field is dying in terms of research but booming in business impactThe field is dying in terms of research but booming in business impact


PitfallsPitfalls• These six ways are only a start These six ways are only a start • They are a good mental exercise and will They are a good mental exercise and will

allow you to train as a researcherallow you to train as a researcher• Great for class projectsGreat for class projects• But But

– Maynot produce radically new ideasMaynot produce radically new ideas– Sometimes a danger of being labeled incrementalSometimes a danger of being labeled incremental– Could be into ‘public domain ideas’Could be into ‘public domain ideas’


What are Bad ideas to pursueWhat are Bad ideas to pursue• X then Y (then Z)X then Y (then Z)

– X+Y is great with true fusion, fusion of dissimilar is bestX+Y is great with true fusion, fusion of dissimilar is best– But avoid a ‘pipeline’ systems paper, where the output of But avoid a ‘pipeline’ systems paper, where the output of

one is THEN channeled into the input of the next stage, one is THEN channeled into the input of the next stage, and non of the components are noveland non of the components are novel

– E.g. I want to build a E.g. I want to build a • Follow the hype (too much competition) Follow the hype (too much competition) • Do because it can be done Do because it can be done

– (Why do we climb? because it is there! (Why do we climb? because it is there! – But only the first one gets a credit. But only the first one gets a credit. – May make you strong, and give you a sense of May make you strong, and give you a sense of

achievement but not a research project. )achievement but not a research project. )


Xd

X++

X X+Y

X

X

neXt


Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Camera CultureMIT Media Lab http://raskar.info

http://cameraculture.info

Self Improvement

How to come up with new Ideas Raskar Feb09