Visual Computing SystemsCMU 15-869, Fall 2013

Lecture 17:

Image Processing Architectures (and their future requirements)

CMU 15-869, Fall 2013

Smart phone processing resources

CMU 15-869, Fall 2013

Qualcomm snapdragon

Image credit: Qualcomm

CMU 15-869, Fall 2013

Apple A7 (iPhone 5s)

Source: http://www.chipworks.com/en/technical-competitive-analysis/resources/blog/inside-the-a7/

~ Imagination PowerVR Series 6 (G6430)Dual-core ARM

▪ Chipworks estimates:- CPU + cache: 17% area

- GPU: 22% area

(Imagination PowerVR Series 6 (G6430))

- Big SRAM block above GPU: 3-4 MB?

CMU 15-869, Fall 2013

Discussion▪ Traditional rule of thumb in system design is to design simple, general-purpose

components. This is not the case with mobile processing systems (perf/watt)

▪ Needs of high bandwidth sensing and media processing are a big part of these designs [image/video/audio processing, 2D/3D graphics]- User interfaces are visually rich- Games- Speech recognition- Photography/video

- Acquire signal, compute images (not directly measuring an image)- More processing --> smarter sensing- More processing --> more !exibility?

▪ Questions for architects:- Re-homogenize, or become increasingly heterogeneous?- How does an application developer think about these systems?

CMU 15-869, Fall 2013

Frankencamera

CMU 15-869, Fall 2013

Frankencamera context▪ Cameras are cheap and ubiquitous▪ Signi"cant processing capability on cameras▪ Many techniques for combining multiple photos to overcome

de"ciencies in traditional camera systems

▪ But... ability to implement techniques on cameras was limited- Cameras not programmable by general public- Where some programmability did exist, interface too basic

(end result was that latency between two photos was high, mitigating utility of multi-shot techniques)

CMU 15-869, Fall 2013

Example: high dynamic range images

Source photographs: varying exposure Tone mapped HDR image

Credit: Debevec and Malik

CMU 15-869, Fall 2013

More multi-shot photography examples

Flash-no-!ash photography [Eisemann and Durand](use !ash image for sharp, colored image, infer actual room lighting from no-!ash image)

“Lucky” imaging

Take several photos in rapid succession: likely to "nd one without camera shake

CMU 15-869, Fall 2013

Frankencamera goals1. Create open, handheld camera platform for researchers

2. De!ne system architecture for computational photography applications - Motivated by impact of OpenGL on graphics application and graphics hardware

development (portable apps despite highly optimized GPU implementations)

- Motivated by proliferation of smart-phone apps

Nokia N900 Smartphone ImplementationF2 Reference Implementation

[Adams et al. 2010]

Note: Apple was not involved in Frankencamera’s industrial design. ;-)

CMU 15-869, Fall 2013

F-cam components

Sensor **

Image Processor

Device(Lens)

Device(Flash)

Extensibility Mechanism

** Sensor is really just a special case of a device

CMU 15-869, Fall 2013

Shot▪ A shot is a command

- Actually it’s a set of commands - Encapsulates both “set state” and “perform action(s)” commands

▪ De"nes state (con"guration) for: - Sensor- Image processor- Relevant devices

▪ De"nes a timeline of actions- Exactly one sensor action: expose- Optional actions for devices- Note: timeline extends beyond length of exposure (“frame time”)

CMU 15-869, Fall 2013

Shot▪ Interesting analogy:

- An F-cam shot is very similar to an OpenGL display list- A shot is really a series of commands (both action commands and state

manipulation commands)- State manipulation commands specify the entire state of the system- De"nes precise timing of the commands (no OpenGL analogy for this)

CMU 15-869, Fall 2013

Frame▪ A frame describes the result of a shot

▪ A frame contains:- Reference to corresponding image buffer- Statistics for image (computed by image processor)- Shot con"guration data (what was speci"ed by app)- Actual con"guration data (con"guration actually used when acquiring image)

CMU 15-869, Fall 2013

Question▪ What problem in conventional camera interface designs does

F-cam address: throughput or latency?

CMU 15-869, Fall 2013

Aside: latency in camera systems▪ Often in this class our focus is on achieving high throughput

- Triangles per clock- Pixel per clock

▪ But low latency is critical in many visual computing domains- Camera metering, focus- Optical !ow, tracking

Example:CMU smart headlight project

[Charette et al. 2012]

CMU 15-869, Fall 2013

CMU smart headlight

data transfer from sensor to processor

data transfer from processor to projector

Problem: current sensor, projector interfaces operate at frame (not pixel or pixel row) granularity.

CMU 15-869, Fall 2013

F-cam “streaming” mode▪ System repeats shot (or series of shots) in in"nite loop

▪ Stops only when application says so

▪ Intended for “live view” (digital view"nder) or metering mode

CMU 15-869, Fall 2013

F-cam as an architecture

Sensor

Image Processing

Device(Lens)

Device(Flash)

Completed Frames

Event Queue

Cmd Processor

RAW Data

Image Buffers

...

Application Commands (“Shots”)

Stream Cmd Buffer

Memory

Frames

Image Data

CMU 15-869, Fall 2013

F-cam scope▪ F-cam provides a set of abstractions that allow for

manipulating con"gurable camera components- Timeline based speci"cation of actions- Feed-forward: no feedback loops

▪ F-cam architecture performs image processing, but...- This functionality is not programmable- F-cam does not provide an image processing language- Other than work performed by the image processing stage, F-cam

applications do all their own image processing (e.g., on smartphone/camera’s CPU or GPU resources)

CMU 15-869, Fall 2013

NVIDIA Chimera▪ Software framework for writing computational photography pipelines

▪ Idea: application provides kernel functions for CPU, GPU (or speci"es how to con"gure/use the ISP) and describes how to connect up the kernels

CMU 15-869, Fall 2013

F-cam extension: programmable image processing

Sensor

Image Processing

Device(Lens)

Device(Flash)

Completed Frames

Event Queue

Cmd Processor

Frames

RAW Data

Image Buffers

...

Application Commands (“Shots”)

Stream Cmd Buffer

Memory

Image Data

CMU 15-869, Fall 2013

Class design challenge 1▪ If there was a programmable image processor, application

would probably seek to use it for more than just on data coming off sensor

▪ E.g., HDR imaging app

CMU 15-869, Fall 2013

Class design challenge 2

▪ Question: How does auto-focus work in F-cam?

▪ How might we extend the F-cam architecture to model a separate autofocus/metering sensor?

CMU 15-869, Fall 2013

Class design challenge 3

▪ Should we add a face detection unit to the architecture?

▪ How might we abstract a face detection unit?

▪ Or a feature extractor?

CMU 15-869, Fall 2013

Architecture is hard.

CMU 15-869, Fall 2013

Discussion▪ Is there a need for a camera “App Store”?