Multiple MIPI CSI-2℠ Cameras

Leveraging FPGAs Ted Marena

Director of SoC FPGA Marketing, Microsemi

Agenda •  History & adoption of MIPI CSI-2 image sensors •  MIPI CSI-2 interface primer •  FPGA usage models •  Applications for multiple MIPI CSI-2 image sensors with

FPGAs •  Summary

2

Evolution of Image Sensor Interfaces

ParallelCMOSinterface subLVDS/HiSPiinterface/etc.

In90s&2000s,parallelinterfacewasthenormInthenextdecade,variousproprietaryinterfaceswereintroduced

~2-3MP

As mobile platforms exploded, so did CSI-2

4

Because of mobile popularity, everyone drifted to CSI-2

•  Just like a decade ago when PC components were used broadly, as mobile adoption exploded, so did the acceptance of MIPI CSI-2

5

Title & Date

“LPMode”–1.2VCMOS“HSMode”–SLVS200;Vcm=200mV&Vdiff=+/-100mV

MIPI CSI-2 D-PHY Overview

•  Waveforms

LowPowerMode:

TransiUonfromLowPowertoHighSpeed

HighSpeedMode

TransiUonfromHighSpeedtoLowPower

LP11,LP01,LP00,HS0,HS-SYNC….HS0,LP-11

ReferencesthePandNchannelvalueinLPmoderespecUvely

ReferencesthevalueoftheSLVS200signalstate

…00011101=‘hB8

MIPI CSI-2 Transition

MIPI Byte Packet Signaling

8’hB8

8’hB8

4-laneexample

byte_data[7:0] {VC,DT} Byte(0) Byte(4) CS ECC Byte(n)…

…byte_clk

sync

byte_data[15:8] WC Byte(1) Byte(5) {VC,DT} Byte(0) Byte(n-1)…

8’hB8 WC Byte(2) Byte(6) WC Byte(1) CS…

8’hB8 ECC Byte(3) CS WC Byte(2) CS…byte_data[23:16]

byte_data[31:24]

8’b00011101HS-SYNCSequence

PacketHeader

VC–VirtualChannelnumberDT–DataTypeWC=16bitwordlengthcount

ECC–ChecksumonwordcountCS–Checksumondata

MIPI D-PHY, CSI-2 Overview

•  Universal D-PHY

Title & Date

1.2VCMOSTransminer

1.2VCMOSReceiver

1.2VCMOSContenUonDetecUonCircuit

SLVS200Transminer

SLVS200Receiver

DynamicTerminaUon

D-PHY vs. C-PHY

•  The majority of MIPI CSI-2 cameras use the D-PHY.

C-PHY,3-wire,embeddedclock,3-phasecoding

D-PHY,diff-pairs,sourcesynchronous,nocoding

TransiUonateverysymbol,usedforclockrecovery

Emulating D-PHY in FPGA •  Normally FPGAs do not have a D-PHY hard I/O block •  They usually interface via translation resistors

11

FPGAI/OsImageSensorI/Os

Key Blocks Used in FPGAs for Imaging

12

AP/uP/uC

MathBlocks/DSP MemoryBlocks

I/OGearing Processor/Micro

FPGAs in Imaging/Video Applications

Bridging is the simplest design.

Acceleration requires more performance & capabilities.

13

FPGAA B FPGA

HWACCA A’

DRAMOp9onal

BridgingAcceleraUon

FPGAs Also Do Processing & Aggregating

Processing could be in an embedded processor or with FPGA fabric, memory and math blocks (DSP blocks).

Aggregation leverages the large I/O capability of FPGAs and the fabric.

14

FPGAAggrega9on

A

A’

DRAM

Processing AggregaUon

FPGAProcessing

A A’

B’

DRAM

B

C

Why Use an FPGA? •  Not enough MIPI CSI-2 or other inputs on an AP/ISP •  Limited numbers of ISP engines on an AP/ISP •  FPGA may perform some processing allowing for a lower

cost AP/ISP •  Newer capabilities that are not available with an AP/ISP •  An FPGA can implement a complete ISP

15

Multi MIPI CSI-2 Camera Applications •  3D camera / virtual reality •  Dual surveillance •  Multiple-image sensor HDR •  180-degree surveillance •  360-degree panorama •  Surround view automotive •  Depth-detection applications •  Drone usage

16

3D Camera Example

DualImageSensors

Bridge

CSI-2 CSI-21-4DataLanes

Dual CSI-2 Bridge

CSI-21-4DataLanes

Syncandformatlogic

AlthoughISPdevicesorenhavemulUplecamerainputs,theyorenbenefitfromanFPGA.Synchronizingtheimagesensors&arrangingthem

3D Camera FPGA Implementation

DualImageSensors

Bridge

CSI-2 CSI-21-4DataLanes

Dual CSI-2 Bridge

CSI-21-4DataLanes

Syncandformatlogic

TheFPGAcanarrangetheimageinaside-by-sideoratop-bonomconfiguraUon.ThismakesiteasierfortheISPorAPtoprocesstheimage.

Dual Surveillance Camera

DualImageSensors

Bridge

CSI-2 CSI-21-4DataLanes

CSI-21-4DataLanes

CSI-2Tx

AlthoughISPdevicesorenhavemulUplecamerainputs,theyorenbenefitfromanFPGAthatcanarrangetheimage.

ISP

ImageBufferMemory

Displayspliter

Dual Surveillance FPGA Function

ThissoluUonallowsanISPtoprocessbothimages.

•  BothimagesarerecoveredintheFPGA•  TheFPGAcombinesthetwoimagesinto

onefortheISP–orenatop-bonomconfiguraUon

•  ThisallowstheISPtoprocessthetwoimagesasone,buttheoutputcanbesplitintotwoimages

Image Sensor HDR Processing

CSI-2 May go to ISP/AP or storage

Each image sensor captures frames at exactly the same time. A short, medium and long exposure is used for each.

ImageBufferMemory

FPGAwithHDRengine

Image Sensor HDR Processing

CSI-2

Short-, medium- and long-exposure images. Processed using local and global tone mapping, motion artifact correction, etc.

ImageBufferMemory

FPGAwithHDRengine

180-Degree Surveillance Camera

Bridge

CSI-2 CSI-21-4DataLanes

CSI-2 Stitching

CSI-21-4DataLanes

CSI-2

TheimagesUtchingfuncUonismoreeasilydoneinanFPGA.Inthisdesign,theenUreISPcouldalsobeintheFPGA.

ISP

ImageBufferMemory

.

.

.3 or more

180-Degree Surveillance FPGA Function

•  MulUpleimagesarerecoveredintheFPGA

•  Theframesarestoredlikelyinexternalmemory

•  TheFPGAperformsananalysistodeterminewheretomergetheimage

•  TheimagesaresUtchedtogether•  Alsolikelyasmoothingtechniqueisused•  Theimageoutputisthenprocessedin

theFPGAorformanedandpassedontotheISPorAP

360-Degree Cameras

CSI-2

Each image sensor frames are captured and combined. Image processing could be in the FPGA or AP/ISP.

ImageBufferMemory

FPGAwithsUtching,fisheye

correcUon,etc

.

.

.

ISPorAP

360-Degree Cameras •  The FPGA performs an analysis to determine

where to merge the images •  The images are stitched together •  Depending on the output format, fisheye

correction may be implemented •  The image output is then processed in the FPGA

or formatted and passed onto the ISP or AP

26

Surround View Automotive

CSI-2

FrameBuffermemory

FPGA

FPD3LinktoMIPICSI-2

FPD3LinktoMIPICSI-2

Surround View Application •  The FPGA implements the stitching of the images •  It formats the image for the ISP/AP • Or FPGA processes the image and drives the display •  FPGA could add overlay such as directional lines

Multi Camera for Depth Detection

CSI-2

The FPGA processes all the images and provides the lowest possible latency for highest accuracy & response.

ImageBufferMemory

FPGAsyncsdualcamerapairs&doestheprocessing

.

.

.

CSI-2

Couldbeoneormorepairs

Multi Cameras for Depth-Based Analytics DualImageSensors

TheFPGAorAPcomputes3Dpointcloudfromatop-downstereopair.Depth-basedanalyUcscandisUnguishadults,kids,peoplefromshoppingcartsforaccuratepeoplecounUng.

Top-DownDepthImage

FlowMetrics™byPercepTonic

Depth Detection for Robotics •  The FPGA synchronizes each camera pair and processes what each

camera pair sees •  The robot can preciously determine the distance for each axis that

has a camera pair •  Parallel processing of the FPGA gives the robot the quickest

response & accuracy

Multi Camera Drone Application

CSI-2

These dual cameras allow a drone to “see.”

ImageBufferMemory

FPGAsyncsdualcamerapairs,muxes,encrypUon,

etc....

CSI-2

GigE/10GE

Encryptedvideo(wifiorotherRF)

PCIe

Multi Camera Drone Application •  The FPGA can merge, mux, pre-process, and run

analytics pre-processing before the AP gets involved for 3D point cloud.

Adds situational awareness so that drones do not crash into anything.

Summary •  Multiple camera applications continue to grow. •  MIPI CSI-2 adoption will grow and be further adopted

outside of mobile applications. •  An increasing number of FPGAs will support MIPI CSI-2

image sensors. •  The parallel architecture of the FPGA is ideal for multiple

camera applications.

34