Optimizing Industrial Embedded Imaging Development

Ultrafast Laser Micromachining

Clear Vision for UAVs

SPIE Photonics West 2019 Preview

January 2019

Supplement to Tech Briefs

4 Photonics & Imaging Technology, January 2019

FEATURES

6 Ultrafast Laser Micromachining

10 Embedded Imaging Development

APPLICATION BRIEFS

14 Cleaning or Etching Items with Unique Geometries

16 Clear Vision for UAVs

TECH BRIEFS

19 Imaging System Makes Back Surgery Safer, Faster, LessExpensive

19 New X-Ray Scanning Method Reveals How ParticlesMove in Solution

20 Decoding Multiple Frames from a Single ScatteredExposure

22 Microresonators Offer Simpler Approach to Sensing withLight Pulses

22 Ultra-Thin "Meta-Lens" Enables Full-Color Imaging

SPECIAL SECTION

24 SPIE Photonics West 2019 Preview

DEPARTMENTS

38 New Products

ON THE COVER

With San Francisco being the host city of PhotonicsWest 2019, Critical Link’s engineering team produceda Powell & Hyde Streets cable car image using theMitySOM proprietary edge-detection algorithm, run-ning onboard a system on module (SOM) without thesupport of a PC. Advances in industrial embeddedimaging technology are making development cyclesfaster and more cost-effective. To learn more, readthe feature article on page 10.

(Image courtesy of Critical Link)

CONTENTS

10 Photonics & Imaging Technology, January 2019

Machine vision systems have tradi-tionally consisted of stand-alonecameras tethered to PC-based

boards or frame grabbers using high-speed interfaces such as Camera Link,CoaXPress and Gigabit Ethernet. Bytransferring this data to host computermemory, off-the-shelf software can beused to analyze the captured images andextract information or make a pass/faildecision. While such systems are effec-tive, they generally rely on a very expen-sive and space-consuming host comput-er to perform much of the image pro-cessing.

Most commonly, image processingalgorithms are implemented using off-the-shelf software packages such as HAL-CON from MVTec and MATLAB fromMathworks. In recent years, there hasbeen a growing demand to perform pre-processing on the image data before it istransferred from the camera to the PC.As a result, many of today’s cameras fea-ture on-board FPGA fabric to performsuch functions as Bayer interpolation,image enhancement, and noise reduc-tion directly on the cameras themselves. While adding an FPGA to the camera

introduces additional capability, it does

not address the never-ending push toreduce the size, weight and power(SWaP) and recurring cost of imagingand machine vision systems, particularlyapplications such as portable medicalinstrumentation, robotics, sensory aug-mentation systems, and unmanned aeri-al vehicles. To achieve dramaticallylower cost and SWaP, alternatives to PC-based systems must be considered.

Alternative ApproachesIn the past, designers needing to elimi-

nate the PC have had the option to eitherdevelop a complete “smart camera” fromscratch or use an off-the-shelf camera withcustom-developed embedded processor/FPGA boards to host the image processingsoftware. Both of these options requireextensive schedule and engineeringresources to develop, and can be expen-sive to produce if volumes are not high. The advent of the System on Module

(SOM) has allowed a third option,which is to select an off-the-shelf proces-sor board (the SOM) and develop sim-plified interface boards to connect cam-eras designed with off-the-shelf modulesor custom-designed sensor boards. SuchSOMs typically contain one or moreprocessors (RISC or DSP, or both),FPGA fabric, RAM, and I/O capability,while supporting multiple operating sys-tems. SOMs are small (2" ¥ 3" for exam-

Figure 1: (a) Critical Link’s MitySOM-A10S is an Intel/Altera Arria 10 SoC board that features dual coreARM and FPGA fabric, on-board power supplies, two DDR4 RAM memory subsystems, a micro SDcard, optional eMMC, on-board RTC, a USB port, and temperature sensors. (b) The MitySOM-A10Sdevelopment kit provides everything needed to initiate your embedded development project, andincludes external interfaces such as USB, FMC expansion connectors, PCI-e expansion headers, andGigabit Ethernet.

New hardware and software tools are making development cycles faster and more cost-effective.

OptimizingIndustrial

Embedded Imaging

Development

By

Trif

f/Sh

utt

erst

ock

.co

m

Photonics & Imaging Technology, January 2019 11

ple), so they can fit into small form fac-tor industrial cameras. An interfaceboard is then used to connect the SOMto external interfaces such as PCI,Gigabit Ethernet, USB3, and CAN Bus.The cost of developing a SOM-based

system is significantly lower than that ofdeveloping a complete camera/proces-sor system, since the most complexaspects of design and layout have alreadybeen performed and tested by the SOMmanufacturer. Then, the interfaceboard design generally requires muchlower design effort, especially whenusing pre-existing development boardschematics and design files as a startingpoint, which some SOM suppliers makeavailable. Furthermore, the SOM suppli-er manages parts procurement and com-ponent obsolescence for the SOM, elim-inating maintenance and supportresponsibilities for the camera designerfor the lifespan of the product.One example of such a SOM is

Critical Link’s MitySOM-A10S, anIntel/Altera Arria 10-based board thatfeatures up to 480KLE FPGA fabric, on-board power supplies, two DDR4 RAMmemory subsystems, a micro SD card,optional eMMC, on-board RTC, a USB2.0 port, and temperature sensors(Figure 1). With an Arria 10 SX proces-sor that features dual-core Cortex-A9 32-bit RISC processors with dual NEONSIMD coprocessors, the multiprocessingunit (MPU) is capable of running anynumber of commercial operating sys-tems, bare metal, and embedded Linux.

Parallel DevelopmentIt is common for SOM suppliers to

offer development kits that include a full-featured base board with multiple inter-faces such as USB2, USB3, RS-232, FMCexpansion connectors, PCI-e expansion

headers, Gigabit Ethernet, HDMI, andCAN Bus. Some suppliers have startedoffering development kits specifically forembedded imaging projects that featurepopular machine vision sensors. Re -gardless of which kit is selected, using anoff-the-shelf development kit means thatsoftware and hardware development canoccur in parallel, which can drasticallyspeed time to market.Development kits give hardware design-

ers the option to use manufacturer-provid-ed schematics and design files as a startingpoint to test out the validity of theirdesigns before committing to their custominterface board. As hardware engineersdevelop the necessary interfaces, softwareengineers can choose from numerouscommercial and open source image pro-cessing libraries to develop the final prod-uct. One such library is the Video ImageProcessing (VIP) suite from Intel, alibrary of FPGA intellectual property (IP)that includes such functions as color spaceconversion, up/down image scaling,frame rate conversion, image convolution,

video I/O formatting, and thresholding. Other off-the-shelf packages include

Mathwork's Image Processing Toolboxthat provides a set of algorithms forimage segmentation, enhancement,noise reduction, geometric transforma-tions, image registration, 3D image pro-cessing and HALCON from MVTec thatoffers FPGA RTL export of the soft-ware’s image processing libraries.

Programming FPGAsAlthough the idea of using an FPGA

for image processing can be dauntingwithout the right experience, new toolsexist to make such development easier.To program the numerous system logicelements, registers, memory blocks, andmultiplier/accumulators that comprisethe FPGA fabric of these devices, devel-opers can use software such as Intel’sQuartus Prime. This enables analysisand synthesis of hardware descriptionlanguages (HDLs) such as VHDL andVerilog to describe the structure andbehavior of digital logic circuits.

Figure 3: Fixed pre-processing functions provided by the sensor board’s FPGA include noise reduction, white balancing, de-Bayering, and an edge detection algo-rithm (shown here, on left) which runs onboard a SOM without the support of a PC.

MitySOM-5CSX VDK

Cyclone V SoC FPGA

DDR3

CameraInterfaceAdapter

OPENCLKernels

(optional)

Camera#1

HDMIMonitor

Camera#2

ALTERAClockedVideo

Output

Dual Cortex-A9Processing

Cores

ALTERAVIP SUITE

FrameFormatter

ALTERAVIP SUITE

IPCores...

MitySOM-5CSX Module – On-board image processing

CameraConnector

#1

CameraConnector

#2

M.2 SSD USB 2.0

EthernetRJ-45

HDMITMDS

Output

Figure 2: The embedded vision development kit uses a MitySOM-5CSx Module with an interface board.The development kit enables rapid prototyping and provides an open architecture for developers toembed off-the-shelf IP or their own proprietary image processing algorithms.

12 Photonics & Imaging Technology, January 2019

Once written in an HDL, the designcan be compiled, timing analysis per-formed and the on-board processorsand FPGA configured to perform therequired function. Quartus Prime, forexample, includes an implementation ofVHDL and Verilog for hardware descrip-tion and visual editing of logic circuits.A number of development tools are avail-

able to speed up the programming of suchdevices for image processing and machinevision applications. One approach wouldbe to use Open Computing Language(OpenCL), a framework for writing pro-grams that executes across CPUs, GPUs,DSPs and FPGAs. Using this method, aprogrammer would write code in C/C++and test and validate the code on a PC.Once verified, processing kernels can berecompiled to register-transfer level(RTL) and dispatched to FPGA gate logic. A similar approach would involve

using the Open Source ComputerVision Library (OpenCV), a C++-basedopen source computer vision andmachine learning software library forcomputer vision applications. OpenCVhas more than 2,500 libraries of algo-rithms used to detect and recognizefaces, identify objects, classify humanactions in videos, track camera move-ments, track moving objects, extract 3Dmodels of objects, and produce 3D pointclouds from stereo cameras.A more advanced approach, however,

involves high-level synthesis (HLS), alsoknown as C synthesis. With this method,code written in C/C++ or MATLAB isanalyzed and transcompiled into a hard-ware description language (such asVHDL) which is, in turn, synthesized tothe gate level using a logic synthesis tool.

Both OpenCL and HLS usually result ina substantial reduction in implementationtime since the software is easier to write and,more importantly, iteration and/or modi-fication can be more easily performed.

Embedded Imaging DesignWhile FPGAs perform pipelined func-

tions very efficiently and quickly, thereare limitations to FPGA-only baseddesigns. Functions such as high/lowpass, thresholding, and Bayer transfor-mations can be partitioned to the FPGA.However, others that require the com-plete image to be iteratively analyzedmay be better suited for running on anembedded CPU or GPU. As an example, the Dart board level

camera series from Basler features anFPGA on the sensor board that doesfixed image pre-processing. Beyondthat, advanced image processing andanalytics would have to be performedoutside the camera’s FPGA. This is per-formed using a SOM module, which isincluded in the Dart’s embedded visiondevelopment kit (VDK) (Figure 2). The VDK consists of the Basler Dart

camera module, an LVDS cable connec-tion to an interface board which fea-tures HDMI, Ethernet, USB 2.0, andGPIO output, and a SOM. Image pre-processing functions such as Bayer con-version, bad pixel correction and noisereduction take place on-board the cam-era module. This leaves developers freeto use the SOM’s on-board ARM andFPGA fabric for their own image pro-cessing functions (Figure 3).This modular concept of sensor board,

processor board, and interface board istaken a step further with Critical Link’s

MityCAM architecture. Figure 4 shows ablock diagram of a three-board systemwhere each board allows for a certainlevel of interchangeability depending onthe requirements of the system. CMOSand CCD sensor boards are availablewith an array of image sensors from topmanufacturers, and new sensors can beintegrated quickly and cost effectively. The central processing board provides

options based on SOMs, both of whichfeature ARM and FPGA fabric in an openarchitecture for developers to embedtheir selected image processing functions.Interface boards can be selected with dualCamera Link, USB3, Gigabit Ethernet,HDMI, USB2, and CoaXPress. Regardlessof the individual board selected, the sys-tem can be arranged in a stack that usesstrong board-to-board connectors, and isavailable in production volume with orwithout mechanical packaging. While embedded imaging and

machine vision systems programmingmay initially appear complex, the num-ber of hardware and software develop-ment tools now available are makingthese development cycles faster and morecost-effective. In applications such asportable instrumentation, autonomoussystems, and process automation equip-ment, embedded systems will reduce thesize, weight and power required and theupfront investment of both schedule andengineering resources.This article was written by David Rice, Vice

President of Engineering, and MikeWilliamson, Senior Engineer, Critical Link(Syracuse, NY). For more information, con-tact info@criticallink.com or visit http://info.hotims.com/72989-201.

Sensor Board PROCESSOR BOARD I/O BOARD

DDR3

CMV8000

SensorPower

suppliesPower

Supplies &Distribution

Alter SOCFPGA +

Dual Core ARM

JTAG Micro-SDConsoleDebug

12C ControlData, Clock, Ctr(LVDS, 18 Pair)

Ref Clocks

SPI Control

Camera Link AConnector

Camera Link BConnector

PowerConditioning

Jack+12 VDC

USB- 2.0GPIOConnector

Tri-ColorStatus LED

LVDS (10 pair)

LVDS Serial Control

LVDS (5pair)

Figure 4: (a) Block diagram of Critical Link’s MityCAM architecture, a three board system that features an array of CMOS and CCD sensor options, two proces-sor boards with ARM & FPGA fabric open for user programming, and interchangeable interface boards with options for Gigabit Ethernet, USB3, USB2, CameraLink, CoaXPress, and HDMI. (b) An example of a 3-board camera system in its stacked configuration.

Imaging Software