"NVIDIA VisionWorks, a Toolkit for Computer Vision," a Presentation from NVIDIA

Copyright © 2016 Your NVIDIA 1

NVIDIA VisionWorks, a Toolkit for

Computer Vision Elif Albuz

May 3, 2016


Computer Vision

Intelligent Video Analytics

Augmented Reality

Robotics Drones Autonomous Driving

Computational Photography


Computer Vision Application Development

Concept

Reference Implementation

Productization

Port to target & optimize


• Deliver high performance, robust

computer vision primitives

• Ease development of computer

vision applications on Tegra

embedded platforms

• Accelerate prototype to product cycle

VisionWorks Motivation

Depth Map

Optical Flow

Corner detection


VisionWorks at a Glance

Thread-safe API

Documentation, tutorials, sample software pipelines that teach use of primitives and framework

CUDA accelerated library

(OpenVX primitives + NVIDIA extensions + Plus Algorithms)

Flexible framework for seamlessly adding user-defined primitives. Interoperability with OpenCV


VisionWorks Toolkit Software Stack

CUDA Acceleration Framework

OpenVXTM API

VisionWorks

API Extensions

OpenVXTM Framework

VisionWorks

Framework

Extensions

VisionWorks

SfM

NVIDIA

Khronos

VisionWorks

Core Library

Source

Samples

VisionWorks Source Samples Feature Tracking, Hough Transform, Camera

Hist Eq., Stereo Sample

NVXIO

Multimedia

Abstraction

VisionWorks-

Plus

VisionWorks

Object Tracker . . .

VisionWorks

CUDA API


JETSON TK1 Pro JETSON TK1

VisionWorks Supported Platforms

Ubuntu Linux 12.04,

Ubuntu Linux 14.04,

Windows 8

Drive PX JETSON TX1

Automotive Embedded Desktop


VisionWorks Primitives

IMAGE

ARITHMETIC Absolute Difference

Accumulate Image

Accumulate Squared

Accumulate Weighted

Add/ Subtract/ Multiply +

Channel Combine

Channel Extract

Color Convert +

CopyImage

Convert Depth

Magnitude

MultiplyByScalar

Not / Or / And / Xor

Phase

Table Lookup

Threshold

FLOW & DEPTH Median Flow

Optical Flow (LK) +

Semi-Global Matching

Stereo Block Matching

IME Create Motion Field

IME Refine Motion Field

IME Partition Motion Field

GEOMETRIC

TRANSFORMS Affine Warp +

Warp Perspective +

Flip Image

Remap

Scale Image +

FILTERS BoxFilter

Convolution

Dilation Filter

Erosion Filter

Gaussian Filter

Gaussian Pyramid

Laplacian3x3

Median Filter

Scharr3x3

Sobel 3x3

FEATURES Canny Edge Detector

FAST Corners +

FAST Track

Harris Corners +

Harris Track

Hough Circles

Hough Lines

ANALYSIS Histogram

Histogram Equalization

Integral Image

Mean Std Deviation

Min Max Locations

NVIDIA

extensions

All OpenVX

Primitives

+ type/mode extension by NVIDIA

NVIDIA extension primitives


Feature

Tracker Stereo Depth

Extraction

OpenCV-NPP-OpenVX Interop

Hough Lines &

Circles

• + Video stabilization

• + Iterative Motion Estimation/Flow

and other platform specific samples (available only on certain platforms)

• Camera Capture, OpenGL interop, Video playback

VisionWorks Sample Applications


Structure From Motion Object Tracker

VisionWorks Plus Algorithms


• VisionWorks provides 3 levels of API

• VisionWorks OpenVX™ Immediate Mode

• Standard specified heterogeneous compute API with individual

function calls

• VisionWorks OpenVX™ Graph Mode

• Heterogeneous compute API with graph optimizations

• Extensible with user defined nodes

• VisionWorks CUDA API

• Direct CUDA API for advanced CUDA developers

VisionWorks Programming Model


• VisionWorks OpenVX Immediate mode API enables developers to come up to speed fast

• Started with original OpenCV implementation of VideoStabilization application

• Ported to VisionWorks Immediate Mode (prefixed as nvxu)

• Outcome: Video stabilization application is accelerated by 4.75x

(including the overhead for Mat to vx_image conversions)

VisionWorks OpenVX™ Immediate Mode

Video Stabilization

Color Conversion

Optical Flow

Stabilized

frames

Cv::Mat to Vx_image

Process pts & Find

Homography

Warp Perspective

OpenCV image

Source Feature

detection

Image Pyramid

1.3x

2.3x

2.6x

10.1

x 1.5x 8.4x


OpenVX - fusion

• OpenVX Graph enables further optimizations

• Buffer reuse

• Kernel fusion

• Developer controlled CPU-GPU execution

• Efficient use of streaming and CUDA textures

VisionWorks OpenVX™ Graph Enabled

Structure From Motion

Color Conversion

Channel Extract

Optical Flow

Harris Track

Image Pyramid

Point

Clouds

Frame Capture/

Processing

Camera Motion

Estimation

Two-view Triangulation

Scale Estimation

Frame

Source


• VisionWorks CUDA API enables developer with low-level access

• Developer manages the data allocations and transfer

• Developer identifies efficient scheduling and pipelining

• Note that this API operates on the GPU, and does not allow switching between CPU and

GPU

VisionWorks CUDA API

Feature Tracking

YUV

frame

Gray

frame

Camera/image/video

Input data Rendering/Output

nvxcuColor Convert

nvxcuChannel Extract

nvxcuOpticalFlowPyrLK

nvxcuHarris Track

nvxcuGaussian Pyramid

RGB

frame as

CUDA

buffer

Array of

keypoints


Debugging with VisionWorks


VisionWorks Documentation


• VisionWorks Toolkit enables demos and applications, shortens

productization time

• OpenVX immediate mode enables quick port from other libraries

• OpenVX graph mode brings advanced optimizations, hiding overheads of

the target platforms

• VisionWorks CUDA API enables users with lower level access

• Heterogeneous API which enables switching from GPU to CPU version of

the pipeline is very powerful, reducing productization time

Conclusion


• http://www.embedded-vision.com/

• https://www.khronos.org/openvx/

• https://developer.nvidia.com/embedded/visionworks

Resources

http://www.embedded-vision.com/



https://www.khronos.org/openvx/

https://developer.nvidia.com/embedded/visionworks

Technology

"NVIDIA VisionWorks, a Toolkit for Computer Vision," a Presentation from NVIDIA