Embed Size (px)
Citation preview
ACCELERATING INDUSTRIAL AUTOMATION WITH DEEP LEARNING AND EMBEDDED VISION
MIRSEE ROBOTICS WHITE PAPER
JANUARY 2019
ABSTRACT Deep learning has the potential to radically improve industrial automation. Deep learning models are able to solve difficult vision applications with high precision and reliability[1]. Unlike conventional programmatic approaches to automating inspection, deep learning allows machines to learn by example. Deep learning models can accomplish vision tasks that would be too impractical or difficult to accomplish with traditional machine vision techniques. Areas where deep learning-based image analysis excel include assembly verification, defect detection, cosmetic flaw inspection, material classification -- tasks that once required human inspection to accomplish. In addition to advances in vision software, new devices that can perform these tasks have emerged. Embedded vision refers to the integration of image capture and processing within the same device and offers many advantages over traditional machine vision implementations[2]. This has been made possible with the advent of high-performance embedded processors and advanced image signal processors. By applying deep learning-based image analysis on embedded vision systems, also known as ‘AI-on-the-edge’, significant improvements to production cost and efficiency can be realized.
© 2019 Mirsee Robotics Inc. All Rights Reserved. Mirsee, Hadron, and the Mirsee logo are trademarks of Mirsee Robotics. All other trademarks and copyrights are the property of their respective owners.
INTRODUCTION
Machine vision systems have been an integral component of modern production lines for several decades. They have substantially reduced the requirement on human inspection and delivered significant performance improvements to production. Machine vision excels at quantitative measurements of a structured scene, whereas human vision excels at qualitative interpretation of complex, unstructured scenes. Machine vision systems have limitations in this regard, and so the reliance on human inspection still remains. Many factors such as part orientation, complex surface textures, glare, and other variables can make machine vision systems unreliable if not properly controlled[1]. Despite being limited in the types of vision tasks that can be performed, the algorithms behind them are very computationally expensive. To meet these demands, machine vision systems implement a variety of technologies to achieve the performance requirements of the production line. Demanding vision applications traditionally use a combination of cameras, rack-mounted or industrial PCs, image capture hardware, and complex network setup. While this configuration can be versatile, it is inefficient, unnecessarily complex, and not optimal if the objective is to maximize performance and lower cost. DEEP LEARNING Deep-learning is a branch of Machine Learning algorithms that learn through forming layers of abstraction[3]. It is suggested the human brain learns in a similar fashion, and that could explain how deep learning-based image analysis perceives the world in a similar manner[4]. Deep learning models are able to perform difficult vision tasks, the ones humans are still relied on to perform, which are too difficult or impossible to accomplish with traditional machine vision systems. Although deep learning based image analysis is capable of performing complex vision tasks, machine vision remains appropriate for performing simpler vision tasks, such as measurement, presence/absence, and barcode reading (Table 1). By combining both traditional machine vision with deep learning, reliance on human visual inspection is further reduced and performance improvements realized.
Table 1. Vision tasks most appropriate for traditional machine vision, deep learning-based vision, or both.
Traditional Machine Vision Both Deep Learning-based Vision
Dimension Measurement Counting Cosmetic Inspection
Presence/Absence Feature Location Assembly Verification
Barcode Reading OCR Deformation detection
Robotic Guidance Defect detection Texture/Material Classification
© 2019 Mirsee Robotics Inc. All Rights Reserved. Mirsee, Hadron, and the Mirsee logo are trademarks of Mirsee Robotics. All other trademarks and copyrights are the property of their respective owners.
EMBEDDED VISION Embedded vision systems integrate image capture and processing within the same device. Expensive dedicated servers performing vision processing are no longer necessary[2]. Modern System-on-Chip (SoC) devices are more than capable of performing these same tasks at a lower cost and higher speed. Given the widespread adoption of open-source tools for computer vision and deep learning, and the large communities supporting these tools, configuring these systems for your application isn’t overly difficult. By utilizing a highly integrated yet configurable embedded vision system, deep learning-based vision can be deployed with less effort, increase the automation of your line[5], and result in reduced system complexity (See Figure 1.).
Fig. 1. Comparison between traditional machine vision and embedded vision implementations.
HADRON EMBEDDED VISION SYSTEM Hadron is designed to be the most versatile high performance and cost effective embedded vision device. Hadron pairs two high-resolution Sony IMX377 CMOS sensors with the fastest embedded processor built specifically for deep learning applications — the NVIDIA® Jetson™ TX2 (Figure 2)[6]. The Jetson TX2 is the ideal embedded processor for both neural network training and deployment. Its small form-factor enables inference-at-the-edge, eliminating network congestion and the need for external vision processing.
Fig. 2. NVIDIA® Jetson™ TX2 System-on-Module.
© 2019 Mirsee Robotics Inc. All Rights Reserved. Mirsee, Hadron, and the Mirsee logo are trademarks of Mirsee Robotics. All other trademarks and copyrights are the property of their respective owners.
The dual camera configuration of Hadron™ enables object localization and 3D sensing capability, exceeding the performance of competing stereo camera products (Table 2). The camera sensors on board are connected over a high speed CSI-2 interface directly to the Jetson™ TX2 processor using custom low-latency camera drivers. This interface offers many advantages over common interface protocols, as outlined in Table 3. The high bandwidth of the CSI-2 interface allows for high resolution image capture at high frame rate and low latency. The high-resolution IMX377 sensors are able to full advantage of the high bandwidth interface (see Table 4 for sensor specifications). Hadron supports variety of resolutions, frame-rates, lens mounts, and camera mounts to make it suitable for any vision application. Combined with a variety of ports available (Figure 3), Hadron offers greatest value and versatility of any embedded vision solution on the market.
Table 2. Stereo camera comparison.
FLIR Bumblebee 2 e-con Systems Tara D3D KIT P1300 Hadron™
Sensor Sony ICX204 ON Semi MT9V024 ON Semi Python 1300 Sony IMX377
Max. Resolution* 1032 x 776 752 x 480 1280 x 1024 4000 x 3000
Megapixels 0.8 MP 0.3 MP 1.3 MP 12 MP
Sensor Type Colour CCD Monochrome CMOS Colour CMOS Colour CMOS
Interface Firewire 400 USB 3.0 USB Vision (USB 3.0) MIPI CSI-2
*Maximum output resolution of the specific sensor Table 3. Comparison of common camera interfaces for the transmission of uncompressed video data.
USB 2.0 FireWire 400 GigE Vision® USB3 Vision® 4-lane MIPI CSI-2℠
Maximum Bandwidth 40 MB/s 50 MB/s 125 MB/s 360 MB/s 1250MB/s
Relative Max. Resolution* Low Low Moderate High Very High
Relative Max. Framerate^ Low Low Moderate High Very High
Relative Latency Moderate Moderate High Low Very Low
*To achieve a 60fps framerate ̂At full-HD resolution
© 2019 Mirsee Robotics Inc. All Rights Reserved. Mirsee, Hadron, and the Mirsee logo are trademarks of Mirsee Robotics. All other trademarks and copyrights are the property of their respective owners.
Table 4. Sony IMX377 sensor specifications.
Resolution Frame Rate (FPS) A/D Conversion bits (bit)
Number of MIPI lanes
Power Consumption (mW)
4000 (H) x 3000 (V) All pixels @ 35fps 34.97 12 4 393
3840 (H) x 2160 (V) 4K UHD @ 60fps 59.94 10 4 305
2048 (H) x 1080 (V) 1080p FHD @ 120fps 119.88 10 4 374
1364 (H) x 720 (V) 720p HD @ 300fps 299.70 10 4 375
Fig. 3. Ports available on all Hadron units.
KEY BENEFITS
➢ Highest performance embedded processor on the market ➢ High-resolution Sony CMOS camera sensors ➢ Highly optimized camera drivers ➢ Built-in image processing hardware ➢ Direct access to hardware vision pipeline ➢ Stereo vision capability ➢ Open-source computer vision, robotics, and deep learning tools pre-installed ➢ Multiple built-in interface options (Ethernet, WiFi, USB 3.0, HDMI, UART, CAN) ➢ Support for thermal camera expansion (FLIR LWIR) ➢ Support for high-speed liquid lens autofocus (Varioptics) ➢ Support for electronic band pass/cut filters ➢ Support for various lens mounts (M12, D14, CS) ➢ Support for internal application-specific expansion modules ➢ Support for battery powered operation
© 2019 Mirsee Robotics Inc. All Rights Reserved. Mirsee, Hadron, and the Mirsee logo are trademarks of Mirsee Robotics. All other trademarks and copyrights are the property of their respective owners.
CONCLUSION Hadron™ embedded vision system delivers deep learning-based image analysis for automating complex vision tasks that until now, only humans could perform. Embedded vision is a modern approach to industrial automation that reduces system complexity and manufacturing cost.
REFERENCES [1] “Deep Learning for Factory Automation: Combining artificial intelligence with machine vision,” Cognex Corporation, Natick, MA, USA, 2018. [Online]. Available: https://www.cognex.com/what-is/deep-learning/for-factory-automation [2] B. Dipert, “Industrial automation and embedded vision: A powerful combination,” International Society of Automation, Durham, NC, USA, 2014. [Online]. Available: https://www.isa.org/standards-and-publications/isa-publications/intech-magazine/2014/may-jun/features/factory-automation-industrial-automation-and-embedded-vision-a-powerful-combination/ [3] J. Schmidhuber, “Deep Learning in Neural Networks: An Overview,” Neural Networks, vol. 61, pp. 85-117, 2015. [4] D. L. K. Yamins and J. J. DiCarlo, “Using goal-driven deep learning models to understand sensory cortex,” Nature Neuroscience, vol. 19, pp. 356-365, 2016. [5] “Deep Learning-Based Solutions for the Automotive Industry,” Cognex, Natick, MA, USA, 2018. [Online]. Available: https://www.cognex.com/resources/white-papers-articles/whitepaperandarticlemain?event=36b2a81b-4241-42bd-b79f-fd66377ed4c6
[6] D. Franklin, “NVIDIA Jetson TX2 Delivers Twice the Intelligence to the Edge,” NVIDIA, March 17, 2017. [Online]. Available: https://devblogs.nvidia.com/jetson-tx2-delivers-twice-intelligence-edge/
Mirsee Robotics Inc. 96 Grand Avenue South Cambridge, Ontario N1S 2L9
Web: https://mirsee.com Email: [email protected]
Tel: +1-855-472-3309
© 2019 Mirsee Robotics Inc. All Rights Reserved. Mirsee, Hadron, and the Mirsee logo are trademarks of Mirsee Robotics. All other trademarks and copyrights are the property of their respective owners.
