Cognitive Computational Vision-Based (CVM) Monitoring

•  Real-Time Autonomous Species Identification.

•  100% IFQ Accountable web-based trip-by-trip vessel EM Near Real Time data acquisition.

•  Autonomous computer vision length determination for stock assessment scientists.

CVM

•  Improve the cost-effectiveness and capacity of our Fisheries Information Systems (FIS) for providing adequate field-based observations.

•  Accelerated implementation of electronic monitoring methods (e.g., digital imaging and electronic log books) to complement fishery observer and catch monitoring programs.

•  Advancement of cognitive computing proof-of-concepts in other Agency natural resource monitoring efforts.

Shorebased CVM unit installation,

Bornstein Seafood, Astoria, Oregon

Expected Results, Relative Importance

Colby Brady1, Guy *Paillet2,, Anne Menendez2, Andrew Bornstein3, Kevin Dunn4, & Sheryl Flores5!

Consistent baseline CVM data augmented and calibrated by port sampler sampling data.

Real Time web-based

species ID

?

. Human EM video review methodologies have limited species determinat ion capabilities. However, in fish cognitive computing platforms offer promise to i n s u r e t h a t s p e c i e s d e t e r m i n a t i o n i s consistently well clarified.

We have incorporated novel parallel processing silicon system-on-chip video sensor cognitive computing hardware linked to ruggedized military Central Processing Unit (CPU) hardware to investigate the functionality of a cognitive computational real-time and near real-time (NRT) IFQ web-based data acquisition, and pattern recognition strategy. The video analytic capability of the unique hardware and customized software platform is also being examined to determine if vessel deck discard behavior training and recognition is possible as a workable proof-of-concept strategic IFQ EM strategy. Specifically, these shoreside and vessel CVM prototypes (or “Cognitive DVRs”) consists of “Cognisight” Miniature Trainable Vision Sensors (MTVS) linked to a miniaturized military computer, further linked to a “CogniBlox” configurable NeuroMem vision board, enabling “smart” on-site real-time massively parallel data mining and sensor fusion.

Sampling site: Bornstein Seafood, Astoria, Oregon!

Cognitive Computing hardware platform tools may be uniquely suited for pattern recognition and

ecosystem intelligence applications.

Introduction

Oregon Department of Fish and Wildlife port sampler support for verified species-specific video annotation

(Image Knowledge Builder)

West Coast IFQ fishing vessel

Installation, Captain Kevin Dunn

F/V Iron Lady, Warrenton, Oregon

1National Oceanic and Atmospheric Administration (NOAA-WCR), 2General Vision Inc., 3Bornstein Seafood (Astoria, OR), 4F/V Iron Lady, 5Oregon Department of Fish and Wildlife (ODFW, Astoria)

COMPLETED: •  Hardware engineering design and manufacture of Windows-based CVM equipment delivery. •  Discard detection software development complete, available for testing. Previous test was through a

sensor aimed at a screen. Recent update now able to scan hard drive EM data. •  Web-based streaming and Cognitive “neuron” object training. •  Wireless remote desktop control with Panasonic Toughpad for captain, port sampler, and observer testing. •  “Fish” recognition conducted (species-specific training pending). •  Autonomous frame grabs of individual fish from 24 live video feed (i.e., initial test and “fish” training

captured >3,000 individual fish images on the conveyor belt within two hours during offload).

TO BE COMPLETED: •  Custom Species-specific image library, Zero Instruction Set Computing (ZISC) real-time algorithm

development (which have a completely different software engineering architecture from traditional post-analysis CPU algorithms).

•  Autonomous length determination algorithm development from individual fish frame grabs.

TO BE COMPLETED: •  Installation of CVM unit tentatively

scheduled for the end of January. •  Test manual and autonomous EM full

trip data acquisition of all sorting events.

•  Compare between windows-based ATOM processor CVM prototype unit and linux/android-based ARM processor CVM unit.

•  Discard detection capability (post-observer verification, from shoreside species Image Knowledge Builder (IKB) image library.

•  Trip-by-trip data web-based data acquisition.

Preliminary Information

CogniSight Miniature Trainable Vision Sensor

Features� Image capture and recognition at 60 image

per second on a single board smaller than of

a matchbox

� Can learn and recognize a region of interest

in the field of view as small as 16x16 and up

to the full frame

� Trained by example. The knowledge is

automatically generated by the neural

network embedded in the CogniSight

recognition engine

� Can classify the region of interest into up to

256 categories

� Output results on 16-bit data bus

� Output video on LVDS bus (Low Voltage

Differential Signaling) @ 60 fps.

� I/O control through I2C master bus

� Autonomous operation under very low power

consumption (12 mA). Can be battery

operated (accept voltage from 4 to 30 volts).

� Setup and Training is made through a very

simple PC-based control panel is delivered

with the board.

� The knowledge of the CogniSight engine can

be saved to and loaded from disk file.

Applications

Image to Actuators � industrial inspection Image to Speech � learning systems

Image to Motion � micro-tracking, UAV Image to Storage � video surveillance

Description CogniSight MTVS is the first miniature vision sensor with autonomous recognition capabilities, fully "hardwired" and sinking less than 15 milliamps...Recognition is performed without a single line of software at 60 frames per second. Although it is intended to recognize video

images autonomously, a temporary connection to a host computer is needed to teach the recognition engine what to recognize or to load a pre-defined engine. For this purpose, the module is delivered with a USB adaptor board. Connection to a host can also be of interest to collect images and monitor the recognition. The CogniSight MTVS module comes with a Control Panel software which lets you adjust the settings of the sensor such as its gain, shutter speed and more. You can select your region of inspection and teach the module to recognize up to 4 categories of regions. Also, you can decide to only view images in which the region of interest belongs to a specific category, with the option to save them to disk for selective recording. The knowledge stored in the neurons of the module can be exported to an Image Knowledge File (*.ikf). This IKF file can loaded on other CogniSight MTVS modules or platforms running a CogniSight engine.

U.S. Department of Commerce�|�National Oceanic and Atmospheric Administration�|�NOAA Fisheries�|�Page 12

A multicore processor The CM1K pattern recognition chip with 1024 identical cognitive memories in parallel

CM1K%Image%Recogni0on%Chip%•  Memory'and'processing'logic'combined'in'a'same'

element'•  Parallel'architecture'of'iden7cal'elements'•  Simple'access'to'all'elements'connected'in'parallel'

Modern%CPU%processors%•  Memory'bo9leneck'•  High'power'consump7on'

CogniMem Technology

Modern CPU processors (far left): •  Memory bottleneck. •  High power consumption. CM1K pattern recognition chip (above and left): •  Memory and processing logic

combined in the same element.

•  Parallel architecture of identical elements.

•  Simple access to all elements connected in parallel.

General Vision Inc., Vessel Prototype

U.S. Department of Commerce�|�National Oceanic and Atmospheric Administration�|�NOAA Fisheries�|�Page 21

512 GB SSD (600 Mbytes/second)

SATA 2

Inside Top

Inside Bottom

Dual Core i.MX6XX ARM

Running Android

WiFi Module

For Tablet connection

Gigabit Ethernet

2 USB HS HDMI Output (test)

Power supply 24 Volts input

Outside IP68

Interface up to 8 cameras HD_SDI

1080p synchronized

CogniBlox (2) Real time

Cognitive Video

NOAA Fisheries staff, in collaboration with ODFW biological port sampler staff, have developed dock-side species-specific CVM training protocols that will enable ongoing Image Knowledge Builder (IKB) library development and feature extraction annotation for ongoing ZISC algorithm development. Valuable ODFW port sampling staff collaboration will enable acquisition of verified data-poor species-specific data, including length and sex information on a fish-by-fish basis. ODFW port sampler staff will be able to access web-based PacFin databases immediately in the field using a Panasonic toughpad (windows –based), thereby reducing potential transcription errors and reconciliation validation. If successful, CVM will create additional baseline information streams, which would enable limited port sampling energies to be more targeted and focused on the tasks that humans are particularly adept at gathering (DNA, scale samples, sex determination), while allowing CVM to gather real-time digital data that cognitive computing systems are more adept at gathering (100% computer vision census of lengths, consistent data-poor complex speciation, etc.).

The National Marine Fisheries Service (NMFS), West Coast Region (WCR) is exploring Computational Vision-Based Monitoring (CVM) in the Individual Fishing Quota (IFQ) groundfish trawl fishery. The goal of the research is see if CVM can reduce monitoring costs while providing better and more timely data as compared to current (1) electronic monitoring/reporting (EM, ER) hardware; (2) speciation methodologies, and; (3) logistical issues with periodic collecting and replacing of hard drives in the field. This project is done in collaboration with General Vision Inc. (a cognitive computing hardware and software developer), Bornstein Seafoods, F/V Iron Lady, and ODFW. Funding for this research project was via a Fisheries Information System (FIS) grant.

This project aims to determine if CVM can be used to automatically determine species and length of individual fish at a shoreside processor offloading conveyor belt. This project also aims to determine if an automatic field-based web CVM data transmission strategy could be developed to reduce the need for contractors to retrieve EM data in the field by manually pulling removable hard drives. If successful, a trip-by-trip automatic data transmission strategy could dramatically reduce data verification times and costs to fishermen and management. The CVM unit has initially been deployed on a bottom trawl processing plant conveyor-sorting belt. In addition to use in a processing plant, a further step will be to place a CVM prototype unit on a IFQ bottom trawl fishing vessel.