Ricoh’s Second Stage Machine Vision: EMPOWERING DIGITAL ... · Ricoh’s Second Stage Machine Vision: EMPOWERING DIGITAL WORKPLACES Ricoh has been providing automation and other

Ricoh’s Second Stage Machine Vision:

EMPOWERING DIGITAL WORKPLACES

Ricoh has been providing automation and other labor-saving systems in many areas using machine

vision. Our machine vision activities are now in their second stage and we are working on how we

can elevate workplaces into a space for knowledge creation. Our system autonomously generates

rules, understands the situation, presents optimal methods, and helps people at work to make

decisions quickly. Ricoh aims to establish a smarter workplace through collaboration between

intelligent machine vision and knowledge workers.

20/11/2017 Version 1.0.0

White

Paper

1 © 2017 Ricoh Company, Ltd.

Table of Contents

1. Innovating Workplaces to Enable the Knowledge Creation .................................... 2

2. Delivering Machine Vision to All Workplaces .............................................................. 3

3. Ricoh’s Second Stage Machine Vision Activities .................................................... 4

3.1. Second stage machine vision in social infrastructure .............................................. 5

・Road Surface Inspection System Based on Stereo Cameras and AI ................. 5

・Public infrastructure inspection systems using drones ............................................... 9

・Autonomous flights of drones with a 3D vision system using an ultrawide-angle

stereo camera........................................................................................................................................ 12

3.2. Second-stage machine vision in Factory Automation (FA) .................................. 15

・Automatic random picking system using a stereo camera...................................... 15

・Image recognition and analysis technology for visual inspection based on

machine learning ................................................................................................................................ 17

3.3. Second stage machine vision in logistics ................................................................... 19

・Flow monitoring using stereo camera ................................................................................ 19

4. Tackling Second Stage Machine Vision with Open Innovation .................................. 20


1. Innovating Workplaces to Enable the Knowledge Creation

With the rapid progress of digital innovation, workplaces are about to face a great

change. Productivity has been improved, costs have been cut, and operations have

been accelerated as we know it. The mode of corporate organizations has changed

dramatically in just a few years, and so has the awareness of the people working

there.

Digital innovation has freed people from the constraints of time and space, bringing

opportunities to create new value. The waves of innovation that permeate offices now

reach the frontlines of many industries—manufacturing, logistics, retail, healthcare,

education, and more.

A technology accelerating this movement is artificial intelligence (AI), which has

gained huge attention recently. Particularly, deep learning based on the multi-layer

neural networks, which is one of the technologies1 of machine learning, has found

applications in a variety of examples, enabling many people to recognize AI as

something familiar. Expectations are rising for the dawn of the new era when

machines (systems) are given human-like intelligence.

As systems gain “smartness,” they will be able to do things that people have been

doing or could not do in the past. A typical example is autonomous car driving. The

technology not only reduces the burden on the driver but also increases safety, and

is especially valuable in that regard. Efforts are underway to equip lighting and air-

conditioners with sensors, enabling automatic control of energy consumption

according to the movement of people, thus creating a more comfortable

environment.

Workplace innovation, however, cannot be accomplished by merely introducing a

1 A scheme of finding characteristic patterns out of big data through iterative learning (statistical processing)


system that features “smartness.” Innovation requires the knowledge creation,

including the likes of value judgments and task setting, which cannot be replaced

by AI no matter how excellent it is. Workplaces are reformed through the consistent

knowledge creation by the people working there, and evolve into places for

producing new value. The value of a “smart” system is determined by how much it

contributes to the knowledge creation by people.

People perform a series of processes unconsciously in their daily tasks. A system

must organically coordinate these processes to assist in the knowledge creation in

a workplace. Knowledge creation in a workplace requires a wide scope of

technologies that encompass input, processing, and output, as well as know-how

in formulating an easy-to-use smart solution system.

2. Delivering Machine Vision to All Workplaces

Ricoh has been nurturing technologies for optics, image processing, and electronic

devices over many years. These technologies are integrated and can provide new

value in the areas of machine vision. Ricoh has already established a prominent

position in factory automation, (FA), automotive systems, security, distribution, and

more.

The concept is described in Ricoh’s Machine Vision white paper. That white paper

reveals Ricoh’s commitment to establishing “an intelligent technology that enables

machines to not only operate according to instructions from people but also to

quickly grasp the situation on behalf of a person and take appropriate action.”

Ricoh’s machine vision facilitates the visualization of things that are not otherwise

Input Output Processing

Collecting information by optimal devices

Analyzing and processing collected information and

converting it into intelligence

Providing results to systems and people in optimal form

Like sensory organs Like brain Like body response

http://www.ricoh.com/technology/whitepaper/pdf/wp_mv_en.pdf


visible or automates a process that otherwise must depend upon human effort. With

machine vision, Ricoh will allow people to take part in activities that will be

increasingly productive and have higher added value.

Our commitment will require a “smart” system that autonomously processes the

entire scope of capture, analytics, and visualization. For Ricoh, this is the second

stage of machine vision. Activities have already begun to enable a system to derive

optimal judgment and action from vision-based information and to present it clearly.

The system will repeat learning on its own, thus improving the accuracy of the

information it presents.

Primary element technologies of second stage machine vision

Capture Analytics Visualize

Stereo camera (including

ultrawide-angle)

Multi-spectral camera

Extended depth-of-field

camera

Polarized camera

Spherical camera

Image processing

Voice processing

Natural language

processing

Data mining

Electronic devices

Printing

Display

Autonomous control

3D modeling

Second stage machine vision is not just an alternative to the human eye. It will

enhance people’s value judgments and task setting, enabling advanced decision-

making and reforming the workplace into a space for the knowledge creation.

Activities in the second stage have already begun, covering a wide range of areas

including social infrastructure, FA, logistics, and security. This white paper describes

the status quo of Ricoh’s second stage machine vision, and the value produced

from these activities.

3. Ricoh’s Second Stage Machine Vision Activities

Ricoh believes that its activities in this second stage of machine vision will lead to

smarter workplaces. The second stage has a vast range of applications, enabling


solutions in a variety of fields. This document provides examples2 in the areas of

social infrastructure, FA and logistics.

3.1. Second stage machine vision in social infrastructure

・Road Surface Inspection System Based on Stereo Cameras and AI

Roads enable traffic of cars and pedestrians, which is an important role as part of

the infrastructure for social and economic activities and local lives. Yet road surfaces

suffer constant deterioration and damage due to external forces such as heavy

vehicle loads. If the roads are left unattended, the deterioration can result in

unwanted incidents, such as a car crash due to the poor surface condition affecting

the steering. To prevent such incidents, road surfaces must be inspected to find

deterioration at an early stage and provide necessary repairs accordingly.

Road surface conditions are evaluated according to three factors: the rate of cracks,

the depth of ruts, and flatness (lengthwise bumpiness). Generally, the inspection

requires a huge specialist vehicle provided with laser measuring instruments and

digital cameras. The specialist vehicle, however, is very costly—it is difficult to

increase their number, and thus to increase the inspection frequency or expand the

range. The vehicle cannot enter narrow roads, so inspection of community roads

has been deficient.

2 Some are under development.

Three factors of road surface conditions: rate of cracks, depth of ruts, and flatness


Ricoh’s Road Surface Inspection System features multiple stereo cameras; yet it is

compact enough to be mounted on a standard motor vehicle. Unlike the

conventional specialist vehicles, it can inspect community roads. The system allows

automation of the work processes from capturing images by stereo cameras to

creating inspection records, reducing the burden of inspection tasks.

The greatest feature of the system

is that the three factors of the road

surface conditions can be

inspected with only the stereo

cameras3. The rate of cracks, for

instance, can be automatically

determined. The cameras capture

the images of the road surfaces

and the system creates two-

dimensional images by combining

feature points.

The automatic determination is based on machine learning4, which is a type of AI.

The created two-dimensional images are processed on a 50-cm grid, and the crack

level (the number of cracks) is automatically determined by AI. A machine learning

model has replaced human sight in the visual determination process, significantly

improving efficiency and accuracy.

3 Click here to view a movie of the road surface inspection using stereo cameras. 4 Machine learning: Ricoh sets machine learning as one of the element development themes for enhancing image recognition technologies and has been studying it and developing its applications.

Road surface monitoring system installed on a standard motor vehicle

http://youtu.be/jff_kpCb3nI


Where tires pass, ruts are caused by friction on the paved surface, sinking of the

roadbed, and transformation of the paving material, for instance. To measure their

depths, Ricoh’s inspection system uses multiple stereo cameras to scan the road

surface, automatically combines the scanned images along the width, and reveals

the three-dimensional shape of the cross section of the road every 20 meters.

Flatness of the paved surface influences the comfort of the drivers and passengers

on traveling vehicles. Ricoh’s inspection system continuously measures three-

dimensional shapes in the forward direction by capturing images by the stereo

cameras and combining them. This eliminates the need for different measuring

instruments or devices for measurement. We need only have the car-mounted

stereo cameras scan the road surface and let the system measure the conditions.

After measurement, the three factors of the road surface conditions need to be

comprehensively evaluated to determine the need for repairs and the locations to

be given priority. The evaluation should be based on the Maintenance Control Index

(MCI). Ricoh’s inspection system calculates an MCI quickly as it can measure the

three factors on a single shooting drive.

The MCI is indispensable for the comprehensive evaluation of the three factors. In

the past, the management of community roads depended on visual measurement

because the specialist vehicles could not enter and numerical data such as the MCI

was not available. Ricoh’s inspection system solves this problem.

Determining the rate of cracks by AI


Ricoh’s inspection system visualizes the road surface conditions by mapping the

pavement conditions according to the measurement results. The mapped images

are colored according to the MCI values, allowing the need for repairs to be quickly

determined.

A demonstrative experiment of the inspection system started in June 2016 with

participation by the Ministry of Land, Infrastructure and Transport, Akita Prefecture,

Semboku City, and Ricoh. A general-use vehicle was used to measure the road

surface conditions for three kilometers of national, prefectural, and municipal roads

in Semboku City, Akita Prefecture. The measurement was done twice, before the

snow season (November 2016) and after the snow season (March 2017). The

inspection system compared well with the specialist vehicle regarding MCI values

for the three factors of road surface conditions. For the rate of cracks, high accuracy

comparable to visual determination was attained through machine learning of many

data samples.

In addition, the demonstrative experiment confirmed how the paved surfaces are

affected by snow, ice, snow clearing by snowplows, and snow-melting agents. The

Road surface condition mapping (example): Measured data is colored according to MCI

values.


experiment revealed the need to grab changes regularly over time, for instance,

and demonstrated the effectiveness of the measurement by a general-use vehicle.

The results of the demonstrative experiment and the issues found were reported to

the mayor of Semboku City on August 4, 2017, at the final debriefing meeting of the

Consortium for the Demonstrative Experiment of the Road Surface Condition

Inspection System*. Ricoh will continue to overcome technical issues and will strive

to develop practical inspection systems that are accurate and easy to use. Ricoh is

committed to implementing safe and secure road maintenance and management.

*Consortium for the Demonstrative Experiment of the Road Surface Condition

Inspection System: A joint project unit established by the Ministry of Land,

Infrastructure and Transport, Akita Prefecture, Semboku City, and Ricoh for the

demonstrative experiment of the road condition inspection system.

・Public infrastructure inspection systems using drones

The aging social infrastructure has recently become a major problem for Japan’s

national and local governments. Many of the highways, bridges, tunnels, and

buildings constructed in and after the high economic growth period (1960’s) have

exceeded their service life of 50 years and need maintenance and repair. The

Cabinet Office has been conducting a national project called the Strategic

Innovation Creation Program (SIP). Under the theme of Infrastructure Maintenance,

Renovation, and Management Technologies (supervised by NEDO: New Energy

and Industrial Technology Development Organization), the Cabinet Office is

promoting development of new technologies for the maintenance, renovation, and

management of aged infrastructure.

http://www.jst.go.jp/sip/dl/k07/kadai_en/k07-50.pdf

http://www.jst.go.jp/sip/dl/k07/kadai_en/k07-50.pdf


Supported by the program, Ricoh has been

developing bridge inspection system with drones5

that assist with close proximity visual inspection of

bridges and systems to develop inspection reports.

This has been done jointly with Tohoku University,

Chiyoda Engineering Consultants, Japan

AeroSpace Technology Foundation, and Tokyu

Construction. It is an attempt to reduce the burden

of inspection work on individuals; a camera is

mounted on a drone to photograph items that require inspection. According to a

2014 survey by the Road Bureau of the Ministry of Land, Infrastructure, Transport

and Tourism (MLIT), Japan has about 700,000 bridges longer than two meters, 20%

of which have been in service for more than 50 years. The percentage will continue

to rise. In July 2014, MLIT started to impose obligations on local governments to

inspect bridges every five years (MLIT Ordinance No. 39). In the implementation of

the ordinance, however, both the dramatic increase in workload as well as the

financial burden must be overcome.

Bridge maintenance and inspection processes are largely classified into field work

and office work. First, the inspection staff conduct visual inspections on-site, chalk

the damaged points, take pictures, and record findings in a field book. They then

return to the office, summarize the data according to the records in the field book,

and generate a detailed survey report on the damage, attaching photographs and

sketches. The workload is burdensome, and report results can vary depending on

the person.

Bridge inspections are often more difficult due to both location and structure. When

a bridge is difficult to access by an inspector, specialist inspection vehicles and

5 When a drone-mounted camera is used to inspect a bridge, the distance between the subject and the drone needs to be kept constant. The drone must enter a narrow space while avoiding beams and attached objects (such as cables, lights, and water pipes). So the drone is contained in a spherical frame about one meter in diameter; the frame comes in contact with the subject, keeping the drone at a distance and preventing direct collision. It was developed by Tadokoro Laboratory, Tohoku University.

Inspecting a bridge with a spherical-shell drone

http://www.ricoh.com/technology/institute/research/tech_inspection_system.html




large-scale scaffolds are often required. Besides the inspection staff, traffic control

staff may also be required depending on the situation. In these instances, it may

also be necessary to address the influence of traffic control on the lives and

economy of people in the surrounding neighborhood. Drones can be a solution to

this issue. Drones with a photography equipment mounted on them can reduce the

required number of people, cut costs, and ensure greater safety.

Additionally, Ricoh’s bridge inspection solution greatly reduces the workload in

office work. The latest image processing technologies help eliminate the

troublesome and time-consuming tasks of capturing data from damaged locations

and generating reports. From photographs (close-up images6), a panoramic view of

the entire bridge is automatically reconstructed in three dimensions (3D), and a

6 When inspecting a bridge, a spherical-shell drone performs a flyby across the inspection area to take close-up photos of all inspection points.

Bridge inspection processes


development image is generated to provide a bird’s-eye view of the points to be

inspected. The system makes it easy to determine (detect) the locations and

seriousness of any damage. In addition, Ricoh provides a document generation

support tool for summarizing the results in a report, conforming to the inspection

procedure recommended by MLIT. Automating the report generation process will

not only reduce the workload but also eliminate the variation in report quality,

reducing the number of man-hours required dramatically.

Beyond bridges, drones within a spherical shell can be safely used where

maintenance and inspection by individuals poses a risk e.g. public structures like

tunnels, super skyscrapers, and plant facilities. Ricoh will continue to develop the

automation of the report generation process, including the efficient identification of

damage that occurs in many different forms. Our goal is to provide a total solution

for infrastructure inspection.

・Autonomous flights of drones with a 3D vision system using an ultrawide-

angle stereo camera

Drones are expanding their applications in business areas thanks to rapid

improvements e.g. increased structural reliability, higher environmental resistance,

and enhanced navigation precision. As more and more drones are put into practical

use, people expect them to fly beyond the boundaries of conventional physical

restrictions. Demand has arisen for drones to undertake a stable flight outside the

Extracting damage points from photographed images and creating a survey report


scope of the operator’s view or without access to global positioning systems (GPS).

Currently, most business-use drones require operation by an expert operator.

However with the expansion of applications, operator shortage is anticipated. This

is another reason why autonomous flight capability is required.

GPS does not help in complex topography because errors in positioning can be in

the order of meters. When drones are used inside a building or tunnel, GPS signals

are unstable or unavailable in the first place. Even in an open space, GPS

navigation is unsuitable for flight due to electromagnetic interference.

Through joint research with the University of Tokyo and Blue Innovation, Ricoh has

solved these issues and enabled autonomous flights and the avoidance of

obstacles by drones based on a 3D vision system. Combined with an inertial

measurement unit (IMU7), our proprietary 3D vision system, equipped with an

ultrawide-angle stereo camera, allows a drone to avoid obstacles autonomously

and fly stably without an operator. Through coordination between a wide-view “eye”

that captures the surrounding conditions and the “semicircular canals” that produce

the sense of balance during flight, the drone autonomously flies in a non-GPS

environment, accurately tracing the path predetermined by the user.

For the autonomous flight system, the University of Tokyo has developed flight

controls, and Ricoh has developed the 3D vision system8. The drone knows where

it is using the ultrawide-angle stereo camera of the 3D vision system, measuring its

distance to the surrounding objects in a wide view and forecasting its movement in

reference to characteristic points and their relative distances. This technology has

enabled stable autonomous flights where GPS signals are unavailable. The wide

viewing angle of the camera allows the drone to keep the characteristic points in

sight even when its attitude changes, thus forecasting its own movement stably.

7 A sensor that detects angular velocity (gyro) and acceleration 8 RICOH 3D Vision System × Drone (Movie here)

http://www.ricoh.com/technology/institute/research/tech_flight_by_3d_vision.html

http://www.ricoh.com/technology/institute/research/tech_flight_by_3d_vision.html

http://youtu.be/GEV9fnEcqX8


Although the path is predetermined by the user, unexpected obstacles may exist on

the path. The 3D vision system generates a 3D map around the flight path

concurrently with movement forecast, so the drone can detect and automatically

avoid unexpected obstacles.

The 3D vision system is unique in that it handles a series of processes in real time9

inside the camera unit, from photography through to positioning forecasting and

generating 3D maps. This is a new technology produced by the fusion of excellent

optical design technology, advanced image processing technology, and elaborate

systems control.

Ricoh’s 3D vision system allows drones to be used without an operator both indoors

and outdoors. It autonomously avoids obstacles whilst flying. Ricoh will continue to

enhance the smartness of the 3D vision system, expanding the possibilities of drone

applications.

9 Fast processing is attained by implementing multiple processes on an FPGA (Field-Programmable Gate Array) package, an integrated circuit configurable to specific design objectives. The processes include image processing, such as characteristic point extraction, and parallax operation for the stereo camera.

Autonomous obstacle avoidance based on 3D vision system


3.2. Second-stage machine vision in Factory Automation (FA)

・Automatic random picking system using a stereo camera

Ricoh has developed a fast and precise stereo camera primarily for industrial

applications. A stereo camera can capture the depth of a subject. Developing it

requires advanced calibration technology, parallax operation technology 10 ,

precision parts mounting technology, and more. Quality requirements are acute in

industrial fields, and developing a module for them requires particularly high skills

and ample experience.

Since the latter half of the 1970’s, Ricoh has been enhancing automation of its own

production facilities using FA and accumulating technical know-how in this area. We

have shared some of the results outside the Company e.g. machine vision, and

have collaborated with other FA manufacturers. Ricoh has been contributing to the

improvement of office productivity (Office Automation - OA) with multi-function

printers (MFPs) and single-function printers and, at the same time, has attained

many achievements in FA.

FA is prevalent in manufacturing, but some processes still depend on human labor.

Random picking is one of them. Parts that are randomly placed on a rack need to

be individually transferred to a part feeder or an assembly line. Their conditions

must be quickly determined while they are on the rack so that they can be picked

up efficiently and properly arranged.

Conventional automatic picking systems used multiple devices and robots, resulting

in poor space efficiency and high cost. Many of their cameras were 2D—they had

a low recognition rate and were not reliable enough to replace human operators.

10 Calibration is the adjustment for detecting and measuring subjects to be photographed, and parallax operation is the processing to recognize the positions and shapes of subjects to be photographed.

http://www.ricoh.com/technology/tech/045_stereocamera.html


Then came a projector-based 3D vision system, but it was too large and slow11 to

readily fit on production lines.

The picking systems based on the RICOH SV-M-S1 industrial-use stereo camera,

is the solution to this difficult issue. With the stereo camera capturing the subject in

real time in 3D, the picking systems can quickly recognize the shapes and

orientation of parts that conventional cameras cannot. Collaborating with a robot

manufacturer that has worked with us before, Ricoh has developed the picking robot

into a system that maximizes the use of the fast and precise stereo camera. Before

implementation, Ricoh Industrial Solutions, an experienced FA systems integrator

within the Ricoh Group, can now propose an effective system from the viewpoint of

the optimization of the whole process in collaboration with the customer. The system

has already been introduced to the production lines of multiple customers,

contributing to improved production efficiency.

At this dawn of the IoT age, the functions

required for the picking system are about to

change. The movement for PLM (Product

Lifecycle Management), a concept for

centrally managing the entire lifecycle of a

product, has become active internationally.

The conventional system based on batch

processing on the part feeder is incapable of

answering the needs sufficiently. Ricoh’s random picking system uses a stereo

camera to recognize individual parts, and respond to such requirements flexibly. By

developing “smart” system technologies that solve the issues of FA and providing

customer-oriented solutions, Ricoh will continue to contribute to production

innovation.

11 The projector-based system uses the phase shift method for 3D measurement, which requires extra processes: pattern projection on the projector and dot data processing on the camera.

Robot picking system

http://industry.ricoh.com/en/fa_camera_lens/sv-m-s1/

http://www.rins.ricoh.co.jp/en/


・Image recognition and analysis technology for visual inspection based

on machine learning

The parts inspection process is considered the “last hurdle” in implementing FA in

production processes. Thanks to the recent improvements in photography

technology and image recognition technology, the applications of machine-vision-

based visual inspection systems have expanded. Nevertheless, their capabilities

have not been fully exploited. The delay is the difficulty in telling the difference

between good and defective products. In addition to damage and flaws, many

factors are included in making a comprehensive judgment based on a variety of

conditions. When the boundary between the good and defective products is

ambiguous, the judgment must be based on knowledge and experience.

Still, results tend to vary when inspections are based on human labor. Depending

on the parts, inspection tasks can burden the inspection staff (excessive use of

eyesight and transfer of heavy objects). Automation of the inspection processes is

a must in improving production efficiency, reducing costs, and effectively using staff

(reassigning people to higher added value tasks). To provide a solution in this area,

Ricoh has been conducting research and development into Image Recognition and

Analysis Technology for Visual Inspection based on machine learning.

Ricoh has been studying machine learning as an element technology for image

recognition. One result is anomaly detection based on a semi-supervised learning

method. This method lets the machine learn correct values (labels) only, so that

anything that contains any other data is detected as an anomaly. Accuracy

increases with the iteration of judgment (iterative learning), and the data quantity

(the number of sample images) can be smaller than the generally used supervised

learning (based on data sets in which all data items are labeled).

Semi-Supervised learning can be enhanced by applying image processing

http://www.ricoh.com/technology/tech/073_imagerecognition.html

http://www.ricoh.com/technology/tech/073_imagerecognition.html


algorithms12 that calculate characteristic quantities from the photographed data of

the parts. Using the enhanced method, Ricoh is developing a precision visual

inspection system that automatically detects defective products. Our ample know-

how in image recognition contributes to the improved precision of machine learning.

In general, defective parts are far fewer than good parts, so it is unrealistic to

prepare learning samples of all defect types. Thus, the semi-supervised learning

method is suitable because it is based only on the characteristic data of good

products. This method can detect unknown defects, and has high precision in

inspecting parts where the shapes of good products vary greatly (parts in the

preliminary processing stage, for instance).

The image recognition and analysis technology for visual inspection based on semi-

supervised learning received an award for excellence over two consecutive years

(2014 and 2015) at the Visual Inspection

Algorithm Contest, sponsored by the

Technical Committee on Industrial

Application of Image Processing and the

Japan Society for Precision Engineering.

Machine learning can be applied to many

fields. Its value is increasing at this dawn

of the Big Data age. Ricoh will continue to

improve the learning precision and

12 SIFT (Scale-Invariant Feature Transform) to extract characteristic points from images (boundaries and edges) and SURF (Speeded-Up Robust Features) to calculate characteristic quantities (information robust against rotation, scale, and illumination)

Function blocks of visual inspection system

Sample detection result

(Visual Inspection Algorithm Contest 2014)


processing speed, aiming to take image recognition and analysis technology to an

advanced level that compares to human judgment.

3.3. Second stage machine vision in logistics

・Flow monitoring using stereo camera

The advancement in machine vision has enabled mass image information to be

used in a variety of fields, but how to obtain useful information is another problem.

It can be solved by the visualization technology, which analyzes the mass

information as necessary and presents the results as visual information. Particularly,

the technology to visualize dynamic information in real time assists quick decision-

making, helping people discover things that are difficult to recognize in the visual

inspection of objects or the investigation of static images.

In a warehouse, for instance, there is a need to observe the motion (flow) of people

after changing the shelf layout to see if the change was appropriate. Ricoh has

developed a flow monitoring system using stereo cameras, and uses it to optimize

the layout in the logistics warehouse of Ricoh Europe SCM B.V. (Bergen op Zoom,

The Netherlands).

Visualizing the flow in a warehouse (heat map)


The flow monitoring system has multiple stereo cameras to monitor the conditions

of the transport work in the warehouse in 3D in real time, and displays the results

visually. The information on the workflow—where people or loads tend to

concentrate, for instance—is classified and shown in shades of different colors,

allowing people to grasp the situation at a glance. The obtained information can be

analyzed from a range of viewpoints to figure out a layout that maximizes efficiency

and minimizes workload.

In the past, flow monitoring was done as fixed-point observation by a video camera.

The fixed-point method required a long time to review, and things tended to be

overlooked when the recording was fast-forwarded. The stereo cameras featured

in Ricoh’s flow monitoring system precisely capture and visualize the information,

such as positions and flows, in 3D, enabling quick and appropriate decision-making.

The visualized images can be analyzed from multiple angles under different

conditions as necessary to determine the optimal layout.

The application of the flow monitoring system is not limited to warehouses. In an

office, for instance, it can be used in an ecology system for controlling energy

according to the flow of people. It can also be used in a security system that issues

an alarm upon detection of an anomaly in the incoming and outgoing flows of people.

Ricoh will continue to further expand the applications, developing the monitoring

system into a “smart” decision-making support system with excellent capabilities,

including automatic analysis of the results obtained from monitoring.

4. Tackling Second Stage Machine Vision with Open

Innovation

This document has introduced Ricoh’s products in the second stage of machine

vision, including those under development. With each one, the developers look

directly at how to increase the intelligent productivity of people, rather than merely

seeking automation or labor-saving. Ricoh’s attempts to make “smart” systems has


just begun. Performance is being improved steadily through on-site verification and

other efforts. Ricoh has accumulated numerous technologies over many years, not

to mention the optical, image processing, and precision mounting technologies, and

they are about to bring a shift in the status quo as a destructive innovation13.

Yet Ricoh will not conduct these efforts on its own. As seen in the examples

introduced in this document, many of our development tasks are conducted through

collaboration with research facilities, with multiple companies, and even with our

customers. We are striving to create new value by bringing about innovation through

the fusion between Ricoh’s proprietary technologies and know-how and the

technologies, knowledge, and know-how of our partners. Keep your eyes open for

Ricoh’s second stage machine vision, which is based on open innovation to

establish smarter workplaces.

13 Using innovative technology to bring dramatic changes to the existing business models and industrial structures


Revision history

Ver 1.0.0 2017/11/20 First edition

Documents

Ricoh’s Second Stage Machine Vision: EMPOWERING DIGITAL ... · Ricoh’s Second Stage Machine Vision: EMPOWERING DIGITAL WORKPLACES Ricoh has been providing automation and other