Machine Vision Technology : Past, Present, and Future

Masakazu Ejiri

Central Research Laboratory, Hitacki, Ltd. Kokubunji, Tokyo 185, Japan

Abstract . This paper discusses the past, present, and

future of machine vision technology, emphasizing its practical aspects. Four types of machine vision technology based on pattern matching, feature parameter, window, and slit- light methods are now being widely used in industry. However, recent applications require more effective use of knowledge-based processing, combined with application-specific methods. Study is fius required for the future progress of machine vision, especially in such areas as sensor fusion, real-time processing, and visualization/enhancement technology. Some new future applications are also discussed.

c

1 Introduction

Japanese industry has been taking the initiative in creating the "machine vision" field and has played a major role in its advancement since its inception. In the past 20 years, a large number of industrial processes have been automated using machine vision, most prominently in the semiconductor industry where production places heavy demands on precision and efficiency. However, many processes remain unchanged due to the difficulties in reliable recognition.

Although "computer vision" research has also advanced greatly in that time, it is rather methodology-oriented, and unfortunately does not suggest pragmatic solutions to difficult application problems. m a t is, research and development seems to be stagnant from the application point of view. To look at the future of machine vision research, it is worthwhile to look back to what we have done, to understand where we are now, and to think about what direction we have to move in.

2 Brief History of Machine Vision

The first vision-based intelligent robots in Japan appeared in 1970, one from the Electro-technical Laboratory of MITI, and the other from the Central Research Laboratory of Hitachi Ltd. Each was controlled by a minicomputer whose capability was far less than today's microcomputers. The configuration of the Hitachi robot is shown in Figure 1. [1,2] This robot is typical of the "intelligent machines", as shown in Figure 2. The outstanding - feajure of this robot was a macroscopic instruction, in this case assembly drawings. This enables the robot to be controlled without any further microscopic instructions corresponding to each robot movement. The concept of the intelligent robot is still alive. [3,4]

Since these developments, considerable work has been done in Japan, especially for industrial applications, and unique, basic, and applied research on industrial machine vision systems has been carried out. The first successful application of image

---- tor

,Objects

Figure 1. An intelligent robot that assembles objects from plan drawings

processing to industrial automation was the defect- inspection machine for printed circuit boards in 1972. [ 5 ] In 1973, an automatic wire-bonding system with time-sharing vision was developed for transistor assembly. 161 This was later extended to the development of systems for automatic assembly of ICs and LSIs. A robot for bolting molds for concrete piles and poles, developed in 1974, was the first application of the dynamic recognition technique of moving objects. [7] Through these pioneering studies, the importance of vision techniques is now recognized in many sectors. In particular, the successful developments of a local pattern matching method [6] and the time-sharing vision systems based on it made a great contribution to the semiconductor industry. At the same time, it encouraged researchers towards further development of machine vision technology.

In the computer vision fields at that time, researchers were struggling with the more general methods of vision using imaging devices looking obliquely at the object. A research paper that showed the possibility of categorizing objects using a vertically-set imaging device did not get much attention at first. [8] However, since the SRI algorithm based on the feature parameter method [9] was proposed, vertical observation gradually obtained much attention as a simple but effective method for industrial object-recognition. This is now a standard vision method for industrial applications.

As for the hardware for a machine vision system, a costly minicomputer was first used in combination with a special-purpose image processor. To lower the cost/performance ratio, a large-scale manufacturing system with 50 transistor assembly machines was developed, where one minicomputer is shared by five image processors, each servicing ten cameras attached to the machines. [6] As the availability of microcomputers increased, stand-alone

o+fJr ;o ronment

Environment understanding

Intelligent intedace Visual, auditory, tactile senson

Figure 2. A model of intelligent machines

Monocular Pattem matching Feature parameter Window

Passive

Spot I i g ti t Shape-from-shading Slit-light Photometric stereo Grid-light S hape-from-texture

i Time-of-flizht

Figure 3 . Methods for machine vision

assembly machines became dominant, combining a microcomputer and an image processor. Nowadays, these machines are again being connected with each other, through a LAN system, to form an integrated manufacturing system with distributed intelligence.

Machine vision was first implemented as application-specific, special-purpose vision systems using binary image processing techniques based on local-parallel image processors. However, as the availability of image-processing LSIs increased, gray-scale image processors became a cost-effective reality, and are now being effectively applied in industry.

3 The state-of-the-art of Machine Vision

3.1 Outline Many types of machine vision technology have

been developed to date as summarized in Figure 3. The passive technology in 3-D vision typically includes binocular vision and the "shape-from-X" method where X implies such words as shading, texture, and motion. However, these methods have not yet been fully applied to industrial use due mainly to restrictions on processing speed and reliability. Instead, the practical machine vision systems commonly used in industry are based on a few simplified, fundamental technologies. These are passive, monocular, 2-D vision technologies:

* pattern matching method, * feature parameter method, and * window method.

The active technology typically uses a structured light method or a time-of-flight method to give the distance between the object and imaging device. One useful structured light method is:

* slit-light method, which is based on the triangulation principle.

These four methods are effective in industrial

xxx

applications, on their own or in combination. However, the technology level attained so far in machine vision applications is still low and can be summarized as follows:

Recognition of geometric features, supplementary features, and a few restricted cases of qualitative features of simply-shaped objects. These objects must be within a predicted area or positioned in a predetermined area. Usually, the illumination is carefully controlled.

At present, machine vision techniques cannot be applied to:

Recognition of most of the qualitative features of even simply-shaped objects, and the geometric and supplementary features of complex objects, especially when the illumination changes drastically such as in outdoor, day-and-night use in all-weather conditions.

The feature parameter method and the window method are flexible enough for general-purpose vision systems, while the others are mainly used in special-purpose systems. More than 300 general- purpose vision models have been developed in Japan, bved roughly on these two methods. However, only 100 models survive nowadays in terms of active sales, although the business is still small and progressing slowly. It is estimated that more than 7,500 machine vision systems are being successfully operated in Japanese industry. The main uses of machine vision arc in the assembly field where part identification and determining position/orientation are the main areas. Machine vision is also used in the inspection field where pattem alignment, dimension measurement, and defect detection are the main areas.

One recent trend in machine vision is to improve machine vision systems using the knowledge of objects and environment (mainly factual knowledge), and the knowledge of the methods of image processing and decision-makiig (mainly rules). These attempts focus on the effective use of knowledge to automatically generate optimal procedures in pre- processing, feature extraction and structural analysis for a given application. The following two features are critical [lo];

(1) Knowledge separation, where the knowledge is stored separately with the processing procedure. Thus, functions can be altered by replacing the knowledge. This also pro'vides a basic framework for automatic knowledge acquisition.

.

(2) Knowledge or model driven, where the results from pre-processing, feature extraction, and structural analysis are compared to the knowledge and the decisions are fed back to the preceding processes. Flexible processing thus becomes possible. The knowledge can take various forms such as explicitly- represented if-then rules and models, or implicit forms that can be derived from these explicit representations.

In industrial applications, however, a simpler s-ystem will be more cost-effective, faster and reliable. In past applications, object types and imaging methods have been somewhat restricted to meet these cost, speed, and reliability conditions. For example, recognition of complex 3-D objects is simplified to a 2-D recognition problem by constraining a certain object surface to a standard base plane., Then, a number of algorithms are developed, and the best is selected and implemented into a procedural high- speed image processor. Thus, "image understanding" technology based on knowledge has seldom been used in attual industrial applications, although it has been useful in simulations in algorithm development. One reason is that knowledge-based processing is still slow. Another is that feedback from the matching process may disturb the rhythm of the manufacturing process.

However, the importance of knowledge separation from processing procedures, and of the knowledge-driven or model-driven techniques for controlling the image processing is gradually increasing, leading to more general and more expansible machine vision. The following sections briefly discuss the four basic machine vision methods from the viewpoint of their knowledge-related aspects. A minimal framework for knowledge separation and a knowledge-driven feature can be seen in these methods.

3.2 Pattem matching method

A typical example of this method is the recognition of characters on the object surface, where the unknown input image is directly compared with standard pattems or templates for each character. These templates can be regarded as factual knowledge of characters, and thus separation of the knowledge from the procedure is in common use. Another example of pattem matching is the recognition of the position of complex objects such as transistors, ICs, and =Is, where a few local pattems are extracted in advance from an object pattem as standard pattems,

XXXI

and are used to detect the best matching positions in the input image. [6] This method is shown in Figure 4. The distances and angles between the detected positions are then verified. Thus, the knowledge used in this method is the local pattems themselves, and the relationships between them. By changing the knowledge,, the system can be adapted to various objects. When the geometric check using distances and angles is verified, the positions for subsequent wire-bonding are calculated. The local patterns should be unique to each other in this method, and the selection of the local patterns from a given object image has also been automated. [ 111 This automatic selection can be regarded as a means of automatic knowledge acquisition.

Emitter

Temp1 a 1c pal terns

Figure 4. Local pattem matching method

Res11 I t,s

I L i

1-1 ' ---Feature gr id pattern ( f o r e n t i r e imagp) corner patterns 1 2 4 8

Figure 5. Structural matching method for feature pattems

Instead of complex local pattems having unique shapes, unique combinations of much simpler pattems in the image can be used to find the position of the object. [12]. Figure 5 shows such a method. Positional relation among simple local patterns (such as comer patterns) distributed in an entire image is represented as a feature grid pattern consisting of their pattern codes. Thus, structural matching of pattems can also be implemented by conventional pattern matching hardware. The knowledge used is the feature grid pattern that represents the structure of the standard object pattem from which an inputting feature grid pattern is searched. The exchange of knowledge enables the system to be adapted to new objects.

3.3 Feature parameter method

This method extracts several geometric features from an object pattern. The features include the area, peripheral length, number of holes, and moments. Thus the object can be represented as a point in n- dimensional feature space, and compared with the standard regions of each object to find the closest object category. Standard regions can be taught by a teaching-by-showing method. Thus, the knowledge in this method is a combination of pattern features of the objects.

Another method uses a decision tree derived from the measured parameter distributions of each object in a teaching process. The feature parameter that distinguishes the distributions most is the first parameter to be examined, as shown in Figure 6. Thus, the method divides the problem into sub- decision problems, and generates a decision tree

Decision t r e e I

Parameter x ,

Figure 6. Decision-tree type feature parameter method

XXXE

automatically. This generation is one means of automatic knowledge acquisition, and the tree is secondary knowledge derived from the primary knowledge of the object parameter distributions.

One recent application of this method using a gray-level image is to a defect-detection machine for light-emitting diodes. [ 13 J The projection distributions of brightness in two orthogonal directions are calculated from a gray-level image of the radiation pattern of the diodes. The Fourier coefficients of the distributions are then used as feature parameters for finding small brightness irregularities. The empirical decision criteria of a skilled human inspector can be learnt by showing normal diodes to the system, and can be installed as a parameter space distribution. The decision is made in terms of Mahalanobis distances to the distribution of normal diodes.

General-purpose machine vision systems based on this method are already widely available, and are mainly used for object discrimination in various industrial processes.

3.4 Window method

Observation of selected portions of an objett image can be used to recognize object category, position, and orientation. [3] Object areas in the windows are the basis for the decision when the windows are appropriately placed in the image field. Some variations of the window method are &own in Figure 7. Relative positions and window sizes can be varied to search dynamically for different objects. Thus, the knowledge used in this method is the sizes and shapes of the windows, the pattern features (mostly areas) in the windows, and the relative positions between windows.

Such a machine vision system detects an object's comer point, center point, or an edge line from each window as shown in Figure 8, and then combines these primary points and lines to give compound features such as distances, angles, and cross-points. [I41 These are used to find the object category, position, and orientation. This vision system has been implemented as a general-purpose system, and is being applied effectively in such fields as VCR assembly, as shown in Figure 9.

3.5 Slit-light method

This method projects a slitqight beam from one direction and observes the object's reflection from another direction. Triangulation gives the distance between the object and the observation point. One

n

(a) Position finding ( b ) Timing finding

(c) Shape discrimination

Figure 7. Window method

Center point Corner point Edge line

(a) Primitive features

1 . I ,

Point v s . point Line vs. point Line vs. line

( b ) Compound features

Figure 8. Features used in a window method

Figure 9. An assembly cell for VCRs

xxxm

typical application of this method is a sealing robot with a slit-light projector and an image sensor on its wrist. A slit-light beam is projected vertically to a seam between two overlapping plates. The image obtained from an oblique direction contains a kink in the slit-light beam. The position of the kink in the image is kept centered by moving the arm, and sealant is applied to the seam through a nozzle. This gives continuous sealing of the curved seam. [ 151

Other applications include inspecting soldering between LSI leads and printed boards. Profiles of the leads are detected by the combination of a scanning spotlight and a linear sensor, and are used for classification into normal, non-contact, mis-aligned, or bridged leads. [ 161

This method is also used for inspecting assembled printed boards. Slit-light beams are projected individually onto each component on the board, and the heights and widths are checked to reveal any mis-placed or mis-oriented components. Conventional gray image analysis can be used in combination with this method to check the polarity of capacitors and the component names by reading the surface marks. [ 171

3.6 Design pattem referring method

In addition to these four conventional methods, various application-oriented, special-purpose techniques have been developed. One recent technique is the inspection of complicated patterns. Until recently, a two-chip comparison method has been widely used to find abnormalities in complex binary patterns, for example, for mask inspection in semiconductor manufacture. However, this method is becoming inadequate for application to recent high-. density pattems, particularly to gray-level images of semiconductor wafers, and a design pattem referring method is becoming more favored. In these cases, pipelined image processing is necessary for providing high-speed abnormality detection. The knowledge to be implemented in the pipeline image processor includes:

(1) pattems to be inspected, (2) substances that form the pattems, and (3) abnormality definition.

These can be encompassed in a generic form as: If a pixel (or a pixel within a certain blob) is at a certain position, and if the feature value(s) of the pixel (or blob) is within a certain parameter range, then the pixel is regarded as a decision and is processed by a certain operation.

processing u n i t ( I ' U )

Figure 10. Basic configuration of knowledge- directed process (generic form)

D r f c c t Fat.al Input randida te Defect dcfert image image image d a t a

Figure 11. Basic pipeline architecture for defect identification

An instance of this generic knowledge is: If a pixel is on a silicon-oxide layer, and if the brightness of the pixel is outside the range Ti to T2, then the pixel is a part of a defect candidate, and is flagged.

A processing unit that uses this type of knowledge can be implemented in hardware, as shown in Figure 10. In this configuration, an input image and a number of design patterns are simultaneously fed into a processing element and a control element respectively. The design pattems are generated from the design data in such a way that they are accurately aligned with the input image. This alignment is executed by a position calibration circuit, prior to their application to the control element. The control element outputs different control parameters depending upon the position being scanned, and the parameters control the processing element. Several processing units, each of which is the combination of a control element and a processing element, can be connected serially or in parallel to form a pipeline- structured inspection machine. Figure 11 shows the basic architecture of such a machine. A wafer

X X X N

, I I I.ighl :: De l e c t Cali- Dcsign

detect.ion c bration pa t te rn c i r c u i t generator

1

c i r c u i t

'Table I

Design da ta

I

CAD sysien (Nain f r a w compri k r )

Figure 12. Wafer inspection machine (for logic ICs)

inspection machine for logic ICs based on this principle is shown in Figure 12. [18,19] A wafer inspection machine for high-density memories has also been developed based on the same principle.[20]

In the candidate extraction stage of the defect detection circuit in these machines, an optimal defect discrimination procedure is selected for each layer depending upon the pixel position currently being scanned. In the defect judgment stage, the design patterns and the input pattern are processed by the same procedure, and the results are compared. One example of such a procedure is illustrated in Figure 13, where pattem-widths are measured using a distance-transform filter for checking defect seuerity. This means that the original design patterns are modified to generate new knowledge for comparison.

Design pa t te rn Input patt.ern

1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 r-l 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 l==A 5 5 5 5 5 5 5 5 5

r"-1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 2 3 3 3 3

5 5 3 9 3 5 5 5 1 4 4 f i 4 4 4 1

Corn par ison 9- Figure 13. Pattern-width measurement for defect

fatality based on distance-transform

The circuit for this operation can also be represented by the generic form shown before in Figure 10.

The calibration circuit is used to measure both the positional error and the pattern-width error between the input pattem and design patterns, using a real-time correlation calculation. The results are fed back in real-time to the timing of design-pattern generation and to the dilation/erosion control of generated pattems, to ensure precisely-registered design gatterns. This calibration circuit further modifies control parameters such as Ti and T2 used in the candidate extraction and defect identification stages, by periodically measuring the frequency distribution of brightness for each layer portion of the pattem. Thus, this circuit serves as a mechanism for modifying the knowledge.

The wafer inspection machines based on this method use a model, i.e. design patterns, as the pattern for comparison, as the signal for controlling the process, as the basic knowledge for deriving secondary knowledge, and as the original standard knowledge for modifying itself to adapt to the changed condition. Thus the method is highly reliable and gives real-time high-density pattern inspection, despite its being knowledge-driven.

4 Future Machine Vision

4.1 Aspects of machine vision research

To achieve higher-level machine vision for a wide variety of applications, it is necessary to further research on computer vision and image processing that form the basis of practical machine vision technology. However, the research in these fields seems to have the following aspects, which can easily lead research in the wrong direction.

(1) Easy-to-start Anyone can start research in this area quite

easily. The analysis of only one image taken under a certain -- usually unspecified -- condition easily satisfies the researcher, due simply to the fact that the result can be a beautiful picture apparently different from the original input image. This illusion may lead one to view the research as quite successful, but it is still superficial, lacking the depth, durability, and adaptability essential for practical applications.

(2) Difficult-to-evaluate Judging the effectiveness of this technology is in

general very difficult, as the heart of the technology is

xxxv

an indefinite algorithm. The benefits of the algorithm can only be evaluated quantitatively by actually applying it. This is one reason why technology transfer to the users is difficult to achieve.

(3) Difficult-to-accumulate Different assumptions are made in every

research paper. Not all of them are practical, making technology accumulation difficult. The processing speed problem is usually ignored, with the assumption that it will be solved in the future as semiconductor technology progresses. This may be partially true. However, there is also the possibility that vision problems currently being studied will be worthless because they will be solved easily by other means before the necessary processing speed is attained.

4.2 Important technologies in machine vision

Bearing these aspects in mind, continuous efforts are needed to develop and accumulate algorithms and apply them to various fields. In particular, the following issues will be increasingly important in future machine vision.

(1) Sensor fusion Many products with different specifications will

be needed to meet the increasingly diverse needs of individuals. Therefore, the following techniques are becoming more important in manufacturing research. PI1

* Product realization * Intelligent manufacturing control

The former is a design technique starting from the conceptualization of a product based on human needs, and finishing with a product design through quick prototyping and evaluation, where the intelligent use of computers will be the key to success. The latter is a control technique at the time that the products are manufactured. The manufacturing system will thus be configured as in Figure 14, where the flow of information as well as of objects must be controlled intelligently. Thus, machine vision will be important for detecting the object and machine conditions.

The data sensed by these vision systems are then integrated to keep the manufacturing system operating optimally, Thus, the importance of sensor fusion technology will incfease. However, the method of integrating such sensed data is still unclear. One possible way to obtain effective sensor fusion technology will be through neural networks.

L V U V V V I Ima ,+I e sensors m n s o r s -

In-process conhol 1 I (adaptive and predictive)

+ + ' t t i t

Real

Knowledge- base, model-base

Object flow Information flow Machine condition System System condition configuration

External dormation

Figure 14. Intelligent manufacturing control system

(2) Inspection for varying specifications In the manufacturing process, at least two types

of machine vision system will be required: one to continuously check for maximum efficiency of operation in each manufacturing machine; the other to check that each product is being correctly manufactured. This inspection function must be distributed to each important stage of the manufacturing process. In the 1990s, mainstream machine vision applications will thus change from assembly to inspection. In particular, inspection techniques that can adapt to a group of products made under different specifications will be crucial.

The outputs from the sensors can directly control the objects being processed and the machines and equipment used in manufacture. This is a means of in-process adaptive control. If a machine vision finds a slight erroddeviation in the object shape being processed, this information must be fed to subsequent processes. Thus, the downstream processes must be modified to adapt to the object before it becomes seriously defective. This method is called predictive control. Feed-forward predictive control will be an important feature in advanced CIM (computer integrated manufacturing) from the viewpoint of no defective products.

(3) One-pass real-time processing Future applications will demand real-time image

processing, as the number of operations required for recognition is growing greatly. It is obvious, from the technological trends in semiconductors as will be discussed later, that adequate image memory capacity is obtainable, even for extremely large images. However, it is usually a waste of time to first store the

X X X B

image into an image memory and then process i t by accessing the image memory. Especially in industrial applications, it is necessary to develop a fast method in which the completion of image acquisition implies the completion of the image processing (or at least pre-processing).

In the wafer inspection machine described earlier, a control image is first generated from the defect candidate image, as shown in Figure 15. In this control image, all holes and downward dents are filled by a recursive, propagation-type distance- transform filter. Therefore, when the two scan lines are observed, it is possible to decide whether the

llead Tai l Body-1 Body-2 . 4

Figure 15. One-pass feature extraction method

S t a r t

from buffer I , I I

Tai 1 Body Head

Figure 16. A sequential machine for one-pass feature extraction

observed blob is the head portion, the intermediate portion, or the tail portion of an object figure. Thus, by controlling the state-transitions of a sequential machine (shown in Figure 16) in response to the two- bit signals (numbered 0, 1, 2, and 3) observed at the vertical position in the two adjacent scan lines, features of each figure can be accumulated until the scan reaches the end. Thus, feature parameters such as area, peripheral length, and projected length can be found for each figure after one vertical scan. [20]

'?his one-pass real-time processing technique can be applied to the usual defect detection and particle analysis problems. However, if the image is a mixture of extremely complicated pattems, this method will not be applicable because the figures would be fused when the control image is produced. An improved method has thus been proposed, and its strictness has been proven theoretically for any complicatedly- patterned figure arrangements. [22] This improved method can measure the object features of such a pattem as shown in Figure 17, which has two object pattems. The feature values are output at the end of every scan of a figure (at a point @ in the figure).

As shown in these examples, raster-scan, one- pass real-time processing techniques based on on-site processing will be increasingly important in the - future. Extensive study will be needed on the wide- range of real-time processing algorithms.

Scan1 iiie Internal number s t a t e

PFl(123 27 5 28 11 0 0 0 0) PFZ(256 30 3 31 2 0 0 0 0 )

Detected pattern features

N I L (SSSS) (LRSLR) (SLRLRLR)

(LSSflR) (LSSHR) (LHflflflR)

NIL

Figure 17. One-pass feature extraction from a complex image (Contains two figures with 4-neighbor connection mode)

xxm

(4) Visualization/enhancement technique Vision systems for complicated 3-D objects,

objects with much more complicated surface pattems, and hard-to-see objects will also be needed. The matching of distorted image signals in SEM images [23] is a typical example of recent approaches. Texture analysis is another. A self-organizing image filter for texture separation [24] has been demonstrated successfully in this area. A knowledge- based inspection of complicated LSI pattems [ 191 is also an example of recent approaches to the recognition of hard-to-see objects.

Recognition of invisible objects is also becoming increasingly important. One example is the inspection of soldering of LSIs surface-mounted on a circuit board, where the LSI leads are completely hidden under the LSI package. For such uses, visualization techniques and image enhancement techniques are the keys to reliable vision systems. Obviously, the use of X-rays is one possibility for visualization of hidden defects. Thus, sensors must be improved by combining them with image processing techniques.

-

-

- -

-

-

4.3 Trends of related technologies

lo6 - "E

lo5 2

lo4

10'

IO2

- 1 0 5 - 1 F - o

10~'

3

0

m U

The fundamental technologies in future machine vision will be:

* processor technology, * memory technology, and * sensor technology.

All of these are core areas in semiconductor technology. Trends in these technologies will be briefly discussed here.

Figure 18. Trend of processor technology

10 1- E m Y) W " 4" - 100;

E E .- N

v

W

-0 @ l G - E 10

- 10 E

- 1 ;

5 - IO2 2 - 1 0 ' 2

L 0

- IO' -

ui Y 0

1980 1985 1990 1995 2000

Figure 19. Trend of memory technology

(1) Processor and memory technology Recent progress in semiconductor technology

has seen certain trends, as shown in Figures 18 and 19. In the late 1990s or early 2000s, we can expect a 200 MIPS microprocessor (CISC-type) and a 1 Giga bit memory chip if lithography limits are expanded and other difficulties can be overcome in each subsequent generation. This means that a memory capacity equivalent to the human brain is attainable. This also indicates that the present supercomputer may eventually become a palm-top personal supercomputer. Therefore, optimistically speaking, almost all the hardware problems conceivable today could be solved by the end of this decade. Most problems exist in software/algorithms and sensors. As for the algorithms; the robustness and the resulting decision reliability are keys to success.

(2) Passive image sensors As for the sensors, sensitivity, resolutiorl, and

dynamic operation range are the major concerns for practical applications. For passive image sensors, the image processing researchers have had to contend with illumination problems in every application. Current MOS and CCD image sensors, and their successors, will not be suitable. A new image sensor with an extremely wide dynamic range must be developed. The lack of dynamic range in current sensors is making image processing extremely difficult. If a wide dynamic-range image sensor were available, a lot of the pre-processing algorithms in image analysis would not be necessary.

From a sensitivity standpoint, the MCP-CCD, a charge-coupled device combined with a micro- channel plate, shows promise for detecting subtle image signals. Avalanche-multiplication image tubes called HARPICON [25] and super-HARPICON, developed recently for HDTV broadcasting, may also

x x m

become feasible imaging devices. These devices have a wider dynamic range and an extremely high sensitivity, allowing a high-resolution color image at less than 1 -lux illumination. Developing these as solid-state imaging devices could revolutionize machine vision applications.

1'3) Active image sensors A range finder based on a triangulation or a

time-of-flight method is typical of an active image sensor as described before. Unfortunately, recent research shows little evidence that resolution can be improved in the future.

The intended use of the range finder is the vision of a mobile robot with an operating distance of a few meters to few tens of meters. In industrial manufacturing applications, however, what is needed is a shorter-distance, higher-speed profile finder, for example, having a resolution of 50 micrometers, an operating range of 1 millimeter, and an operating distance of 5 to 10 centimeters. This would be extremely useful for surface profile detection such as the inspection of object surfaces for pinholes.

(4) Neural network

networks are being studied extensively. A network simulator capable of simulating one million neurons has already been put into laboratory use. With hardware approaches, many neuro-chips are being investigated. One recent topic in this field is the prototype wafer-scale integration of a network with 576 neuro-chips on a single wafer. [26] This indicates the possibility of integrating more than 10,000 neurons on a single wafer by the late 1990s. The primary uses of neural networks in industry will be vision devices to track the flow of parts and products, and sensor fusion devices for intelligent manufacturing control.

As processor-related technologies, neural *

4.4 Future application areas

The 1990s is neither discontinuous with the 1980s nor with the 21st century. Thus, consideratian of the 21st century will be important in determining what is necessary in the 1990s. Almost all problems facing the world today will worsen in the coming century if steps are not taken to solve them now. These problems include environmental damage, population explosion, natural resource shortages, and natural disasters. In all countries, it+ also important to cope with increasing social difficulties. Of course, these are political, economic, and social problems, and machine vision can only make a small

contribution. to the solution. Some application examples in this concem may be:

For a clean and attractive environment * Environmental measurement and analysis * Land and seabed cleaning * Garbage treatment

For a flourishing society * Food engineering such as agricultural

automation, seabed cultivation, and automated animal breeding

*'Efficient physical distribution control * New traffic systems

For a safe and stable society * Traffic safety management * Disaster prevention and rescue * Security management * Customs inspection automation

These applications may include the following new challenges in machine vision technology. These are the recognition of:

* Irregularly shaped objects and their defects

* Shape-changing objects and their degrees of deformation

* Quickly moving objects and their absolute/relative speeds

* Hiddenhard-to-see objects and the seriousness of their defects

* Flexible/soft/untouchable objects and qualitative features expressed in such abstract words as matured, tender, colorful, lovely, and beautiful

* Individualsfliving-things and their existence, numbers, faces, and facial expressions

statiddynamic behavior * Groups of living things and their

The right directions to solve these problems must be set, and clues to the solutions must at least be ascertained within the 1990s.

5 Conclusions

In this paper, the past history of the machine vision research has been reviewed, and its state-of- the-art has been evaluated, stressing the practical standpoint. In the past, the main effort was focussed on how quickly the processing can be completed, which resulted in binary image processing with rather simplified, local-parallel methods. Machine vision gradually progressed to gray-scale image processing as image processing LSIs became available. Nowadays, the impetus is towards knowledge-based

x x m

processing. However, the heart of the industrial machine vision is its speed, and in general knowledge- based processing is still too slow. Therefore, it is necessary to adopt a simplified method where the knowledge is compared with the observed facts, and when matched, a certain decision is made and a certain operation is added to the observed facts. This knowledge-driven processing seems to be promising, as shown by the wafer inspection machine. This machine is a typical "knowledge-directed'' inspection machine where the knowledge-driven aspects are implemented in hardware. In the future, machine vision may be further improved by this extensive utilization of knowledge.

Some new application areas have also been discussed in this paper. However, futurology is always vague and involves many pros and cons. Like other engineering problems, machine vision will also be expected to somehow contribute to the prosperity of future human society and the conservation of its natural environment. A few such contributions are conceivable. Among them are technologies to alleviate disasters, to increase human happiness by providing a comfortable living environment and the stable supply of essential commodities. We must occasionally think of these factors when researching machine vision technology.

References

M. Ejiri, et al. : A prototype intelligent robot that assembles objects from plan drawings, IEEE Trans. Computer, C-21, 2, pp.161-170 (1972) M. Ejiri : Industrial Image Processing, Shokodo Pub., Tokyo (1988) (in Japanese) M. Ejiri : Machine Vision: A Practical Technology for Advanced Image Processing, Gordon & Breach Science Publishers, New York (1989) M. Ejiri, Y. Nakano and T.Chusho : Artificial Intelligence, Shokodo Pub., Tokyo (1988) (in Japanese) M. Ejiri et al. : A process for detecting defects in complicated pattems, Computer Graphics and Image Processing, 2, pp.326-339 (1973) S.Kashioka, M.Ejiri and Y.Sakamoto, A Transistor wire- bonding system utilizing multiple local pattem matching techniques, IEEE Trans. on Syst. Man & Cybern., SMC-

T.Uno, M.Ejiri and T.Tokunaga, A method of real-time recognition of moving objects and its application, Pattem Recognition, 8, 4, pp.201-208 (1976) H. Yoda, S . Ikeda and M. Ejiri : Hitachi Review, 22,9, pp.362-365 (1973). Also in Trans. SICE, 10, 3, pp.284- 289 (1974) (in Japanese) G. J. Agin : Computer vision systems for industrial inspection and assembly, IEEE Computers, 13, 5, pp.11- 20 (1980)

6, 8. pp.562-570 (1976)

[ 101 H. Sakouand M. Ejiri : Image understanding technologies

in industrial applications, Proc. Japan Society of Artificial Intelligence, 4, 1, pp.43-51 (1989)

[ 111 S.Kashioka, Y.Shima and M.Ejin, Automatic template selection technique for the local pattern matching method, Trans. IEICE, J68-D, 5 , pp.1103-1110 (1985) (in Japanese)

[ 121 H. Sakou, T. Miyatake, S . Kashioka and M. Ejiri : A position recognition algorithm for semiconductor alignment based on snuctural pattem matching, IEEE Trans. ASSP, 37, 12, pp.2148-2157 (1989)

[ 131 H.Sakou. H.Yoda, M.Ejiri and O.Obata, An algorithm for inspecting radiation pattems of infrared light emitting diodes, Trans. IEEJ, 107-C, 8, pp.737-744 (1987) (in ..

Japanese)

oriented industrial vision system, Trans. Information [ 141 K.Matsuzaki, M.Isobe, S.Hata and K.Takeichi, Object-

Processing Society of Japan, 29, 12, pp.1119-1128 (1988)

A.Kikuchi, A sealing robot system with visual seam trackirk, Proc. Int. Conf. on Advanced Robotics, pp.35 1

[ 151 S.Sawano, J.Ikeda, N.Utsumi, H.Kiba, Y.Ohtani and

~~

358, TGkyo (1983) [ 161 Y.Nakagawa, Y.Oshida, T.Ninomiya and H.Sakai, Shape

detection of solder joints by spotlight scanning light- section method, Trans. SICE, 22,9, pp.982-987 (1986) ..

(in Japanese)

T.Kobavashi. Visual insDection machine for chip-mounted [ 171 Y.Takagi, D.Katsuta, S.Hata, T.Ohi, K.Murata and

boards Gsing'slit-light mkthod, Tech. Report of image Processing Application Committee, Japan Society of Precision Engineers, 3, 1, pp.1-I1 (1988)

[ 181 M. Ejiri : Knowledge-based approaches to practical image processing, Int. Workshop on Industrial Applications of Machine Intelligence and Vision (MIV '89) Tokyo (April 1989)

Knowledge-directed inspection for complex multilayered pattems, Machine Vision and Applications, 2, pp. 155- 166 (1989)

automatic wafer inspection system using pipelined image processing techniques, IEEE Trans. PAMI, 10, I , pp.4- 16 (1988)

[21 J US-Japan Manufacturing Research Exchange, National Academy Press (1989)

[22] H. Yoda : An algorithm for determining shape descriptors based on video-rate one-pass image processing techniques, Trans. IEICE, J72-D-I1,7, pp. 1017- 1028 (1989) (in Japanese)

[23] H. Sakou, H. Yoda and M. Ejiri : An algorithm for matching distorted waveforms using a scale-based description, IAPR Workshop on Computer Vision, Tokyo (Oct. 1988)

discrimination using self-organized multi-resolution filtering, Trans. IEICE, J73-D-I1,4, pp.562-573 (1990) (in Japanese)

[25] T. Unnai, N. Goto and Y. Takasaki : Newly developed high-sensitivity HDTV camera tube, Hitachi Review, 38,

[ 191 M. Ejiri, H. Yoda, H. Sakou and Y. Sakamoto :

[20] H. Yoda, Y. Ohuchi, Y. Taniguchi and M. Ejiri : An

[24] H. Sakou, H. Matsushima and M. Ejiri : Texture

2, pp.121-126 (1989) [26] M. Yasunaga, N. Masuda, M. Asai, M. Yamada, A.

Masaki and Y. Hirai : A wafer scale integration neural network utilizing completely digital circuits, Proc. IJCNN, Vol.11, pp.213-217 (June 1989)

xxxx