-
FPGA design for real time flaw detection on edgesusing the
LEDges technique
Ygo N. BatistaFederal Institute of Education, Scienceand
Technology of Pernambuco - IFPE
Pesqueira, Pernambuco, BrazilEmail:
[email protected]
Cristiano C. de AraujoFederal University
of Pernambuco - UFPERecife, Pernambuco, Brazil
Email: [email protected]
Abel G. S. FilhoFederal University
of Pernambuco - UFPERecife, Pernambuco, Brazil
Email: [email protected]
AbstractThis work presents a FPGA design for real time
flawdetection on edges based on LEDges technique. The LEDges, onone
hand, significantly reduces the computational effort to per-form
the image segmentation, representation and description. Onthe other
hand reduces the use of costly architectural resourcessuch as
processor and memory. Thus the FPGA design of theLEDges allows the
implementation of automated visual inspectionsystems satisfying the
increasing demand for performance. Wehave developed, implemented
and applied the FPGA design to areal industrial problem, where
defects were successfully detectedon edges of toothpaste tubes. We
achieve lower response timeand lower use of computational resources
than other solutionswhich have same computational complexity.
I. INTRODUCTIONAutomated Visual Inspection (AVI) for dimensional
inspec-
tion is one of the main machine vision applications. AVIenables
inspections in hazardous or hard-to-access environ-ments and can
improve productivity and quality in industrialproduction lines [1].
For instance, the AVI system imple-mented by C. Fernandez [2] made
orange classification 27times faster and mistakes reduced by 65%
when compared tohuman inspection.
Despite the relevance of AVI for industrial automation, itstill
faces some challenges. Many AVI systems are based oncomputationally
intensive algorithms and expensive hardware.This limits the
application of AVI to certain domains. Thusthere is a growing
demand for simpler algorithms implementedin efficient embedded
systems.
In order to improve the inspection rate and the imageresolution,
this paper presents an implementation in FPGAof the LEDges
technique. We applied it as an AVI system to areal industrial
problem: flaw detection on edges of toothpastetubes. We measured
and compared some parameters of thedeveloped AVI system with other
related works. Comparedto the ARM-based implementation of the
LEDges [3], theresults were much better. But these comparisons are
not trivialwhen other techniques are used since every inspection
hasits specific requirements and we cannot test these
approachesusing an identical test set. However, these measures show
thatthe developed system has a very good inspeciton rate.
This paper is structured as follows. We explain the
imageprocessing pipeline in section II while in section III
wedescribe the LEDges technique. We show the architecture of
the system in section IV and present the FPGA implementationin
section V. A case study is described in VI. Results arediscussed in
section VII. Finally, we conclude and present thefuture works in
section VIII.
II. IMAGE PROCESSING STEPS FOR A TYPICALAUTOMATED VISUAL
INSPECTION SYSTEM
The image processing for AVI applications can be dividedinto
three distinct levels of processing. Figure 1 shows theselevels in
a typical AVI process. The first part, low level pro-cessing,
relates to image acquisition and quality enhancement.
KnowledgeBase
Image Acquisition
Preprocessing
Segmentation
Representation
Description
Recognition
Interpretation
Results
Problem Domain
Low
Interm
ediate
High
Level P
rocessing
Fig. 1. Image processing steps of a typical AVI system.
The intermediate-level processing involves the steps of
(i)segmentation, (ii) representation and (iii) description of
theimage. Segmentation may be one of the most important andcomplex
activities throughout the image processing. In this,regions of
interest (ROI) are identified and then representedin terms of its
internal features (pixels that make up the region)or external
features (edges). The next step is to describe theROI into a
digital set of numbers that can be processed by acomputer system.
The high level processing is responsible forimage recognition, and
finally, its interpretation indicating ifthe object under
inspection is or is not within the acceptablestandards for the
industry.
Mauricio Ayala Rincon978-1-4673-2608-7/12/$31.00 2012 IEEE
Mauricio Ayala Rincon
Mauricio Ayala Rincon
-
Finally, the knowledge base contains accumulated knowl-edge
about the inspection, provided by man and / or bymachine [4]. For
instance, accumulated experiences of staffin relation to an manual
inspection to be automated, whichare already known results related
to the problems and theirsolutions.
III. LEDGESThe LEDges [3] is a AVI technique for real time
automated
visual dimensional inspections which is based on simplicityof
the algorithm and low complexity of hardware presentedin [5][10],
but with the improvement of detecting all edgesof the object in the
images acquired by the camera, notjust the outermost edges.
Previously, this capability was onlypossible in the works based on
depth cameras, laser and pro-jectors [11][14], which has a long
processing time, resource-intensive computing, and delicate and
complex image capturedevices. This makes it unfeasible their use in
embedded andreal-time AVI systems with the following requirements:
(i)Simple classification of the object between good or flaw;
(ii)Fast response time without the use of high speed cameras;(iii)
A fuzzy background with many objects making the imageprocessing
harder; (iv) A low space available to assemble thenew AVI system to
preexisting production machinery. To thebest of our knowledge there
are no solutions available thatmeet all these requirements.
The LEDges is described in four steps as presented inFigure 2.
First the object is illuminated with a high powerstructured light
source and its image is captured by the camera.The result of this
step is an image displaying the lighted areasof the objects in the
foreground in higher intensities, while thebackground and the
shadows are displayed in lower intensities.
In the thresholding step the acquired image is easily seg-mented
into two levels of intensity: (i) the background andthe shadows are
indicated by regions with zero intensity, colorblack, (ii) and
light areas in the foreplane are indicated by themaximum intensity,
color white.
In the third step, signature generation, the thresholded imageis
represented by its edges and described as a signature.Each edge is
represented numerically in terms of the distancebetween the
beginning of the image lines and relevant black-to-white and
white-to-black transitions of the thresholdedimage. This set of
distances makes the image signature. Notethat, in general, the
Hough transform, used in [15], [16] wouldrequire greater use of
hardware.
The last step is the comparison, which compares the gen-erated
signature with a standard good objects signature and
generates as output a binary flag that indicates whether
theobject is a good one or there is any defect on its edges.
One advantage of the technique is that it allows all steps
(im-age capture, thresholding, signature generation and compare)to
be executed in a pipelined approach. As a consequence, thememory
usage is significantly reduced, since it is not necessaryto store
the image for processing, reducing the implementingcost and the
total processing time.
As a restriction of the technique, as well as many AVIsystems
based on light reflection, the LEDges depends on thepositioning of
illumination and camera, as well as the objectcolor and texture of
the material. Despite these restrictions,this technique is well
fitted to an industrial environment wherethey can usually be met as
it is a controlled environment.
IV. ARCHITECTURE DESIGN
The definition of the architecture to be used for
implemen-tation of the LEDges is based on the computational
resourcesrequired to perform the image processing and
organizationof these resources in order to meet the industry
requirementsregarding the inspection.
The main objective is to properly meet the specified
func-tionality and to achieve the speed and accuracy rate
required.A secondary objective has economic nature, is to
reduceproduction costs. Related with the user interface, the
flexibilityto update the parameters of the inspection is another
desirablequality. The effort of the engineering team for
development ofthe AVI system and the timing of availability of the
product onthe market also differ between architectures. Finally,
there isconcern about energy efficiency. However, embedded
systemstypically have lower power consumption than the
acceptabledefined by the industry.
The procedure for architecture definition was the following:(i)
Preliminar analysis of the algorithm for the identificationof the
necessary controllers, (ii) computer modeling of theLEDges using
the FREEMAT software (Fig. 3), (iii) optimiza-tion with the
identification of performance bottlenecks in dataprocessing and
verification of really necessary data widths,(iv) preparation of a
list with characteristics and constraints ofpotential architectures
for checking the technical adequacy ofthe LEDges with the
architectures. As a result of this analysis,the specified
functionality can be met adequately, which is themain objective in
the choice of architecture, with the FPGAdesign.
The second priority is to achieve the necessary inspectionspeed
keeping the hit rate obtained in the modeling. In an
SceneImage
captureThresholding
Signaturegeneration
Comparison Result
OutputInput Processing Steps
Fig. 2. LEDges processing steps.
-
Fig. 3. Computer modeling of the LEDges on FREEMAT software. The
acquired and thresholded image are displayed, as well as the image
description asa signature.
architecture based on the low cost FPGA TERASIC kit DE2-70, the
LEDges can be applied to process images with goodresolution (640 x
480 pixels), detecting dimensional errors of0.5 mm for the
mechanical assembly performed and with arate of 100 inspections per
second. Finally, computer modelingallows a fast implementation in
FPGA with good flexibilityto update the parameters. Therefore, the
choice of an FPGA-based architecture achieves the objectives for
the design.
V. FPGA-BASED ARCHITECTUREThe available hardware is the DE2-70
development kit,
including the D5M digital camera and LTM LCD display,all from
the Taiwanese company TERASIC. The DE2-70 isequipped with a low
cost FPGA Cyclone II EP2C70F896C6.In addition to the components
already available in the kit, itwas also necessary to install a
high-power LED, required toperform the LEDges illumination [3].
The AVI system was segmented into six major functionalblocks:
(i) image acquisition; (ii) thresholding; (iii)
signaturegeneration; (iv) comparison; (v) LCD displaying. Each
blockcomprises one or more hardware modules implemented inFPGA as
presented in Figure 4. The figure also shows theinputs and outputs
of the modules.
VI. CASE STUDY: FLAW DETECTION ON EDGES OFTOOTHPASTE TUBES
We implemented and applied the LEDges as an AVI systemto a real
industrial problem consisting on the flaw detectionon edges of
toothpaste tubes in a production line. In this casestudy we tested
the effect (i) of lighting, (ii) of processingonly the R component
in the RGB signal and (iii) of changesin the camera exposure time.
Two parameters of interest were
not adjusted, (i) image resolution and (ii) camera focus, onceit
requires major changes in architecture or new
mechanicalassemblies.
VII. RESULTSWe observed some good features of the LEDges
when
compared to the AVI systems presented in section III.
Thesecomparisons are not trivial since every inspection has
itsspecific requirements and we cannot test these approachesusing
an identical test set.
The Table 1 shows that the less complex systems onlydetect the
outermost edges of the object. Those which detectthe internal edges
have a complex, or very specific hardware,and slow image
acquisition. Furthermore, none of them wereimplemented on embedded
system. In fact most of them wereimplemented in large industrial
computers. Only the LEDgesis able to get the inspection time of
10ms, detecting all edges,although implemented in low-cost embedded
hardware.
Regarding the architecture based on FPGA and digitalcamera, the
image was acquired faster and with superiorresolution than the
microcontroller-based architecture [3]. Thenew inspection rate of
10ms is highlighted. We further notethat the two architectures use
low-cost processors. We usedonly 3% of the total available
resources in the low-costCyclone II FPGA.
The size of the final signature was different between
im-plementations. The FPGA presented four times more pointsthan the
microprocessor-based architecture. The more pointsyou have, the
inspection will be more accurate. But for thesimple case study,
both AVI systems presented hit rate of100%. Other features are
shown in Table 2 in a comparisonbetween implementations based on
microcontroller and FPGA.
-
RGGB RGB1 101 201 301 401 501 601
1
26
51
76
101
126
151
GOOD
BAD
CM
OS
CA
ME
RA
D5M KIT
IMA
GE
CA
PT
UR
E
CAMERASETTINGVIA I2C
RG
GB
TO
RG
BC
ON
VE
RS
OR
+ R
FIL
TE
R
Inspection apparatus
TH
RE
SH
OL
DIN
G
IDE
NT
IFIC
AT
ION
OF
RE
LE
VA
NT
TR
AN
SIT
ION
S
SIG
NA
TU
RE
CO
MP
OS
ITIO
N
CO
MP
AR
ISO
N
DISPLAY SELECTOR
SDRAM1 MEMORY CONTROLLER
SDRAM2 MEMORY CONTROLLER
LCD SETTINGVIA SPI
LCD DATAINTERFACE
SDRAM1
SDRAM2
FPGA
DE2-70 KIT
LCD DISPLAY
LTM KIT
Legend:ACQUISITIONTHRESHOLDINGSIGNATURE GENERATIONCOMPARISONLCD
DISPLAYINGDATA FLOW PIPELINE
Fig. 4. Block diagram for FPGA-based architecture.
-
Table 1: Comparison between techniques
Table 2: Comparison between LEDges implementations
Table 3: Used FPGA resources
TechniqueRelatedworks
NoteInspection
time
Able todetect allthe edges
Embedded /
Use of computational
resources.
Extraction of foreground from image
pairs, with or without specific lightingSun et al. [8]
Very effective for
image segmentation
N/A
(1)No
No (2) /
Moderate
Illumination with infrared light to
identify the foreground
Wu, Boulanger
e Bischof [9]
High performance in
the segmentation100 ms No
No /
Low
Use of depth camerasWang et al.
[11]
Uses specific and
high cost cameras66 ms Yes
No /
Low
Detection of foreground images with
static background
Casares et al.
[4]Simple algorithm N/A No
No (2) /
Moderate
Use of the laser lighting Liao et al. [12]High precision;
complex lighting3000 ms Yes
No /
Moderate
Use of standard projectorsWang et al.
Batista [3]
[13]
High precision;
complex lighting500 ms Yes
No /
High
LEDgesHighperformance
10 ms YesYes /
Very lowN/A - Not avaiable
(1) Need to acquire two images
(2) Technique developed for use in embedded systems
Objectives / Parameters optimized
Main processor
Use of the main processor
Camera technology
Inspection rate
Image resolution
Size of final signature
Able to adjust resolution and inspection rate
Main processor cost
Camera cost
Effort to develop and update design
Flexibility to update the parameters
Hit rate in inspections
(1) Adjustable from 2592 x 1944 pixels @ 70 ms/inspection to 640
x 480 pixels @ 10 ms/inspection
Inspection rate;
Resolution.FPGA Cyclone II 2C70
Costs; efforts of the
engineering team.LPC2148 - Core ARM7 TDMI
FPGA-based ArchitectureMicrocontroller-based
Architecture
300 72
Yes (1) No
3% 9%
CMOS Digital CCD Analog NTSC
10 ms / inspection 16.66 ms / inspection
100% 100%
Parameters
Low Very low
Low Very low
Moderate
Moderade
Low
High
640 x 480 pixels Analog : 120 lines
FPGA ModuleCombinational
functionsLogic registers
Image capture 55 48
RGGB to RGB conversor + R filter 146 94
Camera setting via I2C 229 134
Thresholding 67 32
Identification of relevant transitions 128 93
Signature composition 162 135
Comparison 203 100
Total990 / 68416
(1.5%)
636 / 68416
(0.9%)
Memory usage
12080
32232 / 1152000
(2.8%)
0
19152
0
0
0
1000
-
Table 3 presents the resourcers related to implementationof the
LEDges modules on the low-cost FPGA Cyclone IIEP2C70F896C6. The
report was generated by the Quartus II.As can be seen, the use of
FPGA resources is very low andmost of the resources are used by the
image acquisition.
VIII. CONCLUSIONS AND FUTURE WORK
Several works present techniques for inspections of edgeson
objects placed in the foreground. However, these com-plex solutions
showed long processing time, intensive use ofcomputational
resources, delicate and complex image capturedevices. The technique
implemented in this work, LEDges,performs a robust image
segmentation through a simple andefficient thresholding, which
enables high performance withlow use of computational
resources.
In the case study, inspection of toothpaste tubes, the
FPGAimplementation resulted in a very fast inspection when
com-pared with those presented by competing techniques capableof
performing the same type of inspection. Futhermore, weused only 3%
of the total available resources in the low-costCyclone II FPGA.
Thus it is possible multiple implementationsof LEDges in a single
FPGA, that can be useful for inspectionof objects from multiple
images or several different angles.The hit rate was 100% proving
that it is possible to performimage processing in simple embedded
systems. But makeno mistake, the simplicity can only be achieved
through hardwork (Clarice Lispector).
As future work, it will be necessary to develop a system
forfunctional verification, make more accurate measurements
ofprecision and accuracy, develop a supplement for
FREEMATautomating the code conversion to Verilog. Then the
FPGA-based AVI system will be ready to be tested in new
high-peformance applications.
ACKNOWLEDGMENT
The authors would like to thank Optanica Photonic Solu-tions for
technical support and Rafael C. Neto for his aid withtranslation.
In addition, the authors thank FACEPE for partialfinancial support
in the design development.
REFERENCES[1] E. Malamas, E. Petrakis, M. Zervakis, L. Petit,
and J. Legat, A survey
on industrial vision systems, applications and tools, Image and
VisionComputing, vol. 21, no. 2, pp. 171188, 2003.
[2] C. Fernandez, J. Suardiaz, C. Jimenez, P. Navarro, A.
Toledo, andA. Iborra, Automated visual inspection system for the
classificationof preserved vegetables, in Industrial Electronics,
2002. ISIE 2002.Proceedings of the 2002 IEEE International
Symposium on, vol. 1.IEEE, 2002, pp. 265269.
[3] Y. Batista, LEDges: A new automatic visual inspection
technique for realtime flaw detection on edges. Masters
dissertation. Federal Universityof Pernambuco, Brazil, 2011.
[4] T. Brosnan and D. Sun, Improving quality inspection of food
productsby computer visiona review, Journal of Food Engineering,
vol. 61,no. 1, pp. 316, 2004.
[5] M. Casares, S. Velipasalar, and A. Pinto, Light-weight
salient fore-ground detection for embedded smart cameras, Computer
Vision andImage Understanding, vol. 114, no. 11, pp. 12231237,
2010.
[6] R. Bolle, J. Connell, N. Haas, R. Mohan, and G. Taubin,
Object imagingsystem, May 20 1997, uS Patent 5,631,976.
[7] J. Sun, Y. Li, S. Kang, and H. Shum, Flash matting, ACM
Transactionson Graphics (TOG), vol. 25, no. 3, pp. 772778,
2006.
[8] J. Sun, S. Kang, Z. Xu, X. Tang, and H. Shum, Flash cut:
Foregroundextraction with flash and no-flash image pairs, in
Computer Vision andPattern Recognition, 2007. CVPR07. IEEE
Conference on. IEEE, 2008,pp. 18.
[9] Q. Wu, P. Boulanger, and W. Bischof, Robust real-time
bi-layer videosegmentation using infrared video, in Computer and
Robot Vision,2008. CRV08. Canadian Conference on. IEEE, 2008, pp.
8794.
[10] , Bi-layer video segmentation with foreground and
backgroundinfrared illumination, in Proceeding of the 16th ACM
internationalconference on Multimedia. ACM, 2008, pp. 10251026.
[11] L. Wang, C. Zhang, R. Yang, and C. Zhang, Tofcut: Towards
robustreal-time foreground extraction using a time-of-flight
camera. Citeseer,2010.
[12] C. Liao, J. Su, Y. Lin, and Y. Tarng, Application of the
structuredillumination method for automated visual inspection of
the loudspeakercones, Journal of Materials Processing Technology,
vol. 200, no. 1-3,pp. 5970, 2008.
[13] Z. Wang, H. Du, S. Park, and H. Xie, Three-dimensional
shapemeasurement with a fast and accurate approach, Applied Optics,
vol. 48,no. 6, pp. 10521061, 2009.
[14] M. Chang and S. Park, Automated scanning of dental
impressions,Computer-Aided Design, vol. 41, no. 6, pp. 404411,
2009.
[15] S. Zhang and Z. Liu, A robust, real-time ellipse detector,
PatternRecognition, vol. 38, no. 2, pp. 273287, 2005.
[16] C. Chien, Y. Cheng, and T. Lin, Robust ellipse detection
based onhierarchical image pyramid and hough transform, JOSA A,
vol. 28,no. 4, pp. 581589, 2011.
[17] F. Duan, Y. Wang, H. Liu, and Y. Li, A machine vision
inspector forbeer bottle, Engineering Applications of Artificial
Intelligence, vol. 20,no. 7, pp. 10131021, 2007.
