DefectInspectioninDisplayPanelUsingConcentrated …downloads.hindawi.com/journals/ijo/2019/8039267.pdfHamming distance is used to calculate similarity. e Hamming distance is used to

Research ArticleDefect Inspection in Display Panel Using ConcentratedAuto Encoder

DongHun Ku

School of Mechanical amp Aerospace EngineeringSNU-IAMD Seoul National University Seoul 08826 Republic of Korea

Correspondence should be addressed to DongHun Ku dhku616snuackr

Received 22 May 2019 Revised 19 August 2019 Accepted 11 September 2019 Published 10 October 2019

Academic Editor John T Sheridan

Copyright copy 2019 DongHun Ku is is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

In this paper concentrated auto encoder (CAE) is proposed for aligning photo spacer (PS) and for local inspection of PSe CAEmethod has two characteristics First unaligned images can be moved to the same alignment position which makes it possible tomove the measured PS images to the same position in order to directly compare the images Second the characteristics of theabnormal PS are maintained even if the PS is aligned by the CAE method e abnormal PS obtained through CAE has the samealignment as the reference PS and has its abnormal characteristics e presence or absence of defects and the location of defectswere identified without precisely measuring the height of the PS and critical dimension (CD) Also alignment and defectinspection were performed simultaneously which shortened the inspection time Finally inspection performance parameters andinspection time were analyzed to confirm the validity of the CAE method and were compared with the image similaritycomparison methods used for defect inspection

1 Introduction

Interferometry that uses the light interference phenomenonto measure the height and CD of an object can measure alarge area at one time without damaging the object beingmeasured Because of its advantages of high measurementresolution and speed interferometry has been utilized in thefield of thin-film transistor-liquid-crystal display (TFT-LCD) microelectromechanical systems (MEMS) andsemiconductors Interferometry has been verified as an in-line measuring instrument for manufacturing processes aswell as for research and development in the precision partmanufacturing industry In the TFT-LCD manufacturingprocess an interferometer measures the height and CD ofirregularly shaped objects such as PS halftone and colorfilter [1ndash5]

TFT-LCD has a digital display that displays informationthrough very thin liquid crystals which are most widely usedin the display industry In order to display information thisproduct uses the polarization property of the polarizer andthe change in state of the liquid crystal with liquid-solidproperty to adjust the amount of light passing through e

TFT-LCD consists of two glass substrates with a color filterand a TFT a liquid crystal injected between two glasssubstrates and a backlight unit PS is the unit that maintainsthe gap between the two glass substrates PS is similar to abump and typically has a bell-shaped form e height andsize of the PS are important parameters that are measured inthe manufacturing process because they determine theamount of the liquid crystal that is injected and determinethe degree of deformation caused by the external pressure ofthe display panel [6 7]

Interferometry is used for repeated measurements of PSand inspection A 3D height image and 2D tomographicimage can be obtained from interferometry measurements[8ndash10] e height of the PS can be measured based on the3D height image and the axial CD measurement and thedefect of the shape itself can be inspected using the 2Dtomographic image It is difficult to verify the shape defect ofthe PS based only on the axial CD measurement CDmeasurements in various directions cause the inspectiontime to increase In general an image similarity comparisonmethod was used to inspect the shape of the PS defect reemethods can be used to compare image similarity [11ndash17]

HindawiInternational Journal of OpticsVolume 2019 Article ID 8039267 12 pageshttpsdoiorg10115520198039267

histogram comparison (HC) template matching (TM) andfeature matching (FM) methods ese methods are used todetermine whether the normal PS template to be inspectedexists in the test image If there is a similar normal PStemplate the similarity is outputted If the degree of simi-larity is high PS is considered to be normal while if thedegree of similarity is low it is possible to determine that thedefective shape is an abnormal PS

In this paper the CAE method is proposed which can beused as a replacement for the existing image similarity com-parison method that is used for alignment and defect inspectionof PS images acquired from an interferometry CAE allowsmeasured PS images to be aligned e shape of the PS wasinspected and the PS defect was checked by comparing thereference PS image and the PS image aligned through theCAE Itis possible to identify defects and their location without preciselymeasuring the height of the PS and CD with the CAE methodand it is also possible to reduce inspection time by simultaneouslyaligning and analyzing defects Finally the performance of theproposed CAE method was veried by comparing it with theexisting image similarity comparison methods

2 Theoretical Background

21 Auto Encoder e auto encoder is a learning networkbased on deep learning which learns functions that make theoutput value similar to the input valuee auto encoder hastwo neural networks as shown in Figure 1 e neuralnetwork located at the front is an encoder and the networkbehind it is a decodere encoder extracts the feature of theinput data and the original data are regenerated by usingthis feature and the decoder By training the input andoutput data to be as equal as possible the auto encoder canextract the feature eciently [18] e auto encoder is usedto remove and restore noise and image noise and can be usedfor image classication [19ndash24] As the amount of researchon sensor data utilization and analysis has increased inrecent years it is also possible to use an auto encoder as acorrection technique for sensor data

A typical auto encoder is an articial neural network thatmakes output to be input e target output is used as aninput e encoder is a function of h fθ(x) s(Wx + b)which extracts the hidden feature h isin [0 1]d from the inputvector x isin [0 1]D In this equation θ W b W is d timesDdimension weight matrix and b means d dimension biasvector s is an element-wise activation function that makes anetwork deeper by changing input data so that it is non-linear A sigmoid function or a rectied linear unit (ReLu) isused as an activation function e formula of the sigmoidfunction is shown in equation (1) Feature h that is calculatedin the hidden layer is converted into vector z isin [0 1]D by thedecoder z gθprime(h) s(Wprimeh + bprime) e auto encoder per-forms the process of extracting and reconstructing thefeature from the input data through the encoder and de-coder In conclusion the input x is converted to h throughthe hidden layer and the nal output z is reconstructed fromh Each input x is mapped to the related feature h which isthen mapped to a reconstruction z which satises x asymp z eauto encoder must mathematically minimize the dierence

between input x and output z e dierence between theinput x and the output z is termed loss which is also calledthe average reconstruction error and the goal of the autoencoder is to optimize the parameter θlowast θprimelowast to minimize thisloss e expression of the loss optimization parameterθlowast θprimelowast is shown in equation (2) and loss is calculated withequation (3) Loss function is a scalar function that calculatesthe dierence between input and output and is used as anindicator to evaluate the dierence of datasets Meansquared error and cross entropy error are mainly used as aloss function e loss function L in equation (3) is the mostcommonly used mean squared error function

s(x) 1

1 + eminus x (1)

θlowast θprimelowast( ) argminθθprime1nsumn

i1L x(i) z(i)( ) (2)

L(x z) x minus z2 x minus s Wprime(s(Wx + b)) + bprime( )

2 (3)

22 PS Defect Inspection Using Interferometry Inspection ofthe PS defect of the display panel is performed using a 3Dshape and a 2D tomographic image obtained from an in-terferometry It is possible to measure the height from the3D shape and it is possible to measure the axial CD of the PSfrom the 2D tomographic image [7ndash9] If the dierencesbetween the height or CD of the measured PS and thereference PS values are larger than the threshold it is judgedto be an abnormal PS is PS defect inspection methodconrms the measured value of height x and y axis CDassuming that the shape of the PS is bell shaped Defects indirections other than the x and y axis cannot be detectedusing the PS defect inspection method If these precisemeasured values are not needed and only the presence ofdefects is required the image similarity comparison methodcan be used e HC method [11 12] the TM method[13ndash15] and the FM method [16 17] are used in this paperfor comparing image similarity Image similarity compari-son methods require a test image and a template image fordetection With image similarity comparison methods auser can determine if the template image that the user is

x1

x2

x3

xD

z1

z2

z3

zD

h1

h2

hd

Input Hidden OutputEncoder Decoder

W Wprime

Figure 1 Structure of an auto encoder

2 International Journal of Optics

searching for exists in the test image and how similar it ise location of the PS within the analysis range of interestthrough image similarity comparison methods can be ac-curately determined and the defectiveness can be de-termined by comparing the similarity

e HC method is a simple method used to comparehistograms between template and test image A histogram isa graph that displays the distribution of color informationfor each pixel in an image In the histogram shown inFigure 2 the x axis corresponds to the brightness value thatranges from 0 to 255 and the y axis indicates the number ofpixels corresponding to each brightness value PS histogramdistributions of normal PS and abnormal PS are shown inFigure 2 e normal PS and abnormal PS have differenthistograms that can be distinguished by their color in-formation e HC method has the advantage of beingsimple and requiring less time for analysis However sincethe color information of the image is compressed andcompared the accuracy of the similarity comparison is low

e TMmethod compares the color information of eachpixel of the template image and the test image e templateimage moves in the x and y directions and compares theintensities of the pixels of the test and template images toacquire the closest position and corresponding similarityHamming distance is used to calculate similarity eHamming distance is used to measure the difference be-tween two types of data such as characters bytes andimages and can be expressed as shown in equation (4) x andy are the intensities of the template and test image re-spectively f is the function for mathematical preprocessingwhich is applied to the template and test images e smallerthe Hamming distance the higher is the similarity with thetemplate image e TM method is characterized in that themore complex the image to be searched for and the larger thetest image the longer is the inspection time ere arevarious kinds of TM methods depending on the function ofcalculating the Hamming distance e TM method com-pared in this paper is the TM-SQDIFF method that uses thesquare of the image intensity difference e Hammingdistance of TM-SQDIFF is defined as shown in equation (5)X and Y denote the template and the test image respectivelyxprime and yprime is the size of the template image to be searchedewindow size of the template image is moved in the testimage and the squares of the intensity difference are cal-culated as the Hamming distance is Hamming distancehas a small value at the TM position If the template imageperfectly matches the test image it returns 0 Otherwise theHamming distance becomes larger

H 1113944n

i1f xi( 1113857 minus f yi( 1113857

11138681113868111386811138681113868111386811138681113868 (4)

H(x y) 1113944

xprime yprime

X xprime yprime( 1113857 minus Y x + xprime y + yprime( 1113857( 11138572

(5)

e FM method extracts the features contained in thetemplate and test images and compares the matching degreeof each feature point In this paper the SIFT FM method isused to extract features that are invariant in image size and

rotation SIFT is resistant to scale illumination translationrotation and occlusion of images However the amount ofcomputation required to extract these feature points isconsiderable e flow of the SIFT algorithm is shown inFigure 3 In the scale-space extrema detection step aGaussian pyramid is generated and a difference of Gaussian(DoG) is calculated in order to extract the pole part as afeature point candidate Gaussian pyramids consist of filtersof various scales SIFT using a Gaussian pyramid extractsfeature points by scale so it can be recognized even if thescale of the target object is changed Taylor series is used toextract precise features in the key point localization stepInaccurate feature points are removed from feature pointcandidates that were extracted from scale-space extremadetection In the orientation assignment step the maindirection is assigned to each extracted feature pointGaussian blurring is applied to the 16times16 region around theextracted feature point to calculate the direction andmagnitude of the gradient Since this information containsorientation data SIFT can also recognize rotated objects Inthe key point descriptor step a descriptor is created for eachfeature point e key point descriptor is a spatial histogramof the image gradients that is used in the characterization ofthe appearance of a key point As shown in Figure 4 a keypoint descriptor is created by mixing gradient values aroundthe feature points that were obtained in the previous stepFinally the key point descriptors of each reference and testimage are matched by calculating the hamming distanceImage similarity can be examined through the degree ofmatching of the corresponding feature points

3 PS Defect Inspection Using ConcentratedAuto Encoder

e auto encoder is trained so that A is the output whenimage A is the input and B is the output when image B is theinput In the proposed CAE method training proceeds sothat only one image is the output Additionally training alsoproceeds so that the auto encoder is concentrated on oneimage as shown in Figure 5 e auto encoder is set to inputx1 sim xD and output z1 sim zD while CAE is set to x1 sim xD forinput and z1 for output Training proceeds so that all theinputs output one and the same output When training forinspection of PS defect in this paper the normal PSs that arenot aligned are inserted in the input and one normal PS thatis the align reference is inserted into the output All input PSsthat were not aligned have the same alignment with thereference PS through CAE

e detailed network of the CAE is shown in Figure 6e W times H size of PS images at various locations is used asinput for the CAE e CAE is trained so that the input PSimages become the reference PS image of W times H size ereference PS image is measured so that the PS is at the centerof the image and it becomes the align reference of the PS Aconvolution block consisting of a convolution layer andReLu was used to train the CAE e convolution layertransforms the size and depth of data and extracts theirfeatures Activation functions such as ReLu facilitate theextraction of features through nonlinearity of the data e

International Journal of Optics 3

CAE network shown in Figure 6 uses seven convolutionblocks e convolution layers that comprise the encoderreduce the size of the image to (W8) times (W8) and increaseits depth to 128 e encoder performs training throughfeature extraction of input data e decoder in the next partreconstructs the original size of the output image from theextracted features e convolution layer of the decoderoutputs the feature data as the original depth 1 W times H sizeimagee difference between the final output and referencePS images is termed loss e Adam optimizer is used tominimize the loss additionally the hyperparameter opti-mization of each convolution layer is possible through CAEtraining

Training was performed by setting the image of 10000normal PSs that were not aligned as input and one ref-erence PS as output As the training progresses the inputPS aligns with the reference PS through training as shownin Figure 7 Figure 7 shows the change in normal PSaccording to the training epoch As the number of trainingevents increases the ability to learn improves so that even

small dots on the lower right of the output image can beexpressed in detail is shows that the network that wastrained through CAE outputs the same normal PS with thesame alignment When an abnormal PS is inserted intothis network alignment is the same as the reference PSand the abnormal characteristic is maintained as shownin Figure 8 When a normal PS is the input the sameoutput as the reference image is the output If an abnormalPS is the input the output that differs from the referenceimage is the output It is possible to inspect the PS withoutprecisely measuring the PS using the characteristics of theCAE

A camera with 880times 640 pixels and a resolution of01 μmpixel was used in this experiment e farthest PSsthat were used for training are shown in Figure 9 edistance between PSs is approximately 740 pixels 74 μmetrained network can be calibrated up to a distance of 74 μmSince the reference PS is set to the PS located at the center itis possible to align the PS 37 μm from the reference PS usingthe corresponding network as shown in Figure 10 etraining PS set must contain a PS far away from the referencePS and use a deeper network to enhance alignment of thelearning ability to a PS further away

4 Experiment and Result

Figure 11 shows the flow of defect inspection using CAEefirst item that the defect inspection flow checks for is thepresence of a pretrained network A pretrained network is

0 50 100 150 200 2500

5000

10000

15000

20000

25000

30000

35000

40000

(a)

0 50 100 150 200 2500

5000

10000

15000

20000

25000

30000

35000

40000

(b)

Figure 2 Histograms of (a) normal and (b) abnormal PS

Scale-space extremadetection Key point localization

Key point descriptor Orientation assignment

Figure 3 Flow of the feature matching method SIFT


created by the preliminary training of input images In thispaper this pretrained network is used to align PS and tomaintain the characteristics of the defects If there is apretrained network defect inspection is possible by insertinga test image into the network However if there is nopretrained network it is necessary to measure the inputimage with an interferometer and to train the CAE in orderto create a network

A total of 10000 normal PS images were used for CAEtraining PS images that were not aligned were measured atvarious locations within the camera eld of view (FOV) ePSs have a positional dierence of up to 74 μm within thecamera FOV PS training images were divided into 8 2ratios and distributed to training and validation PS sets ePS training set was used for actual training and the PSvalidation set was used to evaluate the performance of thenetwork that was trained Since the validation set is not usedfor direct training the loss is larger and the accuracy is lowerthan the training set as shown in Figure 12 However thetraining and validation sets gradually converged as theyapproached 100 epochs

e network that trained 100 epochs converged su-ciently and showed stable loss and accurate results enetwork was veried to be sucient for training and vali-dation sets but its performance has not been veried in test

sets comprising new PS images In order to verify testingperformance ve dierent types of PS images (200 testimages of each type) that were not used in training com-prised a test set e loss and accuracy of the test set alsoconverged at approximately 100 epochs as shown in Fig-ure 13 Although the test set showed numerically loweraccuracy as compared to the training and validation setsused in direct training it can be conrmed that the CAEnetwork can be used even in a test set that was not used fortraining Overtting of the trained CAE network did notoccur and it can be applied to various PS images for defectinspection

Network training of up to 100 epochs maintained thecharacteristics of the PS and aligned the PS to the center ofthe image If the image to be reconstructed is complex anddiverse detailed image restoration is possible by increasingthe number of training repetitions or using a deeper net-work When a normal PS is inserted into a network in whichtraining was completed a normal PS is the output If anabnormal PS is inserted an abnormal PS having the samealignment as a normal PS is the output As shown in Fig-ure 14 when abnormal PSs are inserted into the CAE thePSs with the same alignment and abnormal characteristicsare the output

e degree of abnormal characteristics of the PS can beunderstood by performing the dierence operation on thealigned abnormal PS and the reference PS Since PS defectinspection requires judging abnormal products it is possibleto distinguish defect judgements using equation (6) eabnormal PSs that were inspected using these conditions areshown in Figure 15 An abnormal PS has a denite formwhen proceeding with the dierence operation with thereference PS

Abnormal |reference minus result|gt thresholdNormal |reference minus result|lt threshold

(6)

e accuracy of defect inspection can be conrmed bythe confusion matrix Precision recall accuracy and F1-score are the parameters of inspection performance that arefound in the confusion matrix Precision is the ratio of

Image gradientsKey point descriptor

Figure 4 Key point descriptor extraction in SIFT

x1

x2

x3

xD

z1

h1

h2

hd


W Wprime

Figure 5 CAE structure


correctly predicted ones that are expected to be normalrecall is the ratio of correctly predicted true to normal andaccuracy is a well-predicted rate of the overall sampleAccuracy shows the total reliability of the correspondingnetwork while precision and recall reveal the skewness ofthe network Since a trade-off relationship exists betweenprecision and recall the accuracy of the method can also be

confirmed using the average of these two values which iscalled F1-score e F1-score is used as an inspection pa-rameter to evaluate performance of the learning networkCalculation of the F1-score is shown in equation (7) Table 1shows the confusion matrix of the proposed CAE methodand Table 2 shows the result of the inspection performanceparameter

(a) (b)

Figure 9 Outermost PSs used for training

Encoder Decoder

d = 1 d = 32 d = 64 d = 64 d = 32 d = 1d = 128

D images 1 image

W H W2 H2 W4 H4 W8 H8 W4 H4 W2 H2 W H

Figure 6 CAE network

(a) (b) (c) (d)

Figure 7 Change in normal PS according to training epoch (a) 1 epoch (b) 25 epochs (c) 50 epochs (d) 100 epochs

(a) (b) (c) (d)

Figure 8 Change in abnormal PS according to training epoch (a) 1 epoch (b) 25 epochs (c) 50 epochs (d) 100 epochs


(a) (b)

(c) (d)

(e) (f )

Figure 10 Change of outer PS according to training epoch (a) 1 epoch (b) 20 epochs (c) 40 epochs (d) 60 epochs (e) 80 epochs (f ) 100epochs

Measure normal image for aligning and training

Train network

Input test image

Output inspection result

Interferometry

Concentrated auto encoder

Abnormal |reference ndash result| gt thresholdNormal |reference ndash result| lt threshold

Use pretrained network

If pretrained network exists

Yes

No

Compare output image and reference image

Figure 11 Defect inspection flow


TrainingValidation

000

200

400

600

800

1000

1200Lo

ss

20 40 60 80 1000Epoch

(a)

TrainingValidation

000

020

040

060

080

100

120

Acc

urac

y

20 40 60 80 1000Epoch

(b)

Figure 12 (a) Network loss and (b) network accuracy of the training and validation sets

TrainingTesting

000

200

400

600

800

1000

1200

1400

Loss

20 40 60 80 1000Epoch

(a)

TrainingTesting

000

020

040

060

080

100

120

Acc

urac

y

20 40 60 80 1000Epoch

(b)

Figure 13 (a) Network loss and (b) network accuracy of the test set

(a) (b)

Figure 14 Continued


F1 minus score 2precision times recallprecision + recall

(7)

In all cases of methods the recall values are close to 1which indicates that all PSs that are actually normal are

correctly determined to be normal Image similarity com-parison methods and the CAE method are stable in thenormal PS determination In the HC method precision andaccuracy are lowest since the color information of the ab-normal PS is combined by a histogram e HC method can

(c) (d)

Figure 14 Result of abnormal PSs through CAE

(a) (b)

(c) (d)

Figure 15 Result of difference operation between the reference PS and the abnormal PS


be used to determine the approximate defect judgement ofthe PS in a short time but cannot identify the location of thedefect in each abnormal PS TM and FM methods showhigher inspection performance than the HC method Bothmethods are more accurate than the HC method but theycan only extract the presence or absence of defects like theHC method is is a limitation of the image similaritycomparison methods In the case of precision recall ac-curacy and F1-score the proposed CAE method generatedbetter results e CAE method can accurately judge defectsand at the same time identify the location of defects in theimage through subtraction with the test image Since imagesimilarity comparison methods use the entire templateimage and compare the similarity without alignment it ishard to determine the defect of the small part e CAEmethod is more advantageous than other methods for defectjudgement of the small part since the PS to be analyzed isaligned through the auto encoder and then the similarity iscompared

If such a CAE network was trained in advance it ispossible to find defects by inserting a test image into anetwork and comparing the output image with a referenceIn contrast image similarity comparison methods comparethe degree of similarity with the registered template for eachhistogram pixel or feature of the image Image similaritycomparisonmethods require a longer analysis time when thesize of the test image or template becomes larger Table 3shows the inspection times for seven different sizes of testimages

Although the HC method has the shortest inspectiontime the accuracy of the HC method is low and is notsuitable for use in actual manufacturing processes eTMmethod FM method and CAE method can be used toinspect defects because of their high inspection accuracyAmong these the FM method requires more than twicethe amount of inspection time as compared to the othermethods Inspection time is an important parameter inthe manufacturing process Considering defect inspectiontime and accuracy the TM method and CAE method canbe applied to equipment used in the manufacturing

process When image analysis is performed on a smallerimage size the inspection speed of the TM method isfaster In contrast when large image analysis is per-formed the inspection speed of the proposed CAEmethod is faster than the TM method With the largestimage size of 4000 times 3000 the inspection time of the CAEmethod was reduced by 79 as compared to the TMmethod e inspection area is increased in themanufacturing process and cameras with adequate res-olution are predominately used In other words situationsthat require analysis of images with a large pixel size arefrequently encountered thus using the proposed CAEmethod can increase the process yield by shortening theinspection time

5 Conclusion

is paper proposed the CAE method for inspection of PSdefects in the display panel e CAE method has twocharacteristics First unaligned images can be moved intothe same alignment position which allows for the mea-sured PS images to be moved to the same position so thatthe images can be directly compared Second the char-acteristics of the abnormal PS are maintained even if thePS is aligned by the CAE method e abnormal PS ob-tained through CAE has the same alignment as the ref-erence PS and has its abnormal characteristics esecharacteristics of CAE allow for the inspection of defectsin the PS In order to confirm the performance of PS defectinspection using CAE image similarity comparisonmethods and other defect inspection methods werecompared e abnormal PS defect inspection was per-formed through CAE and the confusion matrix and theF1-score were calculated from the inspection resultsResults of this study are summarized below

e CAE method permits movement of the position ofnonaligned PSs e position of PSs that were 37 μm awayfrom the center was moved to the center of the image usingthe CAE method in these experiments Even if the PS wasmoved the characteristics of normal and abnormal PSs weremaintained thus the CAE method can judge defects bycomparing the aligned output image and the referenceimage

e CAE and image similarity comparison methods areuseful for determining a normal PS according to eachconfusion matrix e CAE method is more advantageousthan the image similarity comparison methods for de-termining an abnormal PS since the CAE method candetect small defects during abnormal PS inspection Inaddition F1-score comparison confirmed that the CAEmethod has higher precision and recall values and showedsuperior results of defect inspection In particular the CAEmethod is more effective since the CAE method can an-alyze not only the PS defect but also the location of the PSdefect

As with the image similarity comparison methods theinspection time of the CAE method increased as the size of

Table 1 Confusion matrix of the CAE method

Concentrated auto encoder PredictedNormal Abnormal

Observed Normal 100 0Abnormal 1 99

Table 2 Inspection results between CAE and image similaritycomparison methods

Method Precision Recall Accuracy F1-scoreHC 0904 0940 0920 0922TM 0926 1000 0960 0943FM 0951 0980 0965 0966CAE 0990 1000 0995 0993


test images increased Based on the accuracy of inspectionand inspection time the TM and CAE methods are ap-propriate for manufacturing process equipment e in-spection time of the CAE method and TMmethod was mostsimilar with image sizes of 1280times1024 and 1920times1080 eTMmethod performed better with smaller image sizes whileperformance of the CAE method with large image sizes wasbetter in terms of inspection time It is useful to use the CAEmethod in the manufacturing process to reduce inspectiontime because inspection of large-sized images is frequentlyperformed in order to increase inspection yield

Data Availability

e data used to support the findings of this study are in-cluded within the article

Conflicts of Interest

e authors declare that they have no conflicts of interest

Acknowledgments

is work was supported in part by the Brain Korea 21 PlusInstitute of Engineering Research Institute of AdvancedMachines and Design at Seoul National University

References

[1] L A J Davis D R Billson D A Hutchins and R A NobleldquoVisualizing acoustic displacements of capacitive micro-machined transducers using an interferometric microscoperdquoAcoustics Research Letters Online vol 6 no 2 pp 75ndash792005

[2] S Petitgrand R Yahiaoui K Danaie A Bosseboeuf andJ P Gilles ldquo3D measurement of micromechanical devicesvibration mode shapes with a stroboscopic interferometricmicroscoperdquo Optics and Lasers in Engineering vol 36 no 2pp 77ndash101 2001

[3] C Rembe R Kant and R S Muller ldquoOptical measurementmethods to study dynamic behavior in MEMSrdquo in Lasers inMetrology and Art Conservation pp 127ndash137 InternationalSociety for OptiPS and PhotoniPS Bellingham WA USA2001

[4] J A Conway J V Osborn and J D Fowler ldquoStroboscopicimaging interferometer for MEMS performance measure-mentrdquo Journal of Microelectromechanical Systems vol 16no 3 pp 668ndash674 2007

[5] A Bosseboeuf and S Petitgrand ldquoCharacterization of thestatic and dynamic behaviour of M(O)EMS by optical tech-niques status and trendsrdquo Journal of Micromechanics andMicroengineering vol 13 no 4 pp S23ndashS33 2003

[6] S H Chen H S Koo W Y Chen C H Kang andD Y Goang ldquoP-170 advanced photo spacer technology for

large-sized TFT-LCDrdquo SID Symposium Digest of TechnicalPapers vol 36 no 1 pp 539ndash541 2005

[7] D H Ku S M Lee H N Roh T W Kim and H J PahkldquoImprovement of PS measurement by phase compensationmethod and profile fittingmethod in white light phase shiftinginterferometryrdquo Current Optics and Photonics vol 2 no 4pp 340ndash347 2018

[8] T Jo S Kim and H Pahk ldquo3D measurement of TSVs usinglow numerical aperture white-light scanning interferometryrdquoJournal of the Optical Society of Korea vol 17 no 4pp 317ndash322 2013

[9] C Hyun S Kim and H Pahk ldquoMethods to measure thecritical dimension of the bottoms of through-silicon vias usingwhite-light scanning interferometryrdquo Journal of the OpticalSociety of Korea vol 18 no 5 pp 531ndash537 2014

[10] S-H Park J-H Lee and H-J Pahk ldquoIn-line critical di-mension measurement system development of LCD patternproposed by newly developed edge detection algorithmrdquoJournal of the Optical Society of Korea vol 17 no 5pp 392ndash398 2013

[11] D Van der Weken M Nachtegael and E E Kerre ldquoUsingsimilarity measures for histogram comparisonrdquo in LectureNotes in Computer Science vol 2715 pp 396ndash403 2003

[12] T Fober and E Hullermeier ldquoSimilarity measures for proteinstructures based on fuzzy histogram comparisonrdquo in Pro-ceedings of the IEEE World Congress on Computational In-telligence pp 18ndash23 Barcelona Spain July 2010

[13] J P Lewis ldquoFast template matchingrdquo in Proceedings of theConference on Vision Interface pp 120ndash123 Quebec CityCanada May 1995

[14] A Sibiryakov ldquoFast and high-performance template matchingmethodrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR) pp 1417ndash1424Colorado Springs CO USA June 2011

[15] H Yang and YWang ldquoA LBP-based face recognition methodwith Hamming distance constraintrdquo in Proceedings of the 4thInternational Conference on Image and Graphics (ICIGrsquo07)pp 645ndash649 Washington DC USA August 2007

[16] D G Lowe ldquoDistinctive image features from scale-invariantkeypointsrdquo International Journal of Computer Vision vol 60no 2 pp 91ndash110 2004

[17] E Rublee V Rabaud K Konolige and G Bradski ldquoORB anefficient alternative to SIFT or SURFrdquo in Proceedings of theIEEE International Conference on Computer Vision (ICCV)pp 2564ndash2571 Barcelona Spain November 2011

[18] Y Bengio P Lamblin D Popovici and H LarochelleldquoGreedy layer-wise training of deep networksrdquo in Proceedingsof the Advances in Neural Information Processing Systems(NIPSrsquo06) pp 153ndash160 Vancouver Canada December 2007

[19] P Vincent H Larochelle Y Bengio and P A ManzagolldquoExtracting and composing robust features with denoisingautoencodersrdquo in Proceedings of the 25th InternationalConference on Machine Learning (ICMLrsquo08) pp 1096ndash1103Helsinki Finland July 2008

Table 3 Inspection time for CAE and image similarity comparison methods

Method 640times 480 880times 640 1280times1024 1920times1080 1920times1440 3000times 2000 4000times 3000HC (ms) 28 54 128 251 280 585 1041TM (ms) 49 88 230 551 604 1466 2652FM (ms) 99 196 460 714 1024 2504 5101CAE (ms) 60 105 258 499 544 1180 2090


[20] X Lu Y Tsao S Matsuda and C Hori ldquoEnsemble modelingof denoising autoencoder for speech spectrum restorationrdquo inProceedings of the Interspeech pp 885ndash889 Singapore Sep-tember 2014

[21] A Shantia R Timmers L Schomaker and M WieringldquoIndoor localization by denoising autoencoders and semi-supervised learning in 3D simulated environmentrdquo in Pro-ceedings of the International Joint Conference on NeuralNetworks (IJCNN) pp 1ndash7 Killarney Ireland July 2015

[22] R Wang and D Tao ldquoNon-local auto-encoder with collab-orative stabilization for image restorationrdquo IEEE Transactionson Image Processing vol 25 no 5 pp 2117ndash2129 2016

[23] X Feng Y Zhang and J Glass ldquoSpeech feature denoising anddereverberation via deep autoencoders for noisy reverberantspeech recognitionrdquo in Proceedings of the IEEE ICASSP 2014pp 1759ndash1763 Florence Italy May 2014

[24] K Wu Z Gao C Peng and X Wen ldquoText window denoisingautoencoder building deep architecture for Chinese wordsegmentationrdquo Communications in Computer and In-formation Science vol 400 pp 1ndash12 2013


Hindawiwwwhindawicom Volume 2018

Active and Passive Electronic Components


Shock and Vibration


High Energy PhysicsAdvances in

Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom

The Scientific World Journal

Volume 2018

Acoustics and VibrationAdvances in



Advances in Condensed Matter Physics

OpticsInternational Journal of



AstronomyAdvances in

Antennas andPropagation

International Journal of




Geophysics

Advances inOpticalTechnologies

Hindawiwwwhindawicom

Volume 2018

Applied Bionics and BiomechanicsHindawiwwwhindawicom Volume 2018

Advances inOptoElectronics


Volume 2018


Mathematical PhysicsAdvances in


ChemistryAdvances in


Journal of

Chemistry


Advances inPhysical Chemistry


RotatingMachinery



Journal ofEngineeringVolume 2018

Submit your manuscripts atwwwhindawicom

histogram comparison (HC) template matching (TM) andfeature matching (FM) methods ese methods are used todetermine whether the normal PS template to be inspectedexists in the test image If there is a similar normal PStemplate the similarity is outputted If the degree of simi-larity is high PS is considered to be normal while if thedegree of similarity is low it is possible to determine that thedefective shape is an abnormal PS

In this paper the CAE method is proposed which can beused as a replacement for the existing image similarity com-parison method that is used for alignment and defect inspectionof PS images acquired from an interferometry CAE allowsmeasured PS images to be aligned e shape of the PS wasinspected and the PS defect was checked by comparing thereference PS image and the PS image aligned through theCAE Itis possible to identify defects and their location without preciselymeasuring the height of the PS and CD with the CAE methodand it is also possible to reduce inspection time by simultaneouslyaligning and analyzing defects Finally the performance of theproposed CAE method was veried by comparing it with theexisting image similarity comparison methods

2 Theoretical Background

21 Auto Encoder e auto encoder is a learning networkbased on deep learning which learns functions that make theoutput value similar to the input valuee auto encoder hastwo neural networks as shown in Figure 1 e neuralnetwork located at the front is an encoder and the networkbehind it is a decodere encoder extracts the feature of theinput data and the original data are regenerated by usingthis feature and the decoder By training the input andoutput data to be as equal as possible the auto encoder canextract the feature eciently [18] e auto encoder is usedto remove and restore noise and image noise and can be usedfor image classication [19ndash24] As the amount of researchon sensor data utilization and analysis has increased inrecent years it is also possible to use an auto encoder as acorrection technique for sensor data

A typical auto encoder is an articial neural network thatmakes output to be input e target output is used as aninput e encoder is a function of h fθ(x) s(Wx + b)which extracts the hidden feature h isin [0 1]d from the inputvector x isin [0 1]D In this equation θ W b W is d timesDdimension weight matrix and b means d dimension biasvector s is an element-wise activation function that makes anetwork deeper by changing input data so that it is non-linear A sigmoid function or a rectied linear unit (ReLu) isused as an activation function e formula of the sigmoidfunction is shown in equation (1) Feature h that is calculatedin the hidden layer is converted into vector z isin [0 1]D by thedecoder z gθprime(h) s(Wprimeh + bprime) e auto encoder per-forms the process of extracting and reconstructing thefeature from the input data through the encoder and de-coder In conclusion the input x is converted to h throughthe hidden layer and the nal output z is reconstructed fromh Each input x is mapped to the related feature h which isthen mapped to a reconstruction z which satises x asymp z eauto encoder must mathematically minimize the dierence

between input x and output z e dierence between theinput x and the output z is termed loss which is also calledthe average reconstruction error and the goal of the autoencoder is to optimize the parameter θlowast θprimelowast to minimize thisloss e expression of the loss optimization parameterθlowast θprimelowast is shown in equation (2) and loss is calculated withequation (3) Loss function is a scalar function that calculatesthe dierence between input and output and is used as anindicator to evaluate the dierence of datasets Meansquared error and cross entropy error are mainly used as aloss function e loss function L in equation (3) is the mostcommonly used mean squared error function

s(x) 1

1 + eminus x (1)

θlowast θprimelowast( ) argminθθprime1nsumn

i1L x(i) z(i)( ) (2)

L(x z) x minus z2 x minus s Wprime(s(Wx + b)) + bprime( )

2 (3)

22 PS Defect Inspection Using Interferometry Inspection ofthe PS defect of the display panel is performed using a 3Dshape and a 2D tomographic image obtained from an in-terferometry It is possible to measure the height from the3D shape and it is possible to measure the axial CD of the PSfrom the 2D tomographic image [7ndash9] If the dierencesbetween the height or CD of the measured PS and thereference PS values are larger than the threshold it is judgedto be an abnormal PS is PS defect inspection methodconrms the measured value of height x and y axis CDassuming that the shape of the PS is bell shaped Defects indirections other than the x and y axis cannot be detectedusing the PS defect inspection method If these precisemeasured values are not needed and only the presence ofdefects is required the image similarity comparison methodcan be used e HC method [11 12] the TM method[13ndash15] and the FM method [16 17] are used in this paperfor comparing image similarity Image similarity compari-son methods require a test image and a template image fordetection With image similarity comparison methods auser can determine if the template image that the user is

x1

x2

x3

xD

z1

z2

z3

zD

h1

h2

hd


W Wprime

Figure 1 Structure of an auto encoder





H 1113944n

i1f xi( 1113857 minus f yi( 1113857

11138681113868111386811138681113868111386811138681113868 (4)

H(x y) 1113944

xprime yprime


(5)













0 50 100 150 200 2500

5000

10000

15000

20000

25000

30000

35000

40000

(a)

0 50 100 150 200 2500

5000

10000

15000

20000

25000

30000

35000

40000

(b)













(6)




x1

x2

x3

xD

z1

h1

h2

hd


W Wprime





(a) (b)


Encoder Decoder

d = 1 d = 32 d = 64 d = 64 d = 32 d = 1d = 128

D images 1 image

W H W2 H2 W4 H4 W8 H8 W4 H4 W2 H2 W H


(a) (b) (c) (d)


(a) (b) (c) (d)



(a) (b)

(c) (d)

(e) (f )



Train network

Input test image


Interferometry





Yes

No




TrainingValidation

000

200

400

600

800

1000

1200Lo

ss

20 40 60 80 1000Epoch

(a)

TrainingValidation

000

020

040

060

080

100

120

Acc

urac

y

20 40 60 80 1000Epoch

(b)


TrainingTesting

000

200

400

600

800

1000

1200

1400

Loss

20 40 60 80 1000Epoch

(a)

TrainingTesting

000

020

040

060

080

100

120

Acc

urac

y

20 40 60 80 1000Epoch

(b)


(a) (b)

Figure 14 Continued



(7)



(c) (d)


(a) (b)

(c) (d)







5 Conclusion












Data Availability




Acknowledgments


References

































Shock and Vibration





Volume 2018














Geophysics



Volume 2018




Volume 2018






Journal of

Chemistry




RotatingMachinery








H 1113944n

i1f xi( 1113857 minus f yi( 1113857

11138681113868111386811138681113868111386811138681113868 (4)

H(x y) 1113944

xprime yprime


(5)













0 50 100 150 200 2500

5000

10000

15000

20000

25000

30000

35000

40000

(a)

0 50 100 150 200 2500

5000

10000

15000

20000

25000

30000

35000

40000

(b)













(6)




x1

x2

x3

xD

z1

h1

h2

hd


W Wprime





(a) (b)


Encoder Decoder

d = 1 d = 32 d = 64 d = 64 d = 32 d = 1d = 128

D images 1 image

W H W2 H2 W4 H4 W8 H8 W4 H4 W2 H2 W H


(a) (b) (c) (d)


(a) (b) (c) (d)



(a) (b)

(c) (d)

(e) (f )



Train network

Input test image


Interferometry





Yes

No




TrainingValidation

000

200

400

600

800

1000

1200Lo

ss

20 40 60 80 1000Epoch

(a)

TrainingValidation

000

020

040

060

080

100

120

Acc

urac

y

20 40 60 80 1000Epoch

(b)


TrainingTesting

000

200

400

600

800

1000

1200

1400

Loss

20 40 60 80 1000Epoch

(a)

TrainingTesting

000

020

040

060

080

100

120

Acc

urac

y

20 40 60 80 1000Epoch

(b)


(a) (b)

Figure 14 Continued



(7)



(c) (d)


(a) (b)

(c) (d)







5 Conclusion












Data Availability




Acknowledgments


References

































Shock and Vibration





Volume 2018














Geophysics



Volume 2018




Volume 2018






Journal of

Chemistry




RotatingMachinery











0 50 100 150 200 2500

5000

10000

15000

20000

25000

30000

35000

40000

(a)

0 50 100 150 200 2500

5000

10000

15000

20000

25000

30000

35000

40000

(b)













(6)




x1

x2

x3

xD

z1

h1

h2

hd


W Wprime





(a) (b)


Encoder Decoder

d = 1 d = 32 d = 64 d = 64 d = 32 d = 1d = 128

D images 1 image

W H W2 H2 W4 H4 W8 H8 W4 H4 W2 H2 W H


(a) (b) (c) (d)


(a) (b) (c) (d)



(a) (b)

(c) (d)

(e) (f )



Train network

Input test image


Interferometry





Yes

No




TrainingValidation

000

200

400

600

800

1000

1200Lo

ss

20 40 60 80 1000Epoch

(a)

TrainingValidation

000

020

040

060

080

100

120

Acc

urac

y

20 40 60 80 1000Epoch

(b)


TrainingTesting

000

200

400

600

800

1000

1200

1400

Loss

20 40 60 80 1000Epoch

(a)

TrainingTesting

000

020

040

060

080

100

120

Acc

urac

y

20 40 60 80 1000Epoch

(b)


(a) (b)

Figure 14 Continued



(7)



(c) (d)


(a) (b)

(c) (d)







5 Conclusion












Data Availability




Acknowledgments


References

































Shock and Vibration





Volume 2018














Geophysics



Volume 2018




Volume 2018






Journal of

Chemistry




RotatingMachinery












(6)




x1

x2

x3

xD

z1

h1

h2

hd


W Wprime





(a) (b)


Encoder Decoder

d = 1 d = 32 d = 64 d = 64 d = 32 d = 1d = 128

D images 1 image

W H W2 H2 W4 H4 W8 H8 W4 H4 W2 H2 W H


(a) (b) (c) (d)


(a) (b) (c) (d)



(a) (b)

(c) (d)

(e) (f )



Train network

Input test image


Interferometry





Yes

No




TrainingValidation

000

200

400

600

800

1000

1200Lo

ss

20 40 60 80 1000Epoch

(a)

TrainingValidation

000

020

040

060

080

100

120

Acc

urac

y

20 40 60 80 1000Epoch

(b)


TrainingTesting

000

200

400

600

800

1000

1200

1400

Loss

20 40 60 80 1000Epoch

(a)

TrainingTesting

000

020

040

060

080

100

120

Acc

urac

y

20 40 60 80 1000Epoch

(b)


(a) (b)

Figure 14 Continued



(7)



(c) (d)


(a) (b)

(c) (d)







5 Conclusion












Data Availability




Acknowledgments


References

































Shock and Vibration





Volume 2018














Geophysics



Volume 2018




Volume 2018






Journal of

Chemistry




RotatingMachinery







(a) (b)


Encoder Decoder

d = 1 d = 32 d = 64 d = 64 d = 32 d = 1d = 128

D images 1 image

W H W2 H2 W4 H4 W8 H8 W4 H4 W2 H2 W H


(a) (b) (c) (d)


(a) (b) (c) (d)



(a) (b)

(c) (d)

(e) (f )



Train network

Input test image


Interferometry





Yes

No




TrainingValidation

000

200

400

600

800

1000

1200Lo

ss

20 40 60 80 1000Epoch

(a)

TrainingValidation

000

020

040

060

080

100

120

Acc

urac

y

20 40 60 80 1000Epoch

(b)


TrainingTesting

000

200

400

600

800

1000

1200

1400

Loss

20 40 60 80 1000Epoch

(a)

TrainingTesting

000

020

040

060

080

100

120

Acc

urac

y

20 40 60 80 1000Epoch

(b)


(a) (b)

Figure 14 Continued



(7)



(c) (d)


(a) (b)

(c) (d)







5 Conclusion












Data Availability




Acknowledgments


References

































Shock and Vibration





Volume 2018














Geophysics



Volume 2018




Volume 2018






Journal of

Chemistry




RotatingMachinery





(a) (b)

(c) (d)

(e) (f )



Train network

Input test image


Interferometry





Yes

No




TrainingValidation

000

200

400

600

800

1000

1200Lo

ss

20 40 60 80 1000Epoch

(a)

TrainingValidation

000

020

040

060

080

100

120

Acc

urac

y

20 40 60 80 1000Epoch

(b)


TrainingTesting

000

200

400

600

800

1000

1200

1400

Loss

20 40 60 80 1000Epoch

(a)

TrainingTesting

000

020

040

060

080

100

120

Acc

urac

y

20 40 60 80 1000Epoch

(b)


(a) (b)

Figure 14 Continued



(7)



(c) (d)


(a) (b)

(c) (d)







5 Conclusion












Data Availability




Acknowledgments


References

































Shock and Vibration





Volume 2018














Geophysics



Volume 2018




Volume 2018






Journal of

Chemistry




RotatingMachinery





TrainingValidation

000

200

400

600

800

1000

1200Lo

ss

20 40 60 80 1000Epoch

(a)

TrainingValidation

000

020

040

060

080

100

120

Acc

urac

y

20 40 60 80 1000Epoch

(b)


TrainingTesting

000

200

400

600

800

1000

1200

1400

Loss

20 40 60 80 1000Epoch

(a)

TrainingTesting

000

020

040

060

080

100

120

Acc

urac

y

20 40 60 80 1000Epoch

(b)


(a) (b)

Figure 14 Continued



(7)



(c) (d)


(a) (b)

(c) (d)







5 Conclusion












Data Availability




Acknowledgments


References

































Shock and Vibration





Volume 2018














Geophysics



Volume 2018




Volume 2018






Journal of

Chemistry




RotatingMachinery






(7)



(c) (d)


(a) (b)

(c) (d)







5 Conclusion












Data Availability




Acknowledgments


References

































Shock and Vibration





Volume 2018














Geophysics



Volume 2018




Volume 2018






Journal of

Chemistry




RotatingMachinery









5 Conclusion












Data Availability




Acknowledgments


References

































Shock and Vibration





Volume 2018














Geophysics



Volume 2018




Volume 2018






Journal of

Chemistry




RotatingMachinery






Data Availability




Acknowledgments


References

































Shock and Vibration





Volume 2018














Geophysics



Volume 2018




Volume 2018






Journal of

Chemistry




RotatingMachinery














Shock and Vibration





Volume 2018














Geophysics



Volume 2018




Volume 2018






Journal of

Chemistry




RotatingMachinery








Shock and Vibration





Volume 2018














Geophysics



Volume 2018




Volume 2018






Journal of

Chemistry




RotatingMachinery





Documents

DefectInspectioninDisplayPanelUsingConcentrated …downloads.hindawi.com/journals/ijo/2019/8039267.pdfHamming distance is used to calculate similarity. e Hamming distance is used to