4/2014TechnologiaiAutomatyzacjaMontażu

56

IMPLEMENTATION OF AUGMENTED REALITY IN WELDING PROCESSES

Dorota STADNICKA, Dario ANTONELLI

A b s t r a c tWeldingprocessesarespecialprocessesandspecialattentionshouldbeputonthemwhenwearetalkingaboutthequalityassuranceproblems.StandardqualitymanagementsystemsbasedonISOstandardsindicateareasonwhichthisspecialat-tentionshouldbeput,inordertoobtainresultswhichwillmeettherequirements.Thus,itisnecessarytoensuretheadequateemployees’preparation,adequatecontrolofequipmentusedinaweldingprocess,adequateworkmethodsandsystemssup-portingawelderintheerror-freeweldingprocessrealization.Inthispaper,applyingaugmentedrealityinordertosupportaspotweldingprocessisdescribed.Thesolutionpresentedissimpleandmakesuseoflowcostdevices.Atthesametime,aweldertrainingtimecanbeshortenedthroughtheuseoftheproposedsolution.Thetestsofworkperformedwiththeuseofthepro-posedsolutiongavegoodresultsallowinganoperatortoavoidmistakesduringaweldingprocessperforming.Simultaneously,shorteningoftheprocessrealizationtimeisobtained.

K e y w o r d smistakeproofing,qualityassurance,specialprocesses,AugmentedReality

Introduction

Nowadays,manufacturing systems facemany prob-lemswhichareaconsequenceofamarketsituation.Ifacompanywantstobecompetitiveitmustensureashortleadtime,agoodpriceandagoodqualityofaproductorservice.

Itisparticularlyimportanttoensureadequatecontrolof the special processes such as welding processes,whichdependonmanydifferentfactorsandrequiresuit-ablepreparationaswellaspropercontrolduringtheirex-ecution.Inordertoensuretheabove,acompanyhasto:•• Planspecialprocessesproperly, inparticularwhen

theyarecomplex[1],•• Placeemphasisontheadequatesuppliersselection,

tobeabletodependonthemnotonlyinthematterofqualitybutalsoofdeliveriespunctuality[2],

•• Monitorprocessesinordertomanufactureaproductinagoodwayfromthefirsttime,nottocorrectthem,sinceittakestimeandconsumesmoney[3],

•• Optimize costs by the process andwelderswork’squalityimprovement[4],

•• Implementandmaintainanefficientqualitymanage-mentsystem[5].

Inthiswork,theaugmentedrealityimplementationinaqualitydevelopmentofaweldingprocessrealizationisdescribed.Thepresentedmethodsupports theweldingprocessrealizedbyaweldertodecreasethepossibilityofmakingmistakesduringtheprocessandtomaketheprocess more efficient.

Quality assurance in a welding process

Weldingprocessesarespecialprocessesandaspe-cialprocedureisneededinordertoensuretheirquality.

Theseprocessesarecharacterizedbydifficultieswithas-sessingaresultoftheprocessduringtheprocess.ThestandardISO9001:2008tellsthatwhentheresultsoftheprocesscan’tbeverifiedbymonitoringormeasurementand,inconsequence,mistakescannotbedisclosedbe-foretheutilization,thevalidationoftheprocessshouldbeperformed.Thevalidationoftheprocessesshouldrevealtheirabilitytoobtainplannedresults.

An organization should determine settlements con-cerningtheseprocesses.Whenthesettlementsarecon-cerned,theyshouldinclude:criteriaregardingaprocessreviewandapproval,equipmentapproval,personnelse-lection,usingspecifiedmethods,developmentandem-ploymentofprocedures,indicatingrecordsnecessarytobekept, aswell as the informationon thenecessityofnextvalidation.

The criteria concerning the process review and ap-provalshouldbeconnectedtotheproductoftheprocessassessment which are performed in the validated pro-cessanditscharacteristics.

Theequipment,which isgoing tobeassignedto theprocess,shouldensurethegoodqualityandrepeatabilityoftheobtainedresults.Thatmeanstoensurestablework.Theequipmentshouldbecontrolledinaproperway,forexampleinTPM(Total Productive Maintenance)system.Thelevelofcontrolandthenecessarymaintenanceac-tivities can be determinedwith the use of amethod of amachinecategorizationproposedinwork[6].

Personnel selection is connected with choosing ad-equatelyqualifiedpeopletoperformthespecialprocess.Theselectionshouldbemadeonthebasisofanopera-tor’seducation,experience,trainingsandskills.

Usingspecifiedmethodsmeans, firstofall,develop-ing themadequately to theprocess,aswellas theac-ceptedprocess results,equipmentused in theprocessandanoperator’scompetence.Themethodsusedmay

TiAM_4_2014.indd 56 2014-11-25 12:52:56

TechnologiaiAutomatyzacjaMontażu4/2014

57

alsoconcerntools,devices,measuringinstrumentsusedin the process, environmental conditions which shouldbemaintained during the process, and activitieswhichshouldbeperformedduringtheprocess.

Developingandusingproceduresisalsoindispensa-bletoensuretherepeatabilityoftheprocess.Theproce-duresshouldcontainanswersforthefollowingquestions:Whocanperformtheprocess?Whatexactlyshouldbedone?Inwhatordertheactivitiesshouldbeperformed?Whatkindofequipmentshouldbeused?Whatkindofactivitiesshouldbedonebeforetheprocessstarts?Whatismost importantduring theprocess realization?Whatkindofactivitiesshouldbedoneafter finishing thepro-cess?What kind of the detailed instructions should beused in theprocessand in the treatingof theproduct?Whatisthewayforprocessdocumentingandwhatkindofdataareimportantfromtheevaluationofcorrectnessof the process realization point of view?What kind of asignalcanshowthelackofcorrectnessoftherealizedprocess?Whatkindofactivitiesanoperatorshouldun-dertakeincaseofnoticingsymptomsshowingthelackofcorrectnessoftherealizedprocessorincaseoffailures?

The most variable element of a welding process isthe human operator [7], under the condition that otherelementsoftheprocessarecontrolledinaproperway.Amongthemaincausesofthat,wecanenumerate:thelackofpossibilityofachievingahundredpercentofcon-centration during the whole working day, tiredness of aworker,alowlevelofmotivationtowork,notsufficientunderstanding of procedures and lack of information,whatleadstothewelder’smistakes.

That is why a created quality system and the tech-niques used should support the worker in the weldingprocesses realization.

In order to ensure the quality ofwelding processes,ISO3834standards[8]–[11]arerecommended.TheycanbeusedseparatelyortogetherwithISO9001.Inthesestandards,therequirementsconcerningareassuchasthereviewofrequirementsandatechnicalreview,sub-contractors, welders, personnel formeasurements andcontrol, non-destructive testing, welding equipment,welding techniques, storage and maintenance of ad-ditional materials, heat treatment after welding, meas-urements and control, nonconformities and correctiveactions,calibrationandmeasurementequipmentvalida-tion,identificationandtraceabilityandqualityrecordsareincluded.

Inordertoensurethequalityofweldingprocessesthefollowingactivitiescanbeperformed:•• Controloftheinputmaterialor/andevaluationofsup-

plierqualitymanagementsystems,

•• Periodical control of welding equipment technicalcondition,

•• Monitoringtheconditionsofaweldingprocess(tem-perature,humidity,noise,vibration,etc.)whichcaninfluenceboththeprocessandthewelder,

•• Assessmentofthewelders’workorganization(scopeofwork,workingtime,breaks,etc.),

•• Evaluation of the welders’ work preparation (train-ings, experience, knownmethods, increasing com-petence),

•• Workmethodsusedandtheirstandardization(workinstructions,standarddevices,holders,etc.),

•• Systems supporting the welding process (weldingpath monitoring, augmented reality, control of theprocessparameters).

In thefurtherpartof thepaper,awayforsupporting awelderinproductiontasksrealizationisdescribed.Thepresentedmethodallowstominimizeanumberofmis-takesmadebyanoperatoranditcanbeoneofimportantelementsofthequalityassuranceinaweldingprocess.

Case study: Augmented Reality to improve the execution of spot welding

Resistancespotwelding is commonlyapplied in theautomotivesectortojointhepartsofthecarbodysinceit isa flexible,consistentandaboveallacost-effectiveprocess. It is applied both tomanual and to automaticassembly.

Thequalityassuranceisachievedbyacarefulchoiceof the electric parameters operated by welding qualitycontrolsystems,asdescribedin[12]–[15].Unfortunatelythesemonitoringsystemsareusefulonly forautomaticSpotWelding,astheyrelyonthepossibilityof interact-inginrealtimewiththewelderandontheexactknowl-edgeofwherewillbe thenextweldedpoint.Asolutiontoimprovethemanualweldingshouldpassthroughtheempowermentofthehumanoperator.Weshouldtransferhim/herthesameinformationthatwouldbeavailabletoarobotinafullyautomaticcell.Ourresearchaimsatex-ploitAugmentedReality(AR)by:•• directing the operator to the exact pointwhere the

weldinghastobeperformed;

•• preventing selection errors of thewelding programbycontrollingtheweldingparameterswiththesamecontrol unit used by the robot (possible only if thepositionoftheweldedspotisknown);

•• allowing,tocustomizetheparametersofeverysinglepointtooptimizethespotquality;

•• ensuring, through the use of the Welding QualitySystem(WQS),therespectof jointrequirementsinterms of mechanical resistance and metallurgicalproperties;

•• notifyingtheoperatorandthesupervisionsystemofbadspots;

•• preventingweldinguntiltheclampisnotonthecor-rectpositionandalignment.

TiAM_4_2014.indd 57 2014-11-25 12:52:56

4/2014TechnologiaiAutomatyzacjaMontażu

58

State of the art and known issues of manual welding

Spot welding in automatic and manual assembly stationsThe spot welding process consists of forcing the

passageofhighcurrent(6–15kA)betweentwometalsheetspressedbytwoelectrodesmadeofacopperalloyCuCrZr.Theshapeoftheelectrodesdeterminespositionandsizeofthejoint.

Theweldingsystemmakesuseofacurrenttransform-erfedatmainsfrequency(typically380–500V50/60Hz)whosesecondary,thankstothereducednumberofcoilsiscapabletodeliverthehighcurrentsnecessaryforweld-ing.Inthecaseofmedium-frequencywelding,thepowerof the transformer is instead obtained by using a Me-diumFrequency Inverter that transformsa three-phasemainsvoltage(380V–690V50/60Hz)intoavoltageat 1000Hz–1800Hzfrequency,withsignificantadvantag-esbothfromthepointofviewofenergyefficiencyandofcurrentcontrol.

The proposed solutionInordertostatewhatkindandlevelofassistanceis

requiredbyweldingoperator,anextensivesetofobser-vations and interviewswere performed atGF-Welding, a tool maker.

Theworkflowof the activities performedduring thewelding process is described here.The operator holdstheelectrodesinareferencepointoutsidetheoperationtheatre,thenmovethetooltothefirstpointwitharightpose.Theelectrodesareopenedinthewiderpositiontofacilitate insertionon thesheet.Theyareclosed to theworkingconfiguration(minimumopening).Whenthetoolispositionedcorrectlyandcenteredexactlyontheweld-ingspot,theoperatorselectstheweldingprogram(inten-sityofcurrent,weldingtimeandpressureovertheelec-trodes).Thentheweldingstarts.Thetoolreopensand,ifthere isroomonthesheet, theoperatorrepeatsanewpoint.Eventually,theoperatormovesbackthetooltotheinitial reference position.

Thedesiredactionsare:simplifytheoperator’swork,increase productivity, improve the repeatability of thewelding points, and improvewelding quality.Themereintroduction of AR in the process addresses some oftheobjectives.Thequalitycanbeimprovedonlybyac-tivationofWQSalsointhemanualwelding.Thesystem

canworkonlyiftheexactpositionoftheweldedspotisknown,becauseeverypointhasdifferentexecutionpa-rameters.ThepositioncanbefoundasasideeffectoftheuseofAR.Inordertosuperimposevirtualobjectstothe realworld, it isnecessary to locatewith fairlygoodaccuracythepositionofboththetoolandofthedisplay.Thisdatumcouldbepassed toWQSthat therefore re-ceivesindirectlytheknowledgeofpositionwherethenextpointisgoingtobejoined.

Oncestatedthescopeoftheapplication,itistimetodefineexactlythesetofmessagesthatwillbedisplayedtoassisttheweldingoperator.DuetothepeculiarnatureoftheAR,theformatweusetodisplaythemessageisasmuchimportantasthedatumitself.AtypicaldefectofmostARsystemsistheoverflowofinformation.Thein-terfaceprovidestheoperatortoomuchfactsandfigures,distractinghimfromthetask.Theweldingoperatorhasafewsecondsbeforemovingtothenextpoint,thereforethemessagesshouldbefew,ofimmediateunderstand-ing,possiblygraphicalorsymbolic.Afterinterviewswithtechnicians,thelistofmessagesinTable1wasselected,after carefully discarding everything deemed unneces-sary.

Definition of the dataflow for the AR assisted solutionThe rationale behind the proposed solution is de-

scribedbytheschemeofFig.1.Theweldinggunposi-tionisobtainedbymeansofopticaltrackingofmarkersplacedinthecornersoftheworkingarea.

Fig.1.Schematicrepresentationoftheinformationflow

Table1.Listofthemessagesprovidedtothewelder

Stage of operation Information Metric

Overviewtoprepareassembly Position of all the spots CADrendering

Weldinggunpositioning

Position of the target pointSuggestedmovementGunpositionandorientation

CADrenderhighlightedSymbol(arrow)

Dynamicsymbol

Weldingexecution Selectedprogram Symbol(blink)

Spotqualityanalysis Spotquality Symbol(color)

TiAM_4_2014.indd 58 2014-11-25 12:52:56

TechnologiaiAutomatyzacjaMontażu4/2014

59

Itisalsothepossibletointegratetheobservationwithdatafromencodersmountedontheweldinggun.Notallthe degrees of freedom can bemeasured through en-coder and therefore, presentlywe have not tested thissolution.Anexternalcameraobservestheweldingzoneand theARengineaugments the imagebyadding theweldinginstructions.Theembeddedcameraofthetabletcanbeused insteadof thededicatedexternalcamera.Weusedthisoptiontoavoidplacinganadditionalcam-era in the working area.

Camera calibrationToprovideacrediblecombinationof realandvirtual

objectsitiscrucialtoperformacameracalibrationbothoftheinternalcamerageometricandopticalcharacteristics(intrinsicparameters)andofthepositionandorientationofthecameraframe(extrinsicparameters)withrespecttoareferencesystem[16].

Theintrinsicparameterstakeintoaccountthecameradistortion, too. They can be considered with good ap-proximationasconstants.Extrinsicparametersarevari-ableasthecameraismountedontheweldingtoolthatismovable.Thereisthereforethenecessityforaninitialcalibrationtobeperformedatthebeginningoftheworkandofa real timecalibration toupdate in real time thecameraposition.Thetwoprocedureareperformedusingdifferentmethods.

InitialcalibrationmakesuseofOpenCVstandardrou-tines[17],whileiterativerefinementofthecontrolpoints[18]isundertesting.Weusedasinitialcalibrationtoolei-therablackandwhitechessboardorasymmetricalcirclepattern.InFig.2,thelastversionofthecalibrationobjectispresented,withsuperposedinthefourcornersthefidu-cialsquaremarkersthatareusedtodeterminepositionandorientationofthecamera.

Fig.2.Thecalibrationcirclegridthatconstitutethecanonicalimage witheasytolocalizecontrolpointsandthesquarefiducialmark-ers

Thecalibrationobject(circlegridorchessboard)isob-servedbymanydifferentangles.Ontheobjectitispos-sibletoindividuateasetofpointsPi. The transformation betweentheobjectcoordinates(Xi, Yi, Zi)andtheimagecoordinates(ui,vi)isobtainedthroughaprojectionma-trix[P]inthehomogeneouscoordinatesystem.

(1)

Where[R]and[t]aretherotationandtranslationma-tricesandareextrinsicparameters,[K]isthecalibrationmatrixandiscomposedbyintrinsiccameraparameters.

(2)

Theimagecoordinates(ui,vi)mustbecorrectedforthelensdistortionthatisassumedtoberadialandtan-gential. Neglecting nonlinear effects, the distortion pa-rametersarepresentedasadistortionmatrix[D].

(3)

The meaning of the distortion parameters can befoundin[17].Thematricesareidentifiedsimultaneouslybycomparingseveral(around10)differentsnapshotsofthesameobject.Thisisthefirststepinthecameracali-brationandisusedtoassigntheintrinsicparametersofthecamera.Thesecondstep is todetermineagain theextrinsicparametersofthecamera,positionandorienta-tion,exactlyforeveryframe.Asthecamerahasaframerateof30imagesperseconditisimportanttoadoptfastalgorithmsinordertokeepupwiththevisualizationrate.In this secondstep theparameters tobe identifiedarethecoefficientsofthematrices[R]and[t]andcanbeob-tainedbyusingonlyfourfiducialmarkersinthecornersof the image.

TheapplicationwastestedinthefactoryGF-Welding.Aprofessionalweldertriedtoexecutetheweldingpointsand, after a short training time,wasable to repeat theoperationwithouterrors.

Conclusions

Inthepresentedsolution,itsspecificadvantagesneedto be underlined. Thanks to the simplicity of using thesolution,itispossibletotrainawelderfastandthereisnonecessityof incurringhighcostsof the solution.Si-multaneously, we can achieve considerable improve-mentof theprocessquality. It isalsoworthunderliningthatdecreasingthecostsconcerningsavingthetimeof aweldingprocessand theeliminationofmistakes,canalsoeliminatenecessityofproductsrepair.

REFERENCES

1. Kyu-KabCho,Jung-GuySun,Jung-SooOh:An au-tomated welding operation planning system for block assembly in shipbuilding.Int.J.ProductionEconom-ics60-61(1999)203-209.

2. Ahmad Dargi, Ali Anjomshoae, Masoud Rahimin-ezhad Galankashi, Ashkan Memari, Masine BintiMd.Tap,SupplierSelection:AFuzzy-ANPApproach.ProcediaComputerScience,Volume31,2014,Pag-

TiAM_4_2014.indd 59 2014-11-25 12:52:56

4/2014TechnologiaiAutomatyzacjaMontażu

60

es691–700,2ndInternationalConferenceonInfor-mation Technology and Quantitative Management,ITQM2014.

3. DaweiZhao,YuanxunWang,SuningSheng,Zong-guoLin:Real time monitoring weld quality of small scale resistance spot welding for titanium alloy. Measurement, Volume 46, Issue 6, July 2013, pp.1957–1963.

4. CemilOz,KayhanAyar,SoydanSerttas,OsmanIyi-bilgin,UğurSoy,GuluzarCit:A Performance Evalu-ation Application for Welder Candidate in Virtual Welding Simulator.OriginalResearchArticle.Proce-dia-SocialandBehavioralSciences,Volume55,5October2012,pp.492–501.

5. ISO9001:2008Qualitymanagementsystems–Re-quirements.

6. StadnickaD.,AntoszK.,RatnayakeR.M.C.:Devel-opment of an empirical formula for machine classi-fication: Prioritization of maintenance tasks. Safety Science,Volume63,March2014,pp.34–41.

7. Stewart M, Grout R.:The human side of mistake-proofing. Production andOperations.Management2001;10(4):440–59.

8. ISO 3834-1:2005 Quality requirements for fusionweldingofmetallicmaterials–Part1:Criteriafortheselectionof theappropriate levelofquality require-ments.

9. ISO 3834-2:2005 Quality requirements for fusionweldingofmetallicmaterials–Part2:Comprehen-sivequalityrequirements.

10. ISO 3834-3:2005 Quality requirements for fusionweldingofmetallicmaterials–Part3:Standardqual-ityrequirements.

11. ISO 3834-4:2005 Quality requirements for fusionwelding of metallic materials – Part 4: Elementaryqualityrequirements.

12. DennisonA.V.,ToncichD.J.,&MasoodS.(1997):Control and process-based optimisation of spot-welding in manufacturing systems.TheInt.J.ofAd-vancedManufacturingTechnology,13(4),256–263.

13. WeiLi:Modeling and on-line estimation of electrode wear in resistance spot welding, J Manuf Sci Eng(127)(2005),pp.709–717.

14. Cullen J. D.,Athi N.,Al-Jader M., Johnson P.,Al-Shamma’aA. I., ShawA., & El-RasheedA. M.A.(2008):Multisensor fusion for on line monitoring of the quality of spot welding in automotive industry. Measurement,41(4),412–423.

15. LiY.,LiY.F.,WangQ.L.,XuD.&TanM. (2010):Measurement and defect detection of the weld bead based on online vision inspection. Instrumentationand Measurement, IEEE Transactions on, 59(7),1841–1849.

16. OngS. K., Zhang J., ShenY. &NeeA.Y. (2011):Augmented reality in product development and man-ufacturing. InHandbookofAugmentedReality (pp.651–669).SpringerNewYork.

17. Bradski G., Kaehler A., Pisarevsky V.: Learning-based computer vision with intel’s open source com-puter vision library.INTELTECH-J,9(2),may2005.

18. DattaA.,KimJ.S.&KanadeT.(2009,September):Accurate camera calibration using iterative refine-ment of control points. In Computer Vision Work-shops(ICCVWorkshops),2009IEEE12th Interna-tionalConferenceon(pp.1201–1208).IEEE.

_____________________DorotaStadnicka,PhD–RzeszowUniversityofTechnol-ogy,FacultyofMechanicalEngineeringandAeronautics,ul.PowstańcówWarszawy12,35-959Rzeszów,Poland,tel.:+48178651452,e-mail:dorota.stadnicka@prz.edu.pl.Dario Antonelli, Prof. Ing. – Politecnico di Torino, De-partmentofProductionSystemsandEconomics.CorsoDuca degli Abruzzi 24, I 10129 Torino (Italy), tel. +390110907288,e-mail:dario.antonelli@polito.it.

ZASTOSOWANIE RZECZYWISTOŚCI ROZSZERZONEJ W PROCESACH SPAWANIA

S t r e s z c z e n i eNaprocesyspawania,któresąprocesamispecjalnymi,należyzwrócićszczególnąuwagę,jeżelirozpatrujemyproblemzapew-nienia jakości.Standardowesystemyzarządzania jakościąopartenanormachISOwskazująnaobszary,naktórepowinnabyć zwrócona szczególna uwaga, abywyniki procesu spawania spełniały postawione imwymagania.Należywięc przedewszystkim zapewnić odpowiednie przygotowanie pracowników, odpowiedni nadzór nad urządzeniami wykorzystywanymi wprocesiespawania,odpowiedniemetodypracy isystemywspomagającespawaczawbezbłędnej realizacjiprocesuspa-wania.Wniniejszymartykuleprzedstawionozastosowanierzeczywistościrozszerzonejdowspieraniaspawaczawrealizacjiprocesuspawaniapunktowego.Zaprezentowanerozwiązaniejestprosteiniewiążesięzkoniecznościąponoszeniadużychkosztówwdrożenia.Jednocześniespawaczmożebyćwkrótkimczasieprzeszkolonydostosowaniazaproponowanegoroz-wiązania.Przeprowadzonepróbypracyzwykorzystywaniemzaproponowanego rozwiązaniadałydobrewyniki,pozwalającoperatorowiunikaćbłędówwrealizacjiprocesuspawania.Jednocześnieuzyskanoskrócenieczasurealizacjiprocesu.

S ł o w a k l u c z o w ezapobieganiebłędom,zapewnieniejakości,procesyspecjalne,rzeczywistośćrozszerzona

TiAM_4_2014.indd 60 2014-11-25 12:52:56