Pharmaceutical Pill Recognition Using Computer Vision...

SchoolofInformaticsandComputing,IndianaUniversityBloomington

CharleneTayandMridul Birla

PharmaceuticalPillRecognitionUsingComputerVisionTechniques

Introduction

• Unidentifiedandmisidentifiedprescriptionpillspresentchallengesforpatientsandprofessionals. Takingsuchpillscanresultinadversedrugreactionsthataffecthealthorcouldevencausedeath.Bycomingupwithwaystoeasilyidentifyandverifyprescriptionpills,errorscanbegreatlyreduced.

• Ourgoalistoproduceaframeworkorsetofmethodsthatwilltakeaconsumer-qualityimageofapill(takenfrommobilephones),andhelptoidentifyitbyreturningthemostlikelymatchesfromourdatabasesetofpillimages.

• Eachimageisconvertedtogreyscaleandbackgroundissubtractedtogetanideaofoverallshape.

• Twelveshapefeaturesarealsoextractedforeachimage,including:the7Huinvariantmoments,circularitydegree,rectangledegree,sphericitydegree,concavitydegreeandflatdegree.

• Wecategorizedandlabeledeachpillinourtrainingdatasetbyshapeandtrainedaneuralnetworkontheseimagesandfeatures

Dataset

Overview ofProcessChallenges andFuture Work

References andAcknowledgements

Twoexamplesofconsumer-qualitypillimages Thecorrespondingreferenceimagematchesforthetwopills shownontheleft

TheNationalLibraryofMedicinehasmadepublicasetofpillimagesthatinclude:

• 2000JPEGhigh-resolutionreference images(oneforthefrontandoneforthebackofeachof1000pills)fromtheComputationalPhotographyforPillIdentificationProject.

• 5000JPEGconsumerqualityimagesofthesame1000pills,takenwithdigitalcamerasinvaryinglightconditions ShapeDetermination

Whatifyoucould takeaphotoofanunknown pilland almostimmediatelyfindoutwhatitis?

Marker/ImprintExtraction ShapeClassification

Marker/Imprint Extraction ResultsofShapeDetermination

OriginalImage Imageafterapplyingerosiontechnique Outputaftersubtractingerosionoutputimagefromoriginalimage

• Themostdistinguishingfactorforpillidentificationisthemarkingsorimprintsonthetabletsorcapsules.Toextractthesefeatures,weusedafewmorphologicalimageprocessingoperations toenhancetheimprints.

• Followingthat,weusedopen-sourcedObjectCharacterRecognitionsoftwaretotrytoextractthenumbersandwordsimprinted.

• Scale-InvariantFeatureTransform(SIFT)descriptorsofeachpillwerealsoextractedtotrytocapturearepresentationoftheimage.

“ 3228 25mg “

Lu, W. (2012). Method for Image Shape Recognition wi th Neura l Network. In Adv ances in Computer Scienc e and In formation Engineering (pp. 547-551). Springer Berl in Heide lberg.Cunha, A., Adão, T., & Triguei ros , P. (2014). HelpmePil ls: a mobi le pi ll recognition too l for elderly pers ons. Proc ediaTechnology, 16 , 1523-1532.Us hiz ima, D., Carnei ro, A., Souz a, M., & Medei ros , F. (2015). Investigating Pil l Recogni tion Methods for a New National L ibrary of Medic ine Image Datas et. In Adv ances in Visual Computing (pp. 410-419). Springer In ternational Publ ish ing.

58%

75%70.66%

55%

25%

10%15%

10%

30%

0%

10%

20%

30%

40%

50%

60%

70%

80%Accuracy of Neural Network

Capsule Circle Oblong/Oval/Football

Pentagon Fat Oval Triangle

Square Hexagon Special (all other shapes)

• Whengiventheconsumerqualityimages,theneuralnetworkgivesanoverallof50.7% accuracy.

• Limitedamountofdata- onlyafewimagesforeachspecificpill.Needtogathermoreimagesfromotherresources.

• Lackofdiversityinshape(mostlycircleoroblong)• Imprintschallengingandrequiresmoreresearch.• Colorhardtomatchduetodifferinglight

conditionsandcameralenses.• Createwrappertoclassifyandsuggestbest

matches.

50.68%

74% 75%81% 80%

55%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

Neura l Network SVM KNN Dec is ion Tree Random Forest Adaboost

Precision of Different Classifiers