Upload
chippy-thomas
View
91
Download
0
Embed Size (px)
Citation preview
MET’S School of EngineeringDepartment of Computer Science and Engineering
Academic Year 2015-2016
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 1
SKETCH TO PHOTO MATCHING SYSTEM
Submitted by
Chippy Thomas (GWAOCCS004)
S4-M.Tech CSE (2014-16 Batch)
19-03-2016
May 2, 2023 2
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Introduction
System that match sketch with mugshot
For the identification of suspectsUse when eyewitness are only evidence
Sketch/composites-hand-drawn-software generated-surveillance composites
•mugshot19-03-2016 2
May 2, 2023 3
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Introduction – contd… So matching sketch and photo
Direct matching is not possibleIe..We don’t match them pixel by pixel
Match by finding high-level features from both the sketch and the photo
Features such as the structural distribution and the shape of the eyes, nose and chin etc.
19-03-2016 3
May 2, 2023 4
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Introduction – contd…
19-03-2016 3
May 2, 2023 5
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Hand-drawn composite
19-03-2016 4
•Drawn by forensic artist •Based on description provided by one or more witness•Artist need proper training
May 2, 2023 6
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Software-generated composites
19-03-2016 5
•Drawn by operator using software•Most used software's are identikit and FACES•Operator need training
May 2, 2023 7
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Surveillance composites
19-03-2016 6
•Drawn by forensic artist•Based on poor quality surveillance images
May 2, 2023 8
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Example of Verbal Description
The man was about 5ft. 6in. in height, and 34 or 35 years of age, with dark complexion and dark mustache turned up at the ends . . . He had a heavy mustache curled up, and dark eyes and bushy eyebrows. He had no side whiskers, and his chin was clean shaven. He looked like a foreigner."
19-03-2016 7
Objective
SKETCH TO PHOTO MATCHING SYSTEM
Develop a system that can automatically match sketch with mugshots
System that manage every types of sketch such as -forensic sketches -composite sketches -surveillance sketches
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 8
• Law enforcement agencies disseminate a sketch to media outlets
• Hope that someone will recognise the suspect
• And provide information leading to an arrest.
19-02-2016 2
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Existing system
19-03-2016 9
May 2, 2023 11
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 10
Existing system – contd…
Problem description
SKETCH TO PHOTO MATCHING SYSTEM
•Slow and tedious process
•Fails to leverage resources available to law enforcement agencies (mugshot databases)
•So there is a need for a method that can automatically match facial composites to large police mugshot databases.
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 11
• Automatically match facial composites against mugshots
• For this 2 algorithms are used• Holistic representation (developed by Klare and Jain)• Component representation (developed by Han et al.)
19-02-2016 2
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Proposed system
19-03-2016 12
May 2, 2023 14
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 13
Proposed system – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Working •Sketches that is given as input to the system
•From sketch features are extracted based on 2 algorithms
•There will be a database to hold the features of large mugshot
•By applying 2 algorithms it matches with the mugshot
•2 algorithms produce matching scores
•Combining matching scores of 2 algorithm
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 14
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Figure
19-03-2016 15
Holistic algorithm
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 16
•First step is detection of eye
•Alignment and CSDN
•Feature extraction -SIFT -MLBP
•LDA- learn optimal subspaces & projecting into subspaces•PCA
19-03-2016
18
SKETCH EYE DETECTION
ALIGNMENT AND CSDN
PATCHWISE SEGMENTATION
DENSE GRID OF KEYPOINTS
MLBP
SIFT
LDA Patch-wise concatenation & PCA
Sketch Feature Vector
Mugshot Feature Vector
Repeated for all patches
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
L2 norm
19-03-2016 17
Holistic Algorithm – contd…
Holistic algorithm – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 18
•The same steps are performed for sketch and photo
•Sketch feature vector and mugshot feature vector
•These 2 features are comparing for similarity using L2 norm
•Finding score of matching
19-03-2016
Component-based algorithm
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19
•In holistic algorithm, features are extracted from each patch / grid from images•But component-based algorithm, features are extracted from each component•Per component, features are extracting
19-03-2016
May 2, 2023 21
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 20
Component-based algorithm – contd…
Modules
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 21
Eye Detection
•Accurate and reliable•Eyes are protected and remain reliably unchanged for decades•Eyes are relatively stable feature
19-03-2016
Modules – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 2219-02-2016
Alignment and CSDN
•composite is normalized to a fixed height and width•transformed such that right and left eyes are at the same position for every composite.
•The center-surround divisive normalization (CSDN)•filter is then applied to the composite to compensate change in modality between composite and mugshot.
19-03-2016
Modules – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 23
•SIFT and MLBP Feature Extraction
•SIFT features are extracted from dense grid of keypoints
•Multi-scale local binary pattern (MLBP) features are extracted from a uniform patches across the face,
19-03-2016
Modules – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 24
LDA
•For both SIFT and MLBP features, optimal subspaces are learned for each patch using linear discriminant analysis (LDA).•projecting the patch-wise features into their respective subspaces
PCAPCA is applied to the feature vector to reduce template size
19-03-2016
Modules – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 25
L2 norm•It is used as classifier•For finding similarity between mugshot and sketch features
Sum of Score Fusion•Scores from 2 algorithms are fused using sum of score fusion
19-03-2016
Innovative idea
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 26
•Modifying existing systems with Enhanced Holistic method and Enhanced Composite method.
•Replacing feature extraction methods and classifier
19-03-2016
Innovative idea
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 2719-03-2016
Innovative idea – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 28
•In the proposed scheme it is done by replacing the existing systems algorithms such as SIFT and MLBP with SURF and harlick algorithms.
•Also adding Neural network classifier instead of L2 norm for better accuracy.
19-03-2016
screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 2919-03-2016
•Input sketch
screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3019-03-2016
•Eye detection from sketch
screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3119-03-2016
•Aligned sketch
screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3219-03-2016
•Features extracted from sketch
screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3319-03-2016
•Eye detection from photo
screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3419-03-2016
•Feature extraction from photo
Conclusion
SKETCH TO PHOTO MATCHING SYSTEM
3619-02-2016
•Facial composites are used by law enforcement agencies for the identification of suspects involved in criminal activities
•Sketch to photo system effectively make the use of facial composites
•Automatic
•Can be used with every type of composites
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 35
Number: 01 Guide Name: Mr. JOBI V DAS, AP/CSE 36
SKETCH TO PHOTO MATCHING SYSTEM
•For that 2 algorithms are using here
•Holistic algorithm and component based algorithm
•Match scores from 2 algorithms are combining
•Also filtering based on demographic information
•Along with the matched image some details such as name, age can be displayed.
19-3-2016
Conclusion – contd…
References SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3819-02-2016
•Bonnen K. , B. F. Klare, and A. K. Jain, “Component-based representation in automated face recognition,” IEEE Trans. Inf. Forensics Security, vol. 8, no. 1, pp. 239–253, Jan. 2012.
•Best-Rowden L. , H. Han, C. Otto, B. Klare, and A. Jain, “Unconstrained face recognition: Identifying a person of interest from a media collection,” IEEE Trans. Inf. Forensics Security,Dec. 2014
•Han H. , B. F. Klare, K. Bonnen, and A. K. Jain, “Matching composite sketches to face photos: A component-based approach,” IEEE Trans. Inf. Forensics Security, vol. 8, no. 1, pp. 191–204, Jan. 2013
19-03-2016
References – Contd… SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 39
•Scott J. Klum, Hu Han, Brendan F. Klare Anil K. Jain, ”The FaceSketchID System: Matching Facial Composites to Mugshots” IEEE Trans. Inf. Forensics Security,Dec. 2014 ,Vol. 9, no. 12, Dec 2014
19-03-2016
Quires
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 40
SKETCH TO PHOTO MATCHING SYSTEM
May 2, 2023
?
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 41
SKETCH TO PHOTO MATCHING SYSTEM
May 2, 2023
Thank you