0018-9162/12/$31.00 © 2012 IEEE Published by the IEEE Computer Society MAY 2012 93

IDENTIT Y SCIENCES

Tattoo Image Matching and Retrieval

F or more than 5,000 years, people have ma rked their bodies with tat-toos to express personal

beliefs or to signify group associa-tion. Ranging from simple words to elaborate designs, tattoos have been used to brand criminals and prison-ers; indicate social status, religious affiliation, or gang/organization mem-bership; commemorate events; honor certain individuals (the dead, celebri-ties, deities, and so on); and denote attachment to or respect for various animals, objects, symbols, and other entities. Figure 1 shows examples of

the many different uses of tattoos. Over time, tattooing has been

associated with particular groups such as slaves, indigenous peoples, sailors, motor bikers, and street gangs. In recent years, however, it has gained widespread use as a fashion statement. Consequently, the size of the tattooed population is rapidly increasing, particularly among young adults. For example, according to a 2008 Harris Poll, approximately 14 percent of all people in the US, and 32 percent of those ages 25-29, have at least one tattoo (www.harrisinteractive.com/vault/Harris-

Interactive-Poll-Research-Three-in-Ten-Americans-with-a-Tattoo-Say-Having-One-Makes-Them-Feel-Sexier-2008-02.pdf).

TATTOOS FOR HUMAN IDENTIFICATION

Tattoos often serve as a source of useful information for human identification in forensic applications because their pigments are embedded in the skin to such a depth that even severe burns or other damage often do not destroy them. For this reason, tattoos have been used to identify victims of terrorist attacks

Anil K. Jain, Rong Jin, and Jung-Eun LeeMichigan State University

An automated tattoo image matching system achieves significantly better results for forensic and law enforcement applications than traditional keyword-based matching.

Figure 1. Tattoo examples: (a) tattooed right hand of a Chiribaya mummy from southern Peru, c. 900-1350 A.D. (www.smithsonianmag.com/history-archaeology/tattoo.html); (b) membership tattoo of the 18th Street gang in Los Angeles County; (c) religious tattoo; (d) tattoo commemorating the 9/11 terrorist attack; and (e) tattoo of popular entertainer Michael Jackson.

(a) (b) (c) (d) (e)

94 COMPUTER

IDENTIT Y SCIENCES

Dass, and K. Nandakumar, “Can Soft Biometric Traits Assist User Recog-nition?,” Proc. SPIE Conf. Biometric Technology for Human Identification, vol. 5404, SPIE, 2004, pp. 561-572).

For example, suppose the image of a suspect’s face captured by a surveillance camera is blurred and off-center, as in Figure 2b. Given this poor image quality, the matching performance of a facial recognition system would be extremely low. In such cases, supplementary data from tattoos, if present, can assist the identification procedure.

For this reason, the US Federal Bureau of Investigation is incor-porating tattoos as well as scars and other distinctive marks in its Next-Generation Identification (NGI) system (www.fbi.gov/about-us/cjis/fingerprints_biometrics/ngi).

KEYWORD-BASED TATTOO MATCHING

Law enforcement agencies rou-tinely photograph and catalog tattoo patterns for use in identifying victims and convicts. To assist in format-ting data for tattoos as well as other biometric traits, the ANSI/NIST-ITL 1-2011 standard (www.nist.gov/itl/ iad/ig/ansi_standard.cfm) defines eight tattoo classes and 70 subclasses. Figure 3 lists the classes and, as an example, the subclasses of tattoos of animals and animal features.

Keyword-based tattoo matching is performed by matching the class label of a query with the tattoos in the database. However, this matching based on human-assigned class labels is subjective, has limited performance, and, for popular keywords, returns a large

such as 9/11 and natural disasters like the 2004 Indian Ocean tsunami (J.-P. Beauthier, P. Lefèvre, and E. De Valck, “Autopsy and Identification Techniques,” The Tsunami Threat: Research and Technology, N.-A. Mörner, ed., InTech, 2011, pp. 691-714).

Criminal identification using tattoos is another important and growing application. Tattoos often contain helpful information such as gang membership, previous convictions, years in confinement, and even depictions of criminal incidents, as Figure 2a shows.

In addition, tattoos are among several “soft” biometric traits that can aid in criminal identification when primary biometric traits— fingerprints, face, iris, hand geome-try, and so on—are either unavailable or severely corrupted (A.K. Jain, S.C.

Figure 2. Examples of criminal cases in which a tattoo was useful in apprehending a suspect. (a) A police officer noticed that this “murder map” tattoo on a suspect’s chest depicted an unsolved homicide (http://news.sky.com/home/strange-news/article/ 15977524). (b) A teen suspect was linked to six armed ATM robberies when a distinctive facial tattoo recorded by a surveillance camera matched a tipster’s description (www.topix.com/forum/detroit/TFH20505S1EQ54M3B).

Figure 3. ANSI/NIST-ITL 1-2011 tattoo classes and animal tattoo subclasses.

(a) (b)

ANSI/NIST-ITL 1-2011 tattoo classesClass description Code

Human forms and features HUMANAnimals and animal features ANIMALPlants PLANTFlags FLAGObjects OBJECTAbstractions ABSTRACTInsignias and symbols SYMBOLOther images OTHER

ANSI/NIST-ITL 1-2011 animal tattoo subclassesSubclass description Code

Cats and cat heads CATDogs and dog heads DOGOther domestic animals DOMESTICVicious animals (lions, tigers, and so on)

VICIOUS

Horses (donkeys, mules, and so on)

HORSE

Other wild animals WILDSnakes SNAKEDragons DRAGONBirds (cardinal, hawk, and so on) BIRDSpiders, bugs, and insects INSECTAbstract animals ABSTRACTAnimal parts PARTSMiscellaneous animal forms MANIMAL

MAY 2012 95

problem is the poor quality of tattoo images, which, as Figure 5 shows, can be caused by low contrast, uneven lighting, fading, and obscuration by body hair. This noise in tattoo images results in the extraction of numerous spurious keypoints and thus false matches.

To reduce the number of false matches, we introduced weighted keypoint matching. If Tattoo-ID matches a keypoint in a database image to multiple keypoints from a query image, it considers these keypoints to be not very distinctive and assigns them low weights in computing the similarity measure. Figure 6a shows an example of low local distinctiveness. If the system matches a query keypoint with keypoints from many different database images, it also considers this

For a given query tattoo image (Figure 4a), the system automatically extracts a set of image features using scale-invariant feature transform (SIFT) (D.G. Lowe, “Distinctive Image Features from Scale Invariant Keypoints,” Int’l J. Computer Vision, Nov. 2004, pp. 91-110) and marks these SIFT keypoints with red dots (Figure 4b). Tattoo-ID then compares the query feature set with all the feature sets extracted from tattoo images in the database. It determines the correspondence between two images by matching keypoints (Figure 4c), and finally presents the top K most similar tattoo images according to the number of keypoint correspondences (Figure 4d).

Tattoo matching and retrieval is significantly more challenging than general image retrieval. The main

number of tattoos. For example, in our analysis of 20,000 tattoos, approximately 78 percent matched five keywords. Further, a simple class description in a textual query (“find a dragon tattoo”) doesn’t include all the semantic information in tattoos, as is evident by the large intraclass variability. Finally, the ANSI/NIST classes don’t adequately describe a large number of tattoo designs.

TATTOO-IDTo overcome the limitations

of current keyword-based tattoo matching, we’ve developed Tattoo-ID, an automated tattoo image matching system (J.-E. Lee et al., “Image Retrieval in Forensics: Tattoo Image Database Application,” IEEE Multimedia, Jan. 2012, pp. 40-49). Figure 4 illustrates how the retrieval process works.

Figure 4. Tattoo image matching and retrieval using Tattoo-ID: (a) query image, (b) keypoints in query image, (c) keypoint matching (correspondence) between two images, and (d) top four visually similar tattoos retrieved with the associated match scores (number of matching keypoints). In this case, the system retrieved three duplicates for the query from the database.

Figure 5. Examples of poor-quality tattoo image queries due to (a) low contrast, (b) uneven lighting, (c) fading, and (d) obscuration by body hair.

(a)

16

(a)

(d)

(b)

15

(b)

(c)

15

(c)

7

(d)

Editor: Karl Ricanek Jr., director of the Face Aging Group at the University of North Carolina Wilmington; ricanekk@uncw.edu

IDENTIT Y SCIENCES

About 5 percent of the database images are near duplicates of the same tattoo. These duplicates are present either because the same person has been arrested multiple times, or because different suspects/convicts have the same tattoo (for example, if they belong to the same criminal gang). Our query set had 1,000 manually identified duplicate images from the database. We aug-mented the 64,000 tattoo images with 36,000 randomly selected images from Louis van Ahn’s ESP Game to build a background database of 100,000 images.

Tattoo-ID achieved an average rank-20 retrieval accuracy of 88 percent, a significant performance improvement over keyword-based tattoo matching. In addition, the pro-posed weighted keypoint matching scheme increased the retrieval accu-

racy for low-quality queries from 50 to 57 percent compared to not using the scheme.

Tattoo-ID has been licensed to MorphoTrak, which plans to release a commercial version

for use by law enforcement agencies (http://findarticles.com/p/articles/mi_m0EIN/is_20100119/ai_n48674730). We believe that with the growing use of tattoos for human identification in forensics and law enforcement, such a system will be of great societal value.

Anil K. Jain is a University Distin-guished Professor in the Department of Computer Science and Engineering at Michigan State University. Contact him at jain@cse.msu.edu.

Rong Jin is an associate professor in the Department of Computer Science and Engineering at Michigan State University. Contact him at rongjin@cse.msu.edu.

Jung-Eun Lee is a PhD student in the Department of Computer Science and Engineering at Michigan State Uni-versity. Contact her at leejun11@cse.msu.edu.

The authors thank Capt. Greg Michaud of the Michigan State Police for pro-viding access to its tattoo image database. Part of this research was supported by the FBI Biometric Center of Excellence.

96 COMPUTER

keypoint to be not very distinctive and assigns it a low weight. Figure 6b shows an example of low global distinctiveness. Introducing these two sets of weights has greatly improved matching results, pwarticularly for low-quality tattoo images.

PERFORMANCE EVALUATION

We evaluated the performance of our Tattoo-ID system on an operational tattoo image database maintained by the Michigan State Police that contains 64,000 tattoos. At booking time, if a suspect has a tattoo, its photograph is captured along with the suspect’s fingerprints and mug shots and stored in the data-base along with any metadata about the suspect and tattoo. Figure 7 shows a collection of images from this tattoo database.

Figure 6. Weighted keypoint matching. (a) Low local distinctiveness: five keypoints in the query image are matched to the same keypoint in a database image. (b) Low global distinctiveness: a keypoint in the query image is matched to two different database images.

(a) (b)

Figure 7. Examples of images from the Michigan State Police tattoo database.