SCARS, MARKS & TATTOOS (SMT): PHYSICAL ATTRIBUTES FOR PERSONIDENTIFICATION

Anil Jain and Yi Chen and Unsang Park

Department of Computer Science and EngineeringMichigan State University, East Lansing, MI, 48824

ABSTRACT

Scars, Marks and Tattoo (SMT) are imprints on the skin thathave been shown to be useful by law enforcement agenciesfor identification of a non-skeletalized body. Tattoos also givespecific identifying clues as to the social status, personality,religious affiliations, or affiliations with criminal organiza-tions (e.g., street gangs). However, matching of tattoo im-ages is primarily based on human-assigned class labels. Thisprocess is not only time consuming, but is subjective and haslimited performance. We have designed and built a prototypecontent-based tattoo image retrieval system. Given a queryimage, the system computes the similarity scores between thequery and stored templates based on image content (shape,color and texture) and retrieves the top-N most similar tattooimages from the database. The retrieved images will assistlaw enforcement agents to narrow down possible identity ofa victim or criminal based on the identifying clues providedby the tattoo image (e.g., gang membership, religious affil-iations, etc). The user can also input ancillary informationabout the tattoos (e.g., class, subclass, size, and color) in thequery to speed up the search and improve the quality of re-trieval. The proposed system architecture is compatible withthe American National Standard for Information Systems -Data Format for the Interchange of Fingerprint, Facial, ScarMark & Tattoo (SMT) Information [15]. Our preliminary re-trieval results are based on a database of 4,000 tattoo images,representing 10 different tattoo classes.

1. INTRODUCTION

Tattooing refers to any scar, branding, mark, tattoo, or otherpermanent human body art or modification deliberately placedon the body for purposes of decoration, ornamentation, oradornment [1]. Tattoos have been used in ancient cultures forthousands of years to indicate tribal identity and have servedas “rites of passage, marks of status and rank, symbols of re-ligious, decorations for bravery, marks of fertility, pledges oflove, punishment, amulets and talismans, protection, and asthe marks of outcasts, slaves and convicts”[1].

Today, people choose to be tattooed for cosmetic, reli-gious and magical reasons, and as a symbol of belonging to

Fig. 1. Examples of most popular tattoo designs, based on Internetsearch [2]. (a) Tribal tattoos (account for nearly a third of all tattoodesigns); (b) Japanese/Chinese characters (account for nearly twentypercent of all tattoo designs); (c) Skull tattoos; (d)-(e) Religious tat-toos; (f) Star tattoos; (g) Eagle tattoos (chosen by men for militaryservice; (h) Butterfly tattoos (the most popular feminine tattoo de-sign request).

or to show their affiliation with particular groups (see Fig-ure 1). The National Geographic News stated that 15% ofAmericans have been tattooed (∼ 40 million people) [3]. Ac-cording to Harris Poll conducted in 2003, 36% of Americansaged 25-29 have one or more tattoo [3]. This rise in pop-ularity has been accompanied by the adoption of more per-sonalized and intricate tattoo designs in visible body areas.As a result, tattoos are becoming increasingly useful as toolsfor personal identification primarily in forensic applications.Jain et al. defined ancillary information (such as scars, marksand tattoos) to primary biometric traits (e.g. fingerprints) assoft biometric traits for human identification [4]. There hasbeen an increased emphasis on the use of these soft biometrictraits in various identification tasks when primary biometrictraits are either no long available or are noisy and corrupted.For example, tattoos have helped to identify victims of 9/11terrorist attacks [5] and Asian tsunami [6] (see Figure 2). Infact, tattoo pigments are embedded in the skin to such a depththat even severe burns will often not destroy a tattoo. Scarsand tattoos were also used to identify the body of Al Qaedaleader Abu Musab al Zarqawi, who was killed during a U.S.air strike in Baghdad in 2006 [7]. In September 2006 MichaelMunafo, who broke into hundreds of cars was arrested afterhe was identified based on the tattoos on both his arms and

right leg captured on a surveillance video at a CVS drugstore[8].

(a) (b)

Fig. 2. Tattoos were used for victim identification during (a) theAsian tsunami, December 2004 and (b) the Causeway Street deathinvestigation in Boston, June 2006.

The idea of using tattoos for criminal identification can betraced back to Cesare Lombroso (1835-1909) [9], known asthe “father of modern criminology”, who placed heavy em-phasis on the widespread prevalence of tattoos. Lombrosobelieved that tattoos indicated criminality and stated that “thetattoo can be considered, to use the medico-legal term, as aprofessional characteristic” [10]. Even Bertillonage criminalidentification system based on anthropometry [11], includedindividual body markings - moles, tattoos, scars. In 1959,Burma’s studies [13] suggested that significantly more delin-quents had tattoos than non-delinquents and concluded thattattoos could provide a source of information for determin-ing gang membership (see Figure 5). It is now a commonpractice for law enforcement agencies to photograph and cat-alog scars, tattoos and marks (SMT) as part of the Comput-erized Criminal History (CCH) record (see Figure 3). To en-sure the uniformity in the capture and exchange of data onSMT, the ANSI/NIST-ITL 1-2000 document was released in1993 with classification standards for SMT data. This stan-dard has been recently updated by including more biomet-ric information (ANSI/NIST-ITL 1-2007) [16]. Accordingto these documents, each captured SMT image is assigned aclass label (e.g., human face tattoo, symbol tattoo, large scar,etc.) and manual searches are based on matching the classlabels (see Figure 4). A matching and retrieval system fortattoos (GangNet [17]) based on these “semantic categories”has been developed and is being used by some law enforce-ment agencies. By simply typing a description of the tattoosinto the system (e.g., “a black dragon”), the software can re-trieve names of gang members with black dragon tattoos. An-other image recognition system [18] captures front/side facialviews, scars, marks, and tattoos of arrestees and combines au-tomated facial recognition and physical/demographic data toidentify individuals and create photographic lineups. As anexample, if a witness to a crime notices that the suspect hada tattoo of a snake on his right hand, the system can searchthrough the database of tattoos for possible matches.

The two systems mentioned above, GangNet and DigitalPhotomanager, however, retrieve SMT images from a data-base primarily based on textual (label-based) queries (e.g.,“show tattoos in the database labeled as black dragons”) rather

Fig. 3. Example of complete Computerized Criminal History(CCH) record containing both public and private information.

than image-based query (e.g., “show all tattoos in the data-base that are similar to the query image”). In other words, thecurrent systems require an expert to first assign a category la-bel (from ANSI/NIST-ITL 1-2000) to tattoos both during datacollection and query search. This process is not only timeconsuming, but is subjective and has limited retrieval perfor-mance. Content-based image retrieval [20], on the other hand,automatically determines the image content to compute thesimilarity between two images, rather than relying on human-assigned class labels. Advances in content-based image re-trieval have lead to systems for efficient and accurate retrievalof images from large databases of face, satellite, trademark,vacation, and medical images [21, 22, 23]. We have designedand built a prototype tattoo matching and retrieval system.The block diagram of our system is shown in Fig. 6. Notetattoos are not always unique and therefore, may not achievehigh identification accuracy when used alone. However, re-trieving the top-N most similar tattoo images in the databasemay help assist the law enforcement agents to discover po-

������������������ �������

������������������������ ����������

Fig. 4. Tattoo classes defined in ANSI/NIST-ITL 1-2000.

Fig. 5. Examples of gang member tattoos [14]. (a) Border Broth-ers symbol representing crossing the border into the United States;(b)-(c) Aryan Brotherhood (AB), a prison gang originated in 1967 inSan Quentin; (d) Spider web design tattoo, often found on the armsof racists who have spent time in jail; (e) Nazi Low Riders (NLR), acriminal gang that is characterized by drug trade and white suprema-cist ideology; (f) Black Guerilla Family (BGF) tattoo combines crosssabers, shotguns and black dragons taking over prison towers.

tential identifying clues associated with a victim or criminal.Further, this paper does not focus on improving image re-trieval systems, rather, our goal is to apply image retrievaltechniques currently available in literature onto tattoo imagesand investigate its potential value for law enforcement appli-cations.

2. CONTENT-BASED TATTOO IMAGE RETRIEVAL

As mentioned earlier, textual information, such as class andsubclass labels, is used to annotate and retrieve tattoo imagesin forensics community. But tattoo images are often com-posed in terms of multiple objects and are too complex to beclassified into simple categories. Further, as the size of thetattoo database continues to grow, the current class labels inthe ANSI/NIST standard (Fig. 4) may become inadequate todescribe them. Since humans use color, shape, and texture tounderstand and recollect the contents of an image, it is nat-ural to use features based on these attributes for tattoo imageretrieval. However, it is well known that image retrieval sys-tems based on low-level image attributes (color, shape andtexture) often do not perform well for complex images be-cause these systems do not utilize image “semantics”. So,one may have to utilize both high level class labels as well aslow level image attributes to build a robust system. This is our

�����

�������

��� ����������

������ �������

����������������� �������������������

������������� �!"

�����

�������

���#���$�����

%�����&��%������������� ���������

'���

�()��( �*

�������+

,-.-/010.2341.35

����������

6781934-5:

Fig. 6. Proposed system architecture for automatic SMT matchingand retrieval.

long-term goal as shown in the system block diagram (Fig. 6).Our specific choice of features for capturing low-level imageattributes is based on the literature on content-based imageretrieval.

2.1. Color Image Content

Color is an important feature to describe an image content.While there are a number of color space representation schemes(e.g., RGB, HSI, and YUV), the RGB space is the most widelyused. To extract features in the RGB space, we utilize twocolor descriptors, namely color histogram and color correl-ogram [20, 24]. Color histogram provides the probabilitydistribution of each color appearing in an image and is es-sentially invariant under rotation and translation of the image.Unlike a histogram, a color correlogram characterizes the spa-tial correlation of pairs of colors. A color correlogram is atable indexed by color pairs, where thek-th entry for (i, j)specifies the probability of finding a pixel of colorj at a dis-tancek from a pixel of colori in the image. In our experi-ments, the color histogram and correlogram are calculated bydividing each of the color components into 20 and 64 bins,respectively. For computational efficiency, we use the colorautocorrelogram that only captures the spatial correlation be-tween identical colors in a local neighborhood, i.e.,i = j atpixel distancek = 1, 3, 5.

2.2. Shape Image Content

The shape information contained in a tattoo image is usuallyextracted after the foreground has been segmented from back-ground. Methods used for tattoo segmentation are describedin Section 3. Classical shape representation uses a set of mo-ment invariants. Given a segmented grayscale imageI(x, y),

its central moments of order(p + q) are defined as

µp,q =∑

(x,y)∈I

(x− x)p · (y − y)q · I(x, y), (1)

where(x, y) is the center of the tattoo. This central momentcan also be normalized to be scale invariant. Based on the2nd and3rd order moments ((p+q)=2,3), a set of seven fea-tures that are invariant to translation, rotation, and scale areobtained [19]. These feature sets,Sgray andSgrad can be ex-tracted from the segmented grayscale and the gradient tattooimages, respectively [20].

2.3. Texture Image Content

To capture texture features of tattoo images, EDCV (EdgeDirection Coherence Vector) is used [21]. An edge direc-tion coherence vectorT stores the ratio of coherent versusnon-coherent edge pixels with the same quantized direction(within a quantization of< 10◦). The computation of edgepixels is described in Section 3. A threshold (0.1% of imagesize) on the size of every connected component of edges ina given direction is used to decide whether the region is co-herent or not. This feature is geared towards discriminatingstructured edges from randomly distributed edges when thetwo edge direction histograms are similar.

3. EXPERIMENTAL RESULTS

To demonstrate the performance of our prototype system, wedownloaded from the Internet a set of 100 tattoo images [25].These images can be categorized into 10 different classes [15],as shown in Fig. 7. To increase the database size as well as

Fig. 8. Synthetic variants of a tattoo image: (1-3) increased bluecomponent, (4-6) decreased blue component, (7-9) increased greencomponent, (10-12) decreased green component, (13-15) increasedred component, (16-18) decreased red component, (19-21) reducedwidth, (22-24) reduce height and width (25) added illumination, (26-30) added salt & pepper noise (31-33) reduced height, (34-39) rota-tion.

to account for some intra-class variations that are expected intattoo engravings (e.g., changes in illumination, color, aspectratio and rotation), we added a set of 39 synthetic variationsfor each of the 100 tattoo images (Fig .7). The resulting data-base consists of 4,000 images in 10 classes.

3.1. Preprocessing

Tattoo images are preprocessed to constrain feature extractionin the region of interest. Since tattoos appear on uniform re-gions of skin, edge based operations generally perform wellfor the foreground segmentation. To extract edges, we firstobtain the magnitude and angle of gradient at each pixel byconvolving the image with3× 3 Sobel operators. By thresh-olding the gradient, followed by a morphological closing andopening, foreground is obtained (Fig. 9).

�����������������

�� �������������

��������

�������

�� �������� �����

���������������

�� ����

�� �������

������������������

��������

������������

�������������

Fig. 9. Schematic of preprocessing on tattoo images.

3.2. Matching

Since all color, shape and texture features are fixed lengthvectors, the similarity between any two attributes can be cal-culated using the histogram intersection method. LetH(1)

andH(2) be the histograms of a feature vector from two im-ages. Then the similarity between the two images is definedas:

SH(1),H(2) =∑

i

min(H(1)(i),H(2)(i)). (2)

When the two histograms are completely separated or over-lapped, the similarity score equals 0 or 1, respectively. Thismethod is used to calculate the similarity scoresSC(1),C(2) ,Sγ(1),γ(2) , S

S(1)gray,S

(2)gray

, SS

(1)grad,S

(2)grad

, ST (1),T (2) for color, color

autocorrelogram, grayscale moments, gradient magnitude mo-ments, and texture, respectively. For color histogram, the sim-ilarity scores are calculated by averaging the similarities ineach color component (R, G, B). All attribute histograms arenormalized prior to histogram intersection.

Given two tattoo imagesI(1) andI(2), the overall match-ing scoreMI(1),I(2) is calculated by combining the similarityscores from each of the above attributes with equal weights.That is,

SI(1),I(2) = (3)S

C(1),C(2)+Sγ(1),γ(2)+S

S(1)gray,S

(2)gray

+SS(1)grad

,S(2)grad

+ST (1),T (2)

5 .(4)

Fig. 7. Tattoo database of 100 images in 10 classes [25]: (a) insect, (b) symbol, (c) tribal, (d) dragon, (e) fish, (f) flower, (g) sun, (h) viciousanimal, (i) Chinese character, and (j) bird.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.2

0.4

0.6

0.8

1

Recall

Pre

cisi

on

Performance of the current system

Performance of an ideal retrieval system

Fig. 10. Precision vs. Recall curve for tattoo retrieval results. Therecall achieves81% at the precision rate of81%.

3.3. Image Retrieval and Identification

To demonstrate the retrieval performance on our database, weuse the standard precision and recall protocol, defined as

precision = no. of correctly retrieved imagestotal no. of retrieved images (5)

recall = no. of correctly retrieved imagestotal no. of correct images in the database(6)

For each image in the database, there are 39 variations of thesame image. Therefore, the ideal precision and recall for eachquery should be1 andN/39, respectively, up toN = 39 re-trievals. ForN > 39, precision and recall should be39/Nand1, respectively. The average precision vs. recall curve fortattoo retrieval results are shown in Fig. 10. This is based on

4,000 queries to the system where each image in the databasewas used as a query but the query itself was not included inthe database. As we can see, the precision vs. recall curve weobtain is very close to the idea case, with average top-1, top-10 and top-30 retrieval accuracies on the entire database being98.4%, 95.5%, and91.2%, respectively. Three tattoo retrievalexamples are shown in Fig. 11. The average retrieval time perquery, including preprocessing, feature extraction and match-ing, is 7.04 seconds on a 3.20GHz Pentium 4 processor run-ning MATLAB.

During the identification process, tattoo images retrievedfrom our system will be sent to the law enforcement agenciesfor further investigation. For example, if a tattoo that is verysimilar to the query is retrieved, then the record that containsthis tattoo image will be obtained and other biometric or an-cillary information will be examined to establish the identity.If a retrieved tattoo has been identified as a gang symbol, theninformation such as gang membership can be used to furtherassist establish the identity. In general, tattoos, as externalphysical attributes of human body, are not unique enough toestablish an identity. However, automatic retrieval of tattooimages can certainly assist and expedite person identificationwhen biometrics traits, such as fingerprints, iris, face, or otherancillary information are not available or not identifiable.

Fig. 11. Top-30 retrievals for three different tattoo query images, namely “butterfly”, “tribal”, and “symbol”.

4. CONCLUSIONS AND FUTURE WORK

Scars, Marks and Tattoo (SMT) are playing an increasinglyimportant role in law enforcement, especially due to the ris-ing popularity of tattoos among young people [3]. Tattoo re-trieval based on image content provides a more convenientand faster solution over conventional textual-based retrievalsystems, and has been shown to be potentially useful to as-sist victim and criminal identification. However, due to thecomplex nature of the tattoo images, it is quite challenging toderive salient image-based features. Our initial experimen-tal results are based on a set of simple features associatedwith color, shape and texture that can be efficiently and ef-fectively used to retrieve tattoo images. To further improvethe retrieval performance, especially for large databases, wewill augment this feature set by other salient features derivedfrom frequency domain and other transform domains. Wewill also implement different matching techniques, includ-ing correlation, Kullback-Leibler distance and deformed tem-plate matching (e.g., Thin-Plate Splines). In addition, inte-grating features from different image attributes using variousfusion techniques will be investigated. Currently, we are inthe process of expanding our database by acquiring imageswith higher resolution from a wide range of sources. With alarge database, we will also be able to incorporate high-levelsemantics (tattoo classes) in our retrieval.

5. ACKNOWLEDGEMENTS

The authors would like to thank Fengjie Li and Abhishek Na-gar for their help in data collection and testing.

6. REFERENCES

[1] Wikipedia, Tattoo, December 2006,http://en.wikipedia.org/wiki/Tattoo

[2] Tattoo Designs and Symbols, Top 25 Tattoo De-signs based on Internet searches, December2006, http://www.vanishingtattoo.com/tattoos \ designs \ symbols.htm

[3] Tattoo Facts and Statistics, Oct. 2006,http://www.vanishingtattoo.com/tattoo \ facts.htm

[4] Jain, A. K., and Dass, S. C. and Nandakumar, K.: Cansoft biometric traits assist user recognition?, Proc. SPIEDefense and Security Symposium, Vol. 5404, pp. 561-572, Orlando, FL, April, 2004.

[5] Lipton, E. and Glanz, J.: Limits of DNA Research Pushedto Identify the Dead of Sept. 11, New York Times, April22, 2002.

[6] Decay challenges forensic skills, The Standard-Times,Jan 8, 2005.

[7] Scars used to identify al-Zarqawi, CNN, June 2006.

[8] Downing, T. J.: Tattoos help with identification of sus-pect, The Enterprise, Sept 2006.

[9] Lombroso, C.: Crime: Its causes and remedies, Mont-clair, New Jersey: Patterson Smith, 1912.

[10] Wetzell, R. F., and Becker, P.: Criminals and Their Sci-entists: The History of Criminology in International Per-spective, Cambridge University Press, 2006.

[11] Rhodes, T. F.: Alphonse Bertillon-Father of ScientificDetection, New York: Abelard-Schuman, 1956.

[12] New Orleans Public Library, Images of the MonthGallery, September 2002,http://nutrias.org/∼nopl/monthly/sept2002/bcd1.htm

[13] Burma, J. H.: Self-Tattooing among Delinquents: AResearch Note, Sociology and Social Research, vol. 43,341- 345, 1959.

[14] A Visual Database of Extremist Symbols, Logos andTattoos, December 2006,http://www.adl.org/hate \ symbols/prison \ tattoos.asp

[15] ANSI/NIST-ITL 1-2000 standard: American NationalStandard for Information Systems - Data Format for theInterchange of Fingerprint, Facial, & Scar Mark & Tattoo(SMT) Information, 1993.

[16] ANSI/NIST-ITL 1-2007 standard: American NationalStandard for Information Systems - Data Format for theInterchange of Fingerprint, Facial, & Other Biometric In-formation, 2007.

[17] GangNet: A 21st Century Solution to the Gang Prob-lem, Dec. 2006, http://psd.orionsci.com/Products/Gangnet.asp

[18] Yang, T. C.: New system speeds up criminal identifica-tion, Chicago Tribune, Oct 6, 2006.

[19] Hu, M. K.: Visual pattern recognition by moment in-variants, IEEE Tran. on Information Theory, Vol. 8, pp.179-187, 1962.

[20] Long, F., Zhang, H. J., and Feng, D. D.: Fundamentalsof Content-Based Image Retrieval, in Multimedia Infor-mation Retrieval and Management- Technological Fun-damentals and Applications, Springer, 2003.

[21] Vailaya, A., Jain, A. K., and H. J. Zhang: On imageclassification: City images vs. landscapes. Pattern Recog-nition, vol. 31, pp. 1921-1936, 1998.

[22] Jain, A. K. and Vailaya, A.: Shape-Based Retrieval:A Case Study with Trademark Image Databases, PatternRecognition, Vol. 31, No. 9, pp. 1369-1390, 1998.

[23] Shih, P. and Liu, C.: Comparative Assessment ofContent-Based Face Image Retrieval in Different ColorSpaces, International Journal of Pattern Recognition andArtificial Intelligence, vol. 19, no. 7, pp. 873-893, 2005.

[24] Huang, J. et al.: Image indexing using color correlo-gram, IEEE Int. Conf. on Computer Vision and PatternRecognition, pp. 762-768, 1997.

[25] Online Tattoo Designs, http://www.tattoodesign.com/gallery/