Exploiting the Gap Between Human and Machine Abilities in ...govind/amalia_dissertation.pdf · first character ‘w’ and the image between: - segments 1 and 4 is 5.0 - segments

Exploiting the Gap Between Human and Machine Abilities

in Handwriting Recognition for Web Security Applications

Amalia Rusu

Department of Computer Science and Engineering

University at Buffalo

2

Outline

� Objectives

� Motivation

� Background

� Previous Work

� Research Challenges

� Solutions

� Experiments and Results

� Research Ideas for Future Work

� Contributions

3

Objectives� Quantify human vs. machine ability in reading handwriting under noisy

conditions � Develop HCAPTCHAs-based HIP system on the ability gap between

humans and machines in handwriting recognition

The gap in the ability in recognizing handwritten text between humans and computers.

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12Transformation

Acc

ura

cy

WMR-4000

WMR-40000

Accuscript-4000

Accuscript-40000

CMR-4000

CMR-40000

Humans

4

Motivation

Why handwritten CAPTCHAs?

� Machine recognition of handwriting is more difficult than printed text

� Handwriting recognition is a task that humans perform easily andreliably

� Several machine printed text based CAPTCHAs have been already broken

� Speech/visual features based CAPTCHAs are impractical

� Handwritten CAPTCHAs thus far unexplored by the research community

5

BackgroundHuman Interactive Proofs (HIPs)

� New family of security protocols which allow a person to authenticate as belonging to a select group, for example human as opposed to machine, adult as opposed to a child, himself opposed to anyone else, etc.

� Operate over a network without the burden of passwords, biometrics, special mechanical aids, or special training

Example of HIP system interface to confirm user registration

Please enter the handwritten word as it is shown below:

If you cannot read this image click here

6

� CAPTCHAs are tests used by HIP systems � Completely Automatic Public Turing test to tell Computers and Humans

Apart – CAPTCHA

� CAPTCHA should be automatically generated and graded

� Tests should be taken quickly and easily by human users

� Tests should accept virtually all human users and reject software agents

� Tests should resist automatic attack for many yearsdespite the technology advances and prior knowledgeof algorithms

� Exploits the difference in abilities between humans and machines

(e.g., text, speech, facial features recognition, solving puzzles etc)

� A new formulation of the Alan Turing’s test - “Can machines think?”

BackgroundCAPTCHAs

7

� Suppressing SPAM and wormsOnly accept an email if I know there is a human behind the other computer. Prove you are

human before you can get a free email account.

� Search engine bootsThere is an html tag to prevent search engine bots from reading web pages; it only serves to say

"no bots, please“, but does not guarantee that bots won't enter a web site.

� Thwarting password guessing

Prevent a computer from being able to iterate through the entire space of passwords.

� Blocking denial-of-service attacksPrevent congestion based DoS attacks from denying any user’s access to web servers targeted

by those attacks.

� Preventing ballot stuffingCan the result of any online poll be trusted? Not unless the poll requires that only humans can vote.

� Protecting databasesI.e. eBay: protecting the data from auction portals that search across auction sites to provide listings and price information for their users, but prohibiting copying that data

BackgroundHIP and CAPTCHA Applications for Cyber Security

8Backgroundhttp://www.nytimes.com/2007/06/11/technology/11code.html?pagewanted=1&ei=5070&en=027dd9d726115b61&ex=1186804800&adxnnl=0&adxnnlx=1186650546-8DASjhLwzui2PWUBZqxLLg

“You can make a captcha absolutely undefeatable by computers, but at some point, you are turning this from a human reading test into an intelligence test and an acuity test,” said Michael Barrett, the chief information security officer at PayPal, a division of eBay.”

“Aleksey Kolupaev, works for an Internet company in Kiev, Ukraine, and in his spare time, with his friend Juriy Ogijenko, he develops and sells software that can thwart captchas by analyzing the images and separating the letters and numbers from the background noise. They charge $100 to $5,000 a project, depending on the complexity of the puzzle. Mr. Kolupaev said he had worked both for legitimate companies that want to test their own security and for spammers who seek to infiltrate Web sites.On his Web site, ocr-research.org.ua, Mr. Kolupaev boasts of cracking the captchas of companies like MySpace and PayPal; the site also ranks the effectiveness of each captcha. He says he believes that his work makes the Internet more secure because companies tend to improve the captchas that he critiques.Internet companies have responded to these challenges by making their captchas more complex. On YouTube, for example, the letters and numbers in the captcha float on an uneven grid of colors. On the technology news site Slashdot, random squiggly lines slice through the letters and numbers, as if a child had scrawled with a pen on each puzzle.All these tricks are attempts to disguise the boundaries of the characters, so that software cannot identify the numbers and letters. But often these measures prove too tough for humans to decipher as well. “

9BackgroundUsing CAPTCHAs to digitize books

Protect your site from abuse and help digitize books. Use reCAPTCHA on your site for free.http://recaptcha.net

“reCAPTCHA improves the process of digitizing books by sending words that cannot be read by computers to the Web in the form of CAPTCHAsfor humans to decipher. More specifically, each word that cannot be read correctly by OCR is placed on an image and used as a CAPTCHA.”

10

Outline

� Objectives

� Motivation

� Background

� Previous Work


� Solutions



� Contributions

11

Previous Work

AltaVista URL filter uses isolated random characters and digits on a cluttered background.

PessimalPrint uses a degradation model simulating physical defectscaused by copying and scanning of printed text.

BaffleText uses pronounceable character strings that are not in the English dictionary and render the character string using a font into an image (without physics-based degradations); then generate a mask image as shown above.

[Coates, Baird, Fateman 2001][Chew, Baird 2003]

[altavista.com/sites/addurl/newurl]

12

Previous Work

GimpyType 3 different English words appearing in the picture above.

EZ-Gimpy uses real English words.

Gimpy-R uses nonsense words.

Character morphing algorithm that transforms a string into its graphical form.

[Xu, Lipton, Essa, Sung, Zhu 2003]

[Carnegie Mellon University, www.captcha.net 2000]

13

Previous Work

PIXUses a large database of labeled images. All of these images are pictures of concrete objects (a horse, a table, a house, a flower, etc). The program picks an object at random, finds 4 random images of that object from its database, distorts them at random, presents them to the user and then asks the question "what are these pictures of?"

ECOSounds can be thought of as a sound version of Gimpy. The program picks a word or a sequence of numbers at random, renders the word or the numbers into a sound clip and distorts the clip. It then presents the distorted sound clip to its user and asks the user to type in the contents of the sound clip.

ARTiFACIALPer each user request, it automatically synthesizes an image with a distorted face embedded in a cluttered background. The user is asked to first find the face and then click on 6 points (4 eye corners and 2 mouth corners) on the face.

[Carnegie Mellon University, www.captcha.net 2000] [Rui, Liu 2003]

14Research ChallengesMain components that build up the structure of this dissertation

15

Research Challenges

I. Generation of random and infinitely many distinct handwritten CAPTCHAs

II. Identifying and exploiting the weaknesses of state-of-the-art handwriting recognizers

III. Controlling distortion - so that HCAPTCHAs are human readable (conform to Gestalt laws/Geon theory) but not machine readable

16

Outline

� Objectives

� Motivation

� Background

� Previous Work


� Solutions



� Contributions

17

Generation of random and infinite many distinct handwritten text images� Use handwritten word images that current recognizers cannot read

� Handwritten US city name images available from postal applications

� Collect new handwritten word samples

SolutionsChallenge I

� Create real (or nonsense) handwritten words and sentences by gluing isolated upper and lower case handwritten characters or word images

18

Generation of random and infinite many distinct handwritten text images� Use handwriting distorter for generating synthetic “human-like” samples

� Template-based synthetic handwritten characters generation� Synthetic text is generated by concatenating perturbed versions of the character

templates

Solutions Challenge I

Synthetic human-like word(s) samples.

[Rusu, Govindaraju 2007]

19

Solutions Challenge I

Tracing of Character Templates

The GUI of the tracing program.

A traced template for character x.

20

Solutions Challenge IGeneration of Synthetic Samples

Illustration of various nonlinear transformations performed individually: a) only ascent line variation, b) only x-line variation, c) only descent line variation, d) only text width variation, e) only shearing variation, and f) only baseline variation.

The finalized synthetic handwriting sample with varying width and thickness.

Perturbations of curve-defining points with various degrees of perturbation.

21Solutions Challenge IGeneration of Synthetic Samples

The GUI of the generator program.

22

[Xue, Govindaraju 2002]

[Kim, Govindaraju 1997]

[Favata 1996]

� Word Model Recognizer (WMR)� Character Model Recognizer (CMR)� Accuscript

1 2 3 4 5 6 7 8 9

w[7.6]

w[7.2]r[3.8]

w[5.0]

w[8.6]

o[7.6]r[6.3]

d[4.9]

w[5.0]

o[6.6]

o[6.0]

o[7.2]o[10.6] d[6.5]

d[4.4]

r[7.5]r[6.4]

o[7.8]r[8.6]

o[8.7]r[7.4]

r[7.6]

o[8.3]

o[7.7]r[5.8]

1 2 3 4 5 6 7 8 9

o[6.1]

Find the best way of accounting for characters ‘w’, ‘o’, ‘r’, ‘d’ buy consuming all segments 1 to 8 in the process

Distance between lexicon entry ‘word’ first character ‘w’ and the image between:- segments 1 and 4 is 5.0- segments 1 and 3 is 7.2- segments 1 and 2 is 7.6

Lexicon Driven Model

� segmentation-then-recognition scheme

� segments, isolates and recognizes the characters which make up the word

� uses modules for preprocessing, segmentation, character recognition and lexicon ranking

� grapheme-based recognizer

� extracts high-level structural features from characters such as loops, turns, junctions, arcs, without previous segmentation

� uses a stochastic finite state automata model based on the extracted features

� uses static lexicons in the recognition process

� lexicon driven approach

� chain code based image processing � pre-processing

� segmentation

� feature extraction

� dynamic matching

JunctionLoops

LoopTurns

End

End

Grapheme Based Model

SolutionsChallenge IIExploit the Source of Errors for State-of-the-art Handwriting Recognizers

4

56

7 82 3

1

1 32 4 5 6 7 8i[.8], l[.8] u[.5], v[.2]

w[.6], m[.3]

w[.7]

i[.7]u[.3]

m[.2]m[.1]

r[.4]

d[.8]o[.5]

-Image from 1 to 3 is a ‘u’ with 0.5 confidence-Image from segment 1 to 4 is a ‘w’ with 0.7 confidence-Image from segment 1 to 5 is a ‘w’ with 0.6 confidence and an ‘m’ with 0.3 confidence

Find the best path in graph from segment 1 to 8 : w o r d

23SolutionsChallenge II

Exploit the Source of Errors for State-of-the-art Handwriting Recognizers

Speed and accuracy of Handwriting Recognizers. Feature extraction time is excluded. Testing platform is an Ultra-SPARC [figure taken from Xue and Govindaraju, 2002].

66.4958.140.99466.5658.141.82720000

86.2975.380.08986.2975.380.1441000

94.0689.120.03194.1389.220.044100

98.7796.560.02198.7396.530.02710

Top 2Top 1Top 2Top 1

accuracytime (secs)

accuracytime (secs)

Grapheme ModelLexicon DrivenLexicon size

24

Source of Errors for State-of-the-art Handwriting Recognizers� Image quality

Background noise, printing surface, writing styles

SolutionsChallenge II

Add lines, grids, arcs, background noise, convolution masks and special filters

25

Source of Errors for State-of-the-art Handwriting Recognizers� Image features

Stroke width, slope, rotations


Generate variable stroke width, slope, rotate, stretch, compress word

26

Source of Errors for State-of-the-art Handwriting Recognizers� Segmentation errors

Over-segmentation, merging, fragmentation, ligatures, scrawls


Delete ligatures, use touching letters/digits, merge characters for over segmentation or to be unable to segment

27

Source of Errors for State-of-the-art Handwriting Recognizers

� Recognition errors

Confusion with

similar lexicon entries,

large lexicons


Increase lexicon size and density; lexicon availability

T r u t h

W M R r e s u l t s

( T o p c h o i c e f i r s t )

A c c u s c r i p t r e s u l t s

( T o p c h o i c e f i r s t )

I m a g e

O r l a n d o o v l a n d o o v l a v d o o n l a n d o o r l a n o l o o r l a u d o o v i a n d o o r l a h d o a r l a n d o o r l a n d o

o v l a n a o

o l l a n d o o v l a n d o o r l a n o l o o r l a n d o

o v l a n a o o v l a v d o o n l a n d o o v i a n d o o r l a n d a a r l a n d o

L a c k a w a n n a l a c k a e v a n a l a c k a w a w a l a c k a w a u a l a c k o w a n a l a c k a w a n a l a c k a w a n n a

l a c k a w a r n a l a c k a w a n r a l a c k a m a m a l a c ta w a n a

l a c k a w a r n a l a c ta w a n a l a c k a w a r r a l a c k a w a w a l a c k a w a n a l a c k a w a u a l a c k a w a n n a

l a c k o w a n a l o c r a w a r a l a c k a w a n r a

C l a r e n c e c l a r l n c l c l a r l n c e c l a r e n c l c e a r e n c e c l a r e n c e

c b a r e n c e c l o r e n c e c l a h e n c e a a r e n c e c l a w c e

c l a ie n c e c l a r e n c e

c l a te n c e c l a r l n c e c e a r e n c e c l a v e n c e c l a r e n x e c l a s e n c e c l o r e n c e c l a ie x c e

B u f f a l o b u f f a i o b u f f a l o

b u t f a l o b u i f a l o b u f f r i o r u f f a l o b u l f a l o b u f i a l o b u e f a i o b u l l a l o

r u f f a l o b u f f a l o

b u f f r l o b u f f a i o b u f f r i o b u l f a l o b u i f a l o b u t f a l o b u e f a l o b u f i a l o

28

Postal applications / Address InterpretationCensus forms, medical forms reading Bank check reading

CAPTCHAHuman Interactive ProofsChallenge/Response protocolsCyber security applications

Processing of historical documents (original paper aged and deteriorated, palm leaf manuscripts)Information leakage through document redaction

SolutionsChallenge III Ensuring Human Recognition

29

� Gestalt psychology is based on the observation that we often experience things that are not a part of our simple sensations

� What we are seeing is an effect of the whole event, and is more than the sum of the parts (holistic approach)

� Organizing principles - Gestalt laws:� law of closure� law of similarity� law of proximity� law of symmetry� law of continuity� law of familiarity� figure and ground

� By no means restricted to perception only – that is just where they were first noticed� memory

OXXXXXX XOXXXXX XXOXXXX XXXOXXX XXXXOXX XXXXXOXXXXXXXO

**********

**********

**********

[ ] [ ] [ ]

SolutionsChallenge III

30

� Geon Theory of Pattern Recognition – recognition by components

� Geons are building blocks of perception

� Object recognition = perceiving Geons (geometrical ions)

Solutions Challenge III

a) Basic geons. b) Objects constructed from geons.

Object recognition is size invariant.

Object recognition is rotational invariant.

31

� Two important aspects of Geons� Edges

� Intersections


Evidence of Geon Theory when objects are lacking some of their components: Recoverable vs. Non-recoverable objects.

32

Gestalt laws: law of proximity, symmetry, familiarity, continuity, background

Create horizontal or vertical overlaps


Control Overlaps

For same word, smaller distance overlapsFor different words, bigger distance overlaps

33

Gestalt laws: law of closure, proximity, continuity, familiarity

Add occlusions by circles, rectangles, lines with random anglesEnsure small enough occlusions such that they do not hide letters completely


Control Occlusions

34

Gestalt laws: law of closure, proximity, continuity

Add occlusions by waves from left to right on entire image, with various amplitudes / wavelength or rotate them by an angleChoose areas with more foreground pixels, on bottom part of the text image (not too low not to high)

Control Occlusions


35

Gestalt laws: law of closure, proximity, continuity, background

Use empty letters, broken letters, edgy contour, fragmentationBreak characters such that general image processing techniques cannot reconstruct the original image

Control Letter Fragmentation


36

Gestalt laws: law of background, familiarity

Add occlusion using the same pixels as the foreground pixels (black pixels), arcs, or lines, with various thickness

�Curved strokes could be confused with part of a character�Use asymmetric strokes such that the pattern cannot be learned

Control Extra Strokes - Waves


37

Gestalt laws: law of background, familiarity

Add occlusion using the same pixels as the foreground pixels – black jawsExtra strokes are confused with character components when they are about the same size, thickness, and curvature as the handwritten characters

Control Extra Strokes - Jaws


38


Split the image in two parts on horizontal and displace the parts in opposite directionsLearn reasonable position for horizontal displacement, adjust and decide the range based on human/machine results

Control Word Displacement


39


Split the images in four parts, either by a vertical/horizontal line or by diagonals, and spread the parts apart – mosaic effectSymmetry in displacement helps image reconstruction for humans - find best range and apply accordingly

Control Word Displacement


40

flip-flop

vertical mirror

horizontal mirror

Gestalt laws: memory, internal metrics, familiarity of letters

Change word orientation entirely, or the orientation for few letters onlyUse variable rotation, stretching, compressing

Control Letter/Word Orientation


41

Human abilities rely on context – comprehension helps disambiguate uncertainty

Use the words in sentences

Other CAPTCHAs - Reading Comprehension

I see a on the wall.

The is a great source of information.


42

Devanagari CAPTCHA

Other CAPTCHAs – CAPTCHA for other scripts


Various transformations applied on Devanagari symbols: a) Displaced images, b) Mosaic images, c) Noisy images, d) Overlapped images, e) Varying horizontal stroke width, f) Varying vertical stroke width.

a. b. c.

d. e. f.

43

Other CAPTCHAs – graph-based CAPTCHA


44

Methods to attack HCAPTCHAs


Background noise that cannot be reverted. The truth words are: Los Angeles, Silver Creek, Young America.

Background noise that can be reverted. The truth words is: Wlsv.

Background noise

45



a) Before pre-processing, b) After pre-processing. The truth words are: WSeneca, Young America.

Wave transformation

46



Overlapping transformation with bad reverse: a) before pre-processing, b) after pre-processing. The truth words are: Matthews, Paso.

Overlapping

47

Input. � Original (randomly selected) handwritten image (existing US city name

image or synthetic word image with various length or sentence)� Lexicon containing the image’s truth wordOutput. � Handwritten CAPTCHA imageMethod. � Randomly choose a number of transformations � Randomly establish the transformations corresponding to the given number� If more than one transformation is chosen then

� A priori order is assigned to each transformation based on experimental results� Sort the list of chosen transformations based on their priori order and apply them

in sequence, so that the effect is cumulative


SolutionsHandwritten CAPTCHA Generation Algorithm

48

Outline

� Objectives

� Motivation

� Background

� Previous Work


� Solutions



� Contributions

49

Experiments and ResultsHCAPTCHA-based HIP as a Challenge Response Protocol for Security Systems

� Initialization

� Handwritten CAPTCHA Challenge

� User Response

� Verification

Automatic Authentication Session for Web Services.

Internet

User

Authentication Server

Challenge

Response

User authentication

The user initiate the dialog and has to be authenticated by server

Internet

User

Authentication Server

Challenge

Response

User authentication

The user initiates the dialog and has to be authenticated by server

50

Experiments and Results

HCAPTCHA test online at URL: http://www.cedar.buffalo.edu/˜ air2/captcha/captcha.php

Handwriting-based HIP system: challenges and verification online.

51

� No risk of image repetition� Image generation completely automated: words, images and distortions

chosen at random

� The transformed images cannot be easily normalized or rendered noise free by present computer programs, although original images must be public knowledge

� Deformed images do not pose problems to humans� Human subjects succeeded on our test images

� Test against state-of-the-art: Word Model Recognizer, Character Model Recognizer, and Accuscript� HCAPTCHAs unbroken by state-of-the-art recognizers

Experiments and ResultsTesting and Evaluation

52

Handwritten US city name images that defeat both WMR and Accuscript recognizers.

Handwritten “nonsense” word images that defeat both WMR and Accuscript recognizers.


53

Low accuracy of handwriting recognizers vs. humans on a subset of test images .

3.19%

4.41%

Accuscript

12.04%Random Non-sense Words

9.28%City Names

WMRTest Images

82%

Humans

Accuracy

12

Number of

Students

0%

Accuscript

Accuracy

0%

WMR

Accuracy

15

Number of

Test Images

Low accuracy of handwriting recognizers for general image transformations. The lexicons are created so as to contain all the truths of test images. Total number of tested images is

4,127 for city names and 3,000 for non-sense words. (and so is the lexicon size)



54

The accuracy of handwriting recognizers. A set of 4,100 images was tested for each kind of transformation using lexicons with size 4,000 and 40,000. [Rusu, Govindaraju 2005-6]


N/A

N/A

3.4%

4.9%

6.8%

5.7%

5.8%

25.6%

1.7%

3.1%

3.5%

0.3%

0.0%

1.2%

5.2%

4,000

CMR

N/A

N/A

2.1%

3.2%

5.1%

4.3%

4.2%

19.9%

1.3%

2.2%

2.4%

0.2%

0.0%

0.9%

3.8%

40,000

0.2%0.7%0.1%0.5%Flip-Flop

0.9%4.4%0.5%3.8%Overlap Different Words

0.4%2.4%3.6%12.9%Horizontal Overlap (Large)

0.6%2.9%10.7%24.4%Horizontal Overlap (Small)

3.9%12.6%14.4%27.9%Vertical Overlap

0.4%1.6%5.3%16.4%Black Waves

4.3%10.6%7.0%15.4%Occlusion by waves

17.4%32.3%20.3%35.9%Occlusion by circles

0.8%3.6%1.3%5.1%Jaws/Arcs

3.0%9.0%6.4%14.3%Mosaic

3.4%8.8%10.3%19.8%Displacement

0.0%0.0%0.2%0.5%High Fragmentation

0.2%0.2%0.0%0.0%Small Fragmentation

0.0%0.1%0.4%0.9%Empty Letters

2.5%6.4%5.7%12.7%All Transformations

40,0004,00040,0004,000Lexicon Size

AccuscriptWMRHW Recognizer

55

The accuracy of human readers. A word image is recognized correctly when all characters are recognized.



65.2%89Horizontal Overlap (Large)

76.7%90Horizontal Overlap (Small)

87.5%88Vertical Overlap

80.0%90Black Waves

80.5%87Occlusion by waves

67.8%90Occlusion by circles

71.9%89Jaws/Arcs

74.4%90Mosaic

78.6%89Displacement

74.4%90High Fragmentation

73.9%88Small Fragmentation

82.0%89Empty Letters

76.1%1069All Transformations

AccuracyTested ImagesTransformations

Max. value

Min. value

56


AP 2AP 2 =18 =32

AP2

Parameterization of the difficulty levels of HCAPTCHAs� Determine image complexity

• Human visual perception is sensitive to contrast and border perception

• Perimetric complexity vs. effective image density� Perimetric complexity =

� Image density = the number of black pixels / the number of total pixels

� Test humans’ difficulty and recognizers’ accuracy vs. image complexity

57


0

2000

4000

6000

8000

10000

12000

14000

16000

18000

20000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Sample images

Per

imet

ric

Co

mp

lexi

ty

ArcJawsBlackWaveEmpty LettersSmall FragmentationHigh FragmentationHoriz Overlap SmallHoriz Overlap LargeMosaicVertical OverlapDisplacementOcclusion by CirclesOcclusion by Wave

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Sample images

Imag

e d

ensi

ty (b

lack

pix

els

/ all

pix

els)

ArcJawsBlackWaveEmpty LettersSmall FragmentationHigh FragmentationHoriz Overlap SmallHoriz Overlap LargeMosaicVertical OverlapDisplacementOcclusion by CirclesOcclusion by Wave

0

20

40

60

80

100

0 5000 10000 15000 20000

Perimetric Complexity

Hu

man

s R

eco

gniti

on A

ccu

racy

A snapshot of handwritten images and corresponding perimetric complexity (perimeter squared divided by area of black pixels). A snapshot of handwritten images and

corresponding image density (black pixels vs. total pixels).

Humans recognition accuracy vs. perimetriccomplexity as a percent of correct answers per bin (with a total range for perimetriccomplexity of 100 equal bins, complexity range 0-20000).

The gap between humans and machines in reading handwriting by category of distortions – Humans’ accuracy vs. machine accuracy for each transform

– Determine most effective transformations that increase the gap


58

Outline

� Objectives

� Motivation

� Background

� Previous Work


� Solutions



� Contributions

59

Research Ideas for Future Work

� Develop a new authentication mechanism of e-mail addresses� In order to contact a person by e-mail, first receive back an alias e-mail address to use

instead of the original one, which has been rendered by the deformation algorithm

� Explore other instances of open pattern analysis problems to build CAPTCHAs - to benefit both online security and pattern recognition research� Recognizing engineering drawing, describe the weather in a scene, recognize a person in a

photo, count the same objects in an image, etc.

� Develop methods for computing a password/PIN assurance and similarly quantify and evaluate different kinds of CAPTCHAs

� Similarly investigate handwritten HIPs for enabling the detection of phishingattacks� Unlike traditional HIPs where the computer issues a challenge to the user over a network, in

this case the user issues a challenge to the server- The user authenticates the server and can detect which websites are spoofed in order to trick users

into revealing private information

60

Research Ideas for Future Work

� Explore other authentication schemes for individuals and groups – group membership that distinguish classes of people

� Use methodologies used to create HIPs for generating new graphical passwords schemes that incorporate the cognitive aspects of human reading and understanding

� Explore general aspects of security and biometric systems that identify people based on handwriting and CAPTCHAs� Handwriting recognition and biometrics share many similar techniques for image analysis, feature

extraction, segmentation, pattern classification and matching

� Create research bridges between fields who have forensics and information security in common: cryptography, biometrics, image analysis, secure processing, computer networks, legal and ethical issues, privacy, and more

61

Contributions

� Machine Learning and Pattern Recognition - Exploring human versus machine abilities in handwriting recognition will help researchers improve machine capabilities by incorporating the cognitive aspects of human reading

� Cognitive Science - Advancing the understanding of “how” humans read handwriting assisted by cognition

� Developing a novel HIP security protocol using handwriting recognition

� Empirically proving that handwritten HIPs are superior to currently used HIPs for cyber security applications

62� Securing Cyberspace using Handwriting Recognition and Gestalt Laws of Perception, with Venu Govindaraju, in preparation, to be submitted to International Journal on Document Analysis and Recognition (IJDAR).

� Synthetic Handwriting Generator for Cyber Security, with Uros Midic, and Venu Govindaraju, accepted, to appear in Proc. of 13th Conference of the International Graphonomics Society (IGS 2007).

� The Influence of Image Complexity on Handwriting Recognition, with Venu Govindaraju, in Proc. of 10th IAPR International Workshop on Frontiers of Handwriting Recognition (IWFHR 2006), IEEE Computer Society.

� Graph-based CAPTCHA: An application for Cyber Security, with Adrian Rusu, Christopher Clement and John Wanies. Poster at 17th Annual Student Research Poster Symposium & 60th Annual Eastern Colleges Science Conference, Saint Joseph's University Chapter of Sigma Xi, Philadelphia, PA, April 22, 2006 – received Outstanding Poster Presentation in Mathematics, Computer Science, Engineering, Geology, and Physics Posters.

� Tree-Based CAPTCHA, with John Wanies, Christopher Clement and Adrian Rusu. Poster at 9th Annual Rowan Science, Technology, Engineering, & Math (STEM) Student Research Symposium, Rowan University, Glassboro, NJ, April 21, 2006.

� A Human Interactive Proof Algorithm Using Handwriting Recognition, with Venu Govindaraju, Proc. of 8th International Conference on Document Analysis and Recognition (ICDAR 2005), IEEE Computer Society, ISBN 0-7695-2420-6, Vol. 2, pp. 967-971, 2005.

� Visual CAPTCHA with Handwritten Image Analysis, with Venu Govindaraju, Proc. of 2nd International Workshop on Human Interactive Proofs (HIP 2005), LNCS Vol. 3517, pp. 42-52, 2005.

� Challenges that handwritten text images pose to computers and new practical applications, with VenuGovindaraju. Proc. of IS&T/SPIE 17th Annual Symposium Electronic Imaging Science and Technology, Conference on Document Recognition and Retrieval XII, SPIE Vol. 5676, pp. 84-91, 2005.

� Handwriting Word Recognition: A New CAPTCHA Challenge, with Venu Govindaraju. Proc. of 5th International Conference on Knowledge Based Computer Systems, Allied Publishers Pvt. Ltd, pp. 347-357, 2004.

� Handwritten CAPTCHA: Using the Difference in the Abilities of Humans and Machines in Reading Handwritten Words, with Venu Govindaraju. Proc. of 9th IAPR International Workshop on Frontiers of Handwriting Recognition (IWFHR 2004), IEEE Computer Society, ISBN 0-7695-2187- 8, pp. 226-231, 2004.

63

Thank You

Documents

Exploiting the Gap Between Human and Machine Abilities in ...govind/amalia_dissertation.pdf · first character ‘w’ and the image between: - segments 1 and 4 is 5.0 - segments