BE'IRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. VOL. 16. NO. IO, OCTOBER 1994

Control Structure for Interpreting Handwritten Addresses

Edward Cohen. Jonathan J. Hull, and Sargur N. Srihari

Abstract-This correspondence describes the control structure for anintelligent handwritten address interpretalion system. The system takesa greyJevel address image, segments lhe address into llnes and words,parses the address into meaningful syntactic categories, recognizes wordsusing dynamically generated lexicons, and determines the destination codewith the aid of postal dlrectories.

Indcx Terms-Text processing, handwriting recognition, image process-ing

I. INTRODUCTION

The Handwriuen Address Interpretation System (HWAIS) takes anoff-line handwritten postal address image and determines a uniquemail delivery point (e.g., a mailbox). ln the United States, each

Manuscript received May 20, 1992: revised February 21, 1994. Recom-nrended for acceptance by Associate Editor T. Dean.

E. Cohen is with Accu-Sort Systems Inc..5l I School House Road, Telford,PA 18969 USA; e-mail: cohen@cis.udel.edu.

J.J. Hull, and S.N. Srihari are with the Center for Documenl Analysisand Recognition (CEDAR1. State Universiry of New York ar Buffalo, i26Bell Hall, Butillo, NY t4260 USA; e-mail: hull@cs.buffalo.edu and sri-hari@cs-buffalo.edu.

IEEE Loe Number 940525 l.

0r62-8828/94$04.00 @ 1994 |EEE

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. VOL. I6. NO. IO. OCTOBER I994

&,3 t"tf . *,, **** 1'. tf , - . .s-$t

I ,$m 'F**e*o

n i f : ! t { } f : : t .q4 I t . t r' s - Y - - . -

5; l t?

{ c )

F ixarnp les o l har t t iu r i t l c r r rd t l ress l l l ocks ex l rac led l ionr l i ve rna i l

del iverv point hus been ussigneri a nuntber. a del iverv point cocletDPC). which rs a ZIP (-ode- u' i th lppenclecl digits. The HWAIS nrustnorrlr l l l ) locrte and |elcl lhr- ZIP Cocle irnd strcet adclress or P.O. Boxnurlbcr to dctennine the DPC. Crn, lnd slatc names r lay be usetl torcso lve arnb igu i t ies . Eranrp lcs o l address t rken f ro r l l i ve n ta i l l shorvnin I i ig . I t i l l L rs t l i te sonre o l the cha l lenges .

' l ' h is cor rcspondence

ou l l ines th r ' I l \ \ 'A lS cont r 'o l s t fuc t t i re rv i th ex ln rp les . Sec l l l . [ ] l1br g |ea t r ' r ' t l t ' l i i l s l rnc l ! ' \ ( cns ions tu o ther donr l rns .

Thc Lrnited Sti t tL' \ Posl l l Ser-r ' ice tLISPS t funded the re sclrchdescr ibcd hc ' r t ' t ( lF .D, {R) and l *o s i rn i la l e f lb r ts (ERIM and CCiA; .Wlt i lc the ot l ter groLrps hlrd cornpar-ublL- perlbrnlance of ZIP Codelocation und recogrrrt ion (6(r( i vs. CEDr\R's 76{4 ). our rcsearc}rwas the f i lst i r t tcurpt to clctclnrine DPC l lorn handwrit ten addresses(thc othei groups havr' not puhl ished t igules for corrparable sys-I e r l r s ) .

' l -his col lcspondence is olgitnizeci as f ir l lows. Sections I l de-

scr ibes l lgo l i th rns . Sec t ion l l l de tu i l s horv the svs ten t p rocessestu'o huntlr i r i l tc 'n i tdcl less cxlrnples. Section lV describes per' fonnlncecva lua t io r r an t l c rper in re l r t l i l r csu l ts .

l l HW,{lS Al_c;() ' i l iH\ ' l Dr--scRrr,Tlol

The cont lo l f lo r r in the cur rcn t l lWAlS has tcn operar iona l s taees .

,1 ) Prcploccssinu:B.l l -cxt l ine separlt iorr bascti orr posit ionl l inf olnrat ion (se parating

i ln at i i l rcs,s into te\ l l i l lL 'si :( - i Fcr r tu rc i r l cn t i l i cu t ion :Dt l ' c r t l i r re 'epara t ion u \ in ! I c i r tu re ident i f i ca t ion (cor lb in inc B

and C r :E i

- le r t l i nc ' \ r ' !n rcn t i t i i ( ) r r ( t l i r i r l i nc l in rs in l0 \ \o rd \ ) :

l ] t \ \ ' o rd c lu : : i l j ca t ion :Ct t irading nrt lch oi r .r 'ort i t ia:sihczrt ion to \plt i i l l \ \ ,ntax:l l i J {eco t r r i z ins u o rc ls thu t h r rve : r h igh s \ n tax c lass i t i ca t ion

t t l : t i r ' h :

/ ; l - . i n l ! i i t r : i n t i i i \ l i i r ( ) n \ t f : r i n \ \ ( l j - d r ec0gn i t iOn

. l r l 1 l t l c i i l b l c \ - l l i n t i e c r r r l i r a l i t t n i s t i t L i r r d . r l e - r r r e un i n t e rp l e -

i i r l i0n. ot l ' rc I r , . i :c . l .c l tuf t ) to \ ie l t H Lint i l no re l i rb i t - s) 'nta\-r l a . , i l t t l r l i r r n : l : i t chc . ! l t i t l i r ; t i l ab l t .

ffi"!'ffi,$$$r&-d

t$*ud,\{"{_" #,y*,

( d )

P re pnre ssi tr,g

lnput int lges are uddrcss blocks front l ivc nrai l scanned at 3(X)p ixe ls per inch u i th 2 -56 sh lc lcs t l l ' g re r ' . T ' , io p reprocess ing s teps(thresholcl in-s l3l ancl guidc l inc' r 'ernoval) are pelfornrcd lo producea binary irra,ue (whe le text pixe ls are black).

I rr i t i u I Ti ' .vt Li t tc St, purut iot t

D iv id ing the ac ld ress in to t r - \ l l i nes lcqu i res l \ \o s l i tees . Pos i t ionu linfornral ion sepi lr i l tes "easr'- lo-gt l i rce" conrponent\. sta-9e D usesfeature iclenti t ica{ion to ref irrc those results.

L ine sepa lar ion (F ig .2 ) d i r i c lcs a b in i r rv in rase rn to ver t i ca l s t r ips .Each st l ip is converted to i l scl ol ' blocks. rvhelc' blocks indicatethe presence of tert. Overl lppinu blocks f lont ad.jacc-nr slr ips rreconnecte(l to t i l rnr a topolo,uical l l l rph reprcsenting lcxt l ines (Fig.3). rvhere blocks and block connectron\ ure rclrcserrted as lrodes anrinode connectiorts respectivelt ' . Hc'urisl ics adjust the gmph to corrcctfo r typ icn l s i tu t t ions (e .g . . l scenders) .

After f inal izing the block arrangenrent. connectcd contponents areassigned text l ines based on their corresponding blocks. For exanrplc.a component locatecl in blocks l ' rorn a singlc texl is assigncd l()that l ine. A component's rvhosc blocks overl lrp severir l text l incs isass i -qned to l se t o l ' l i nes- Snr l l l components l cons idered no ise) a rc no lass igned to l incs . ; \ l l l i ne l ss ignrnents ( inc luc l in .g no ise cor rponents tnray be rehnecl in stage D.

F t ul rrrt' I I L' rt t ilir'u t i t t tt

Featurc ident i l i ca t ion re r ;L r i rc r shrpc idc -n t i l i ca t ion an( i spa t ia l con-hr rna t ion . ' l i r i rnprove specc l . n ros t p rocess inu r t -qu i r i r rg in tage in lo l -mat ion (c ._u . . shrpe ident i f i cu t ion r i s r lonr ' l i r s t : spu t r l l con f i rn ta t io r rrequ i res on lv svnrbo l i c in lb l r la t ion . Each connec lec l conrponent i :i n i t i a l l l , i d c n t i f i e d a s h a v i n u t o r n o l h r r r i n g ) a p u r r i c u l a r s h a p e t i . r - . .d i tshes . cornnr ls . c l i sconnect r .d . r -hu ts ' and d ig i ts ) . L )u f ing th i \ s tag t .

lH l l t t lu r i t t cn , s e rc r r f t tn r r r i t te 'n in l \ \ { ) \ t r ( rk ( . \ . - l -hc

r r ' r t i c r l b r r and t i rc

bo t ton l cu f \ r f , rc \ \ r i l tL ' l t i r t r l t c ' l roLc anr j lhc l t0 r rzor i l l r l h ; r r i s r r r i i l cn : , i: I separa tL- s l r0kr ' . \ \ ' l t t ' n thc ' l ro r iz i tn t l l \ l t ( )kc i \ t )o l eon i ld ! l cd to the o thers t rokc- the hor izor r l r l \ l fok ! - i \ eon : ic lc red lo bc r r l i : cor l rc r tcd r -ha t . See F tg .:l( a ).

IEEE TR, \NSACTIONS O\ P , {TTERN A\AL\ 'S IS A\D \ {ACHI \E I \TELLICENCE. VOL. I6 . \O. I ( I . OCTOBER I99J

ffiu:rq;Fig. 2. L ine separat ionand the lef t s ide shows a text component uscd to c letcrnr ine the hei-eht . Each texr l inc in thc acldress has a s l ight lv . l i f f . r "nt color shading. indicat ingthat the shading al_eor i thnt deternr ined the l ine struclure correct ly .

l rT\ |

it54nt@rrlf t t r llLbrlFdru

ll l l t lI .]* | _-__, I -+- I| | -t'+- -=___J t I f *F-.+--l

t t t l

I j -+' l I lI | | r t I * f - I| ++-t__l I l+-f l

iEn rlaTrrj) l -IC,,FqF-]

( d )

Fig. 3. Line scparation heuristics stages: (a) ori-einal image and the blockscreated from il, (b) topological _rraph crcatcd from original block set (doublearrows indicate connections between nodes). (c) topological graph aftcrheuristic rnoditications. (d) final block ser which allows each worcl to beassigned to a separate line.

components may have mul t ip le ident i f icat ions based on shapes. Forrnstance, a short hor izontal bar may be ident i f ied as having the shapeof a dash and a disconnected . r -har (as in Fig. 4) .

Spatial information refines decisions. For example, a componentwi th a d isconnected J-har shape must have a component (C) to i tsimmediate left in the same rext line, C' must either be identified asthe digit i or as an unrecognized digit, C' must not be a comma, dashor disconnected i-hat (see Fig. 4).

)--

Fig. -1. Exarnples of leature ident i f icat i t rn: (a) hor izontal bar has shape ofdisconnected i-hat or dash. but locarion next to a .-r shaped cligit i<Jentifies baras a disconnected ; -hat . (a) hor izontal bar has shape of d isconnected $5$-hator dash. hut locat ion el imiuates disconnected i -hat possib i l i tv and suggestsa dash .

,.dio*Prli.lJ nJ?dn

Final Line Separation

Component position information (used in stase B) can be insuf'-f icient to separate l ines. Componenrs from oni , .*, l ine may bepositioned closer to another (e.g., commas are often closer to thetext line beneath their own). Heuristics are used to make new lineassignments for such components.

Other components (typically characters from adjacent text lines rhattouch) must be forcibly segmented (by segmentation lines) and placedin more than one text line. This segmentation line is determined by

Fig. 5. Different types of word breaks.

minimizin-q the mean square error in a first-order polynomial whosedata points are comers of bounding boxes of components alreadyassigned to a single line. Each segment of a split component isassigned to its nearest text line.

lR- U6

$-+"r-a-t e 73 FNJ-/r{r-rLf a 73,

?1r-#"e!.i#,"ef"-#"Ur-#"e!^-{^*e&,#"eI^,#*e-P^,,#,"e!^;ff"aL-:#"

N&,{f^-t aN7N/dr{rr{-tN*a 7

44,$^*tNAN7

s*/+rartW A 7

N&r{f't-5*NAN7

&-ttov*f J,N2J,(

0ir-r^-tNa 7 h

FFF,

r-F

J+rtrzqNt J.,7

3333JJ

J

ANALYSiS AND MACHINE INTELLIGENCE, VOL. I6 . NO. IO, OCTOBER i991

TABLE, IREsr:rrs oF READIN6 Ztp CIDE Wnrrrex oN HexowntTtEN ADDREssES

Correctly read imases found in the USPS directorv 746 74-OVo

Correctly read images not found in rhe USPS directory l 8 l-8Vo

Total Correct 7g 75.n%Reiected images that contained ZP Codes 22l 2r.9%

Rejected images that contained no ZP Code l l 2.2%Total Reiecl 232 23.0%

Errors in images that contained ZP Codes I I t . 1 %Errors in imases that did not contain ZIP Codes I O. lVo

Total Error t2 1.2%

Total imapes l(m r00-0%

have characterist ics of a "city" word. meaning i t must be classif iedas a "c i ty " o r " tex t " . bu t the "c i t1 " ' c lass i l i ca t ion wou lc l have a h igherconfidence match. State narne and ZIP Code are rnatched sintr lar ly.

Matches between image text and svntax are graded by rank-ing high-confidence syntax rnarches rvirh a hi,uhly probable synrarover low-confidence svntax l l t l tches *i th less probable s1,ntax.

' I 'he

IIWAIS svstern Dlatches asainst l i re dif lerent syntaxes: P.O. + Bo.r+ tt trntber (alf on one l iney. stratt-rrt t t i lDat + sltrrel-nunte (al] on oneline). ZIP Ctxle \on l-2 l ines. -s-digit ZIP Cule rnd 4-digit add-onrnay be on separate l ines). r ' l t .r 'n(nt( + stut( t t t t t tL + ZIP Cotle ( ' ,ytl-- : l l ines). and stutc num( + Zl P Code (on I-3 l ines). The HWAIStries to t ind the ntost cornplele ntatch possible (e._u.. tr r i l , r 'nonc +stut? nant( + ZIP C.txlc nratch u, i l l be choscn oter t7, lP Cr.rr le matcht.

R e c o g t t i : i n g lN'ortl.s

Nunrbers (ZIP Code. P.O. Box nunrbcr or street nuniber) aresegnrented into digits and recognizecl. For sratc-t ibbreviat ions. nse-qnrent-and-classify algori thrn designed t irr charac(ers is usedl andlol other w'orcl tvpcs. a dict ionarv is usetl to lssist ir t recoenit ion.S e e l l l f o r d e t a i l s .

IJ.si tt g Scnruntit l nlltrntatiort to (' ottstruirt ll rtrtl fl et ogtti ti ott

Once al l relevant r.r 'ords have passed rhrough the init ial recognit ionphuse. semantic inforrnation is used to intprovc rccognit ion perfbr-rr lnce. ZIP Codes are chcckcd against a LrSI)S directory. Only 4.1. iof possible -5-digit ZIP Cotles arc assi-9ned: so roughly,5692 of errorslure clctected.

I l ' the state nanrc is rcco_snized. ir is conrparcd to the f irst tw,o ZIPC--ode digrts and sevelal act ions can be takcn. Whcn state name andZIP Code asree. the ZIP Codc is accepted as correcr even i f digits arerecognized rvith lcrv confidence. l l ' the state nanre is recognized withhi-eh confidence and the f irst arrd/or second digit of rhe ZIP Code donot have high-confidence recosrr i t ion. the ZIP Ccxle rvi l l be correctedto con'espond to the state narne i f possible. l f rhe state nanre and ZIPCode are recoenized rvirh hish confidence and thev confl ict, the LlPCode is accepted as correcl since irs recognit ion is more rel iable.

The HWAIS uses inforrnatiun from strcet nurnbers and P.O. Boxnurnbers. I f a 9-digit ZIP Cocle is recognizcd, i ts 4-digir add-on can becontirrned bv recognizing rhe P.O. Box nurnbcr. I f a 5-digit ZIP Codeis rccosnized. i ts. l-diuit arld,on can be deterntinecl bl,recognizing t lreP.O. Box nuntber. The' HSAIS can locatc ancl recognize P.O. Boxnunrbers in -5792 of rhe inrages u,hich contain thent.

l i .r ZIP Code lncl street nuntber of urr lddress are determinc-d.a cl ict ionarv of pt issiblc street nal les is extractct l from a USPSdircctor\ ' . When a streel name is chosen. the HWAIS uses t irerecognized ZIP Cocle an(l slreet address (street nunrber and nlr.rc)to assign a 4-digit arl t l -orr. Adding the last t \r 'o digirs of the streetnurlber ro thc'ZIP+-l C'ode creates the DPC. On aclcjresses containirrgstrcel addresses u,hose ZIP Codes rvere aulorlat ical l l located andrecognizetl . the H\\AIS locates and recosnizes l lgi of the strc.eladdre,rses.

lEEE TRANSACTIONS ON PATTERN

Fig.6. Dif lcrent \ \a\\ ol sL'prr i l l i t ]3 I t(- \ t l i l rc into wrtrr ls. Trv0 laruest urpsgive sixth,uroupinr lnrnr top. Corrcet choice ( l i rsl and tburth l i rreest ! : i r l )5)is second frcl ln botlorrr.

'l'a.t t I-i rt a Sc.qttt r' rt ! tt I i r t rt

Components o l cuch te r t l i ne r rc 'c lus lc rec l in to u ,o rds to c rL . i l t cdifTerent hvpotheses. Wurcls ure f i ;rnted bv ordering a text l ine'sconiponcnls frorn lef l to r ight and sc-lcct ing rvord breaks bctuecrrcomponents. Word-brclk locations trrc hvpothcsized based on lror-iz-ontal spaces between conrponenls. a shif t in t l re mean r,r.r t icalpos i t ion o l cornponcnts . and the locu t ion o f cornn tas (see F ig . 5 ) .

Worcl-brerk Iocirt ions ure orcleret l bused on their distarrces (acornma-gap r lrstance is conrpute(l as the sum ol distances on ci l l rr .rside of the c() l l l lnal a shif t in nrL-an \,ert ical posit ion is aluavs a u'ordbreak). Di l l trent worcl-break groupings 1Fig. 6) ure chosen ro scparit tL'a l ine into l--5 ivords (e.9.. t trr lhree uords. use the top t\ \ 'o $idestrvord breaks for one set. usL- f irst and third uiclest rrord breaks l i rranother set).

l|ord C lassificorion

Each u'orcl hypothesized bl text l ine scgmentation is classihed(into one or rnore categories rvith assigned conl idence values) baseclon features ancl spatial locations of i ts conrponents. Word categoriesare ( l turu( t( t . l ( includins lr nuntbcr'-ol ' -chlracters estint l te). ( i / \ ' .stt t ta ubltrcyit t t i r tr t . t l i .qi ts { including u nurnbcr ol ' -digits est intalc ).digirs l l t l l t tnct l- l t . t- t luslt . / , lP+4 ( ' tnlc:. lxrrcr i l t ' . r . P.O.. Br;, i . lndnrii.;c .

Grudi rt,q lI ottl Cutc,qttri(,.f /(/ .t.\'/r1rl.\

Ou l s rn tax is t l l o - r l in ten ,s ion l l . s ince sor t l s a re ho l i zon ta l l r o r -dered a long u tex t I ine anc i te r t l i ncs a re ver t i ca l l v o rdered. ( )necomnron hlnt l* ' l i t ten address svntax hls the bottorn nrost l ine corr-ta in ins th rcc u 'o rds . c i r r . s ra re . and Z IP C ' tx lc (CSZt . I ro r rh is s ln r l rfo be prescnt. the bonont-nrost l ine l l tu\t be dir isible into three r iurr lsand each nru:t bt 'ntatched to i ts erpected fornr.

- fhe f irst uord ntust

IEF-E TRANS. \CTIONS O\ P \ I 'TER\ A \ ,AL\SIS A\D \ IACHI \E INTELLIGENCE. \JOL I6 - NO. IO. OCTOBER I99 .1

\--,ou, f",,* ga*"'P{'^ue-Rc.Bq( 5C:1r

Tc\ii" 7O\r'o\lG\:

8o .bq( 5Gj1

\rL\3

Box 1.00state abbrev 1.002 chars 1 .002 digits 0 34

( l ' )

text fol lowed by a comnra) and 7-chars u' i th high confidence (1.00).

Similarly,, the second word is classi l ied as PO (a short rvord with acomponent recognized as a zero at the end). Bo-t (a short *'ord), and -lclars (the i is considered to be noise ) r'i,ith high confidence. The wordis also considered to be 4 digits \ \ ' i th nroderate confidence (0.66) because the f irst and last connected components arc recognized as zeros.

' fhis word segmentation results in a bottorn-l ine syntax of ci t- t ,--l-r'hors-S rlrgit.r (c) correspondin-e to (it\'-state-/1P. Based on thismatch, the system selects the -5 cllgrts rvord as the ZIP Code candidate.The ZIP Code candidate is segmented (d) and isolated digits arerecognized. Digit recognit ion results and their confidences are shownin (e). The recognized ZIP Code 4J6lJ is checked against the USPSZIP Code directory and fbund to be val id, but i t would ordinari lybe rejected because the f irst di-ci t (which looks l ike a 4 or a 9 ) isrecognized as a 4 with low confidence (0-299). However, the state

To\tu!" I O\'o \rGtrPO 1.00 5 d ig i tsBox 1 .oo Box3 chars 1 .00 5 chars4 digits 0.66

( c )

Modified4 0.9993 0.9886 0.9971 0.9943 0.994

( c i

5 0.9966 0.9963 0.999I0 .996

( g )

43613-0639

Fig.7. Example ol 'HWAIS proccssing nn ima-ec. (a) Grey- level in iage. (b) Thresholded inage. (c) Ci t1, State. and ZIP Codc Candidates. (d) SegmentedZ I P C o d e . ( e ) D i - e i t s r c c o g n i t i o n r e S u | l s t b r Z I P C o d c ( d i g i t r e s u l t a n c l c o r r e c l c o n f d e n c e ) a n c l a d j u s t e c l d i g i t r t . c o g n i t i o rand Box-number Candic latcs. (g) Di-s i t recosni( ion resul ts of P.O Box nunrber. (h) 9 d ig i t ZIP+4 Code (u 'h ic l t is a DPC).

I I I . ExAMpLES or HWAIS oN AN ADDRESS lulaces

Figs.7 and 8 shorv sorne intermediate steps of images processedby the HWAIS. The f irst exanrple shows hou' a poorly recognizedZIP Code has i ts confidence increased by the state name recognit ionresults and how the P.O. Box number is located and recognized todetermine a DPC. The second example demonstrates horv the I'IWAIScan correctly locate a ZIP Code atier ini t ial ly fai l ing to do so, andhow a street address can deterrnine a DPC.

The grey level for the f i lst example (Fig. 7) is image shown in (a).and the thresholded image in (b). No guide l ine removal is requiredfor this image; and, the address is separated into text Iines and words.

Each text line is segmented into several different word groupings.Word classifications for one three-word grouping of the line areshown in (c). The first word is classified as cirv (because it has

2 d ig i ts 1 .00Box 1 .00PO 0.80state abrev 0.802 chars 0.80

4 d i g i t s 1 . 0 0Box 1 .004 chars 0.80

1 . 0 01 .000.80

city 1.007 chars 1 .00

Original4 0.2993 0.9886 0.9971 0.9943 0.994

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. I6, NO. IO, OCTOBER I994

#_**{ ?idd&S*** 0",aW'-^l

| fr

* } t 'Y

sli;;.*{dq f*o*.u* i&tr : r.r {1/1\e^ Li I t-u J

&

{ {v#- X" ff_

'ffrn-*'"". ilt^ 3ts L ?/1^; brt,

"(fr!^ t*,r,"^ ) Tha. o tcf-'

c/:,{(b)

5 chars5 digils

1 . 0 00.24

1 . 0 00 . 1 1

fafSfNCNI-NqN5ffi!4 t--,"^ )

5 chars 1 .005 digits 0.1 1

1 . 0 00.800.64

c rg55 chars 0 .606 digits 0.24

Box2 charsstate abbrev

( r )

c rafSCNfNONdNS

0 0.9921 0.9930 0.9938 0.9955 0.990

( h )

Fig. t t . Exatrp ie of HWAIS proccssing an image. (a) Grev- level image. (b) Thresholded inage. (c) Fi rsr h l ,porhesis for c i rv. srate, and ZIP Code candir lares.( d ) F i I s l s e g n r e n t e d 7 - I P C o d e . 1 e 1 F i r s t d i g i t s r e c o g n i 1 i c l n r e s u l t s f o r Z l P C o d e ( d i g i t r e s u l t a n d c o r r c c t c o n t i d e n c e ) . ( f ) S e c o n c l l r y p o t h e s i sZIP Code candidates. (g) Seconci se-qmenled ZIP Code. (h) Second dig i ts recogni t ion resul ts for ZIP Code (c l ig i t resul t and corrcct conf idence).

name is recognized as Ohio and since al l Ohio ZIP Codes begin withthe digit J. the confidence of the l i rst digit is boosted. and the ZIPCode is accepted.

The H\\IAIS must also interpret the P.O. Box l ine ro delermine aDPC. Thc middlc l ine of the address is parsed to P.O. Bo.r4-digits(f) rvhich nratches P.O.-Bo.r,Bo.r-nr.u' t tbt,r. This l ine could also matchto 2 r/i,qit.r 2-thars,4-churs for a street address ltne (street-number-

s!) '(et-nant( ), but the P.O. Box l ine ntatch has higher confidence.The Bor number is reco-unized as 5639 19) r.vi th high confidence.

ancl the 4-digit add-on 0639 is derermined using a lookup in a USPSdirectory. In this instance. the ZIP Code and the 4-dieir add-on formrhe 9-d ig i t DPC (h t .

( g )

{ l: *

"fukb^) ^tO tw5PO 0.80 Boxstale abbrev 0.80 5 digits3 chars 0.804 d ig i ts O.24

( c )

1 0.9935 0.4382 0 .910I0 .8055 0.990

( e )

The second example (F ig .8 ) shows rhe grey- leve l image (a ) andthe thresholded image (b). Each text l ine is separated and wordhypotheses within each l ine are created.

After classif icat ion. the top parsin-e hvpothesis for the cirr, , .r /arz,-ZIP for this address irnage is found in the second I ine frorr thebottom. One three word hyporhesis (c) fbr this l ine locares J t iar ' . i -slalc-abbreyiat ion-5-digirs as a syntax ntatch. This match is incorrect,since the f irst digit of the ZIP Code is included in rhe srare name.This mistake occurs because rhe ZIP Code contains a connecredcomponent lconsist ing of two Iarge touching di-eits. 0 and 8) thatis est imated to contain three digits (making the last rvord a -5,digitsu'ord). ln addit ion. the nt iddle u'ord has tu,o nrediurn-sized connected

IEEE TRANS. \CTIONS ON P\TTERN ANAT, \ 'S IS AND I \ I , \C I I INE INTELLIGENCE. \ 'O t - . I6 . NO l0_ ocToBER l99l I 0 i )

318 CITY VIEW BLVD .4006318 E MAIN ST -331 ' l318 E MOUNTAIN RD -1853318 LOCKHOUSE RD .1212

{ k )

3tr L yt4 8tu8 digits 0.04

-3311-1 853-4006-1212

3 d ig i ts 1 .00Box 1 .003 chars 0.80

{ l )

318 E MAIN ST318 E MOUNTAIN RD318 CITY VIEW BLVD318 LOCKHOUSE RD

3 0.9861 0.9988 0.999

(-1,

01085-331 1-18( l ) (m) ( n t

Fig. 8. (cont inued). Exanrple of HWAIS processing an image. ( i ) Street number ant l name candidates. ( ) Street number d i -e i t recogni t ion resul ts. (k) Or ig inalstreet name dict ionarv and - l -d ig i t addons. ( l ) Ranked strect name dict ionarv and 4-dig i t addons. (m) 9-dig i t ZIP+4 Code. (nt Final DPC.

01085-331 1

components which proviclcs a -{ood match to the .rtatc abbrcviationclassi f icat ion.

The ZIP Code fbr this parsc is segrnented (d) and the digit rcsultsare shown in (e). Digit recognit ion results are not al l high confidencc.and the state namc is not recognized: so no digit corrections arc madcand the ZIP Code candidate is rejected.

The second parsinr hypothesis chosen for the cir.y-state-ZlP isthe correct one. This parse ot ' the same l ine into three ivords givesa syntax rnatch of 5-r'hars-sttttt,-ubltrcyitttion-6-rlrgits. 1'he 6-digit.rnratch to the ZIP Codc is al lowed because digit cst inrat iorr r.rrsrequired, and the HWAIS kno,,vs estimation erors nray occur. TheZIP Code is segmented (g) and al l digits are recognized with highconfidence (h). The state narne is st i l l not recognized. so i t does rrotaffect the digit confidences.

The street address l inc ( i . t is correctly located bl matching thesecond line. fronr thc top. to -i-zli,grt.s--5-r'har'.s. Thc street nuntberis correctly ' recognized as - l l8 with high confidence (). Usin-t :rUSPS directory. the street number, and the 5-digit ZIP Code. theHWAIS creates a dict ionary' (k). This dict ionary contains al l val idstreet names in the 01805 ZIP Code area whose street number is 318.Their associated 4-digit adcl-ons (k) are also shown. Thc abbre." ' iat ionsin the street names are expanded (e.g., E MAIN ST becomes EASTMAiN ST, EAST MAIN STR. EAS'| MAIN STREET. E MAIN S1'.E MAIN STR, etc.) to create an expanded dict ionary. This dict ionarl,and the street name image are passed to the "word recognit ion withdict ionary" algori thm producing a ranked dict ionary ( l) . Confidencelevels of the ranked dictionary indicate that E. Main Sr. is the correctstreet name. A ZIP+4 Cocle is created by adding the 4-digit add-onto the 5-digit ZIP Code (m). The last two digits of the street number(18) are then added to the ZIP+4 Code to create the f inal DPC (n).

IV. PERFoRMANCE EVALUATIoN

The performance of the HWAIS is described in two phases, readingthe ZIP Code present on the mail piece and assigning a DPC.

The system performance for reading ZIP Codes on a set of 1008images of live mail addresses is 75.8o/o correct with a l.2Eo enor ruIe

(Table I). Since al l DPC assignrnent b) ' our HWAIS rel ies on ZIPCode determination, the 75.8%, represents an upper bound on DPCassignment.

The system performance on DPC uas tested on another setof 973 images (al l addressctl to locations inside Buffalo). Theseaddresses were chosen because only Butfalo-area DPC directoriesivere avai lable during test ing. ZIP Code recognit ion performance forthese irrrages is 67.4%, correct with l . l lc,/c errors. The drop in ZIPCodc recognit ion performance was due to the nature ol ' the ZIP Codes

\ i . c . .Z lP Codes in th is se t con ta incd manv 4 ' s and 2 ' s , 14222 wasmost common; and our digit recognit ion algori thnr perforrns rvorston 4 ' s and 2 's ) .

DPC performance varied fronr 33.17c correct with 19.6010 errorsto 24.4o/c correct with 4.3%, errors. depending upon the acceptancethreshold level. This performance is the current state of the artfor ful ly automatic systems thal detcrr l ine DPC's from handwrit tenUnited States' addresscs. While insuff icient fbr practical appl icat ions.continuin-q ef ' forts are underway to improve pcrfornrance.

ACKNOWLEDC i\I EN'I-

We would l ike to thank M.-Y. Clhen. A. Comnrike. P. Cul len, J.

Favata. R. Fenr ich. E. Kle inberg. C.-C. Kuan. A. Kundu. D.-S. Lee,

C. O'Connor, G. Seni . J . Tan and J. Zhou for contr ibut ing ideas and

code towards the HWAIS. This u 'ork w'as supported by the USPS

under Task Order 104230-86M-1990.

REFERENCES

I I ] E. Cohen, "lnterpreting handwritten text in a constrained domain." Tech.Rep. 92-{6, State Univ. of New York at Buffalo. Feb. 1992.

l2 l E, . Cohen, J. J . Hul l . and S. N. Sr ihar i . ' 'Understanding handwri t ten textin a structured environment: Detennining zip codes from addresses." inChat-acter and Handu'ritirtg Retognition: E.rpanding Frorrtiert, P. S. P.Wang, Ed. Singapore: World Scient i f ic . 1991r pp. 221-264.

[3] N. Otsu, "A threshold select ion method f rom gray- level h istograms."IEEE Trans. Syst . , Man Cvbern. , vol . SMC-9, no. I . pp. 63-66. Jan.t 979 .