Recovery of drawing order from single-stroke handwriting ...visgraph.cs.ust.hk/biometrics/Papers/Signature/pami2000-09-01.pdf · Recovery of Drawing Order from Single-Stroke Handwriting

Recovery of Drawing Order fromSingle-Stroke Handwriting Images

Yoshiharu Kato, Member, IEEE Computer Society, and Makoto Yasuhara, Member, IEEE

AbstractÐThis paper describes a new method to recover a drawing order of a handwritten script from a static 2D image. The script

should be written in a single stroke and may include double-traced lines. After the script is scanned in and preprocessed, we apply our

recovery method which consists of two phases. In the first phase, we globally analyze the graph constructed from the skeletal image

and label the graph by determining the types of each edge. In the second phase, we trace the graph from the start vertex to the end

vertex using the labeling information. This method does not enumerate the possible cases, for example, by solving the traveling

salesman problem and, therefore, does not cause a combinatorial explosion even if the script is very complex. By recovering a drawing

order of a handwritten script, the temporal information can be recovered from a static 2D image. Hence, this method will be used as a

bridge from the offline handwriting character recognition problem to the online one.

Index TermsÐHandwriting, cursive script, character recognition, stroke recovery, temporal information, image processing, OCR.

æ

1 INTRODUCTION

WHEN compared with online handwriting characterrecognition which uses special input devices such as

tablets, offline handwriting character recognition is muchmore difficult because the only available information is astatic two-dimensional (2D) image obtained by scanningprewritten characters on a paper; the dynamic informationof the pen-tip (stylus) movement such as pen-tip coordi-nates, pressure, velocity, acceleration, and pen-up and pen-down can be captured by a tablet in real time but not by animage scanner. The offline method, therefore, needs toapply complex image processing techniques to segment andanalyze character shape for feature extraction.

The online method is based on the availability of a one-dimensional (1D) vector whose elements are pairs of �x; y�coordinates of the pen-tip. On the other hand, the offlinemethod can be considered as an attempt to recognize theimage of this 1D vector when projected onto paper, with notemporal information. Therefore, if the drawing order isrecovered from the static 2D image as shown in Fig. 1a toobtain the vector data as shown in Fig. 1b, the offlinerecognition problem can be turned into the online one,which is based on the temporal information, resulting inbetter recognition capability. Thus, the recovery of thetemporal information can be considered a bridge from theoffline method to the online one.

In this paper, we propose a new approach to recover adrawing order of a single-stroke handwritten script from astatic 2D image. This method does not cause a combinator-ial explosion, unlike the previously proposed methods,even if a script is very complex.

This paper is organized as follows: In Section 2, therelated works are described briefly. In Section 3, thedrawing order recovery problem is described. In Section4, the proposed method is described. In Section 5, theexperimental results are presented. In Section 6, thediscussions on the recovery method are made. Theconclusions are presented in Section 7.

2 PREVIOUS WORKS

Lee and Pan traced signatures in an offline environmentwith a set of heuristic rules [1], [2]. Govindaraju et al. tracedhandwritten words by determining the smoothest path ateach intersection with a set of heuristic rules [3], [4].Doermann and Rosenfeld provided a taxonomy of local,regional, and global clues to recover temporal clues fromthe static image [5], [6], [7]. Their method is writinginstrument dependent and needs to detect substroke-levelfeatures from the gray-scale image. One of the objectives ofour study is to clarify what kinds of strokes can berecovered and what kinds of strokes cannot if we use onlya minimum set of heuristics. Hence, if there are some kindsof strokes which cannot be recovered by our method andneed to be applied more heuristics, the results of their studymay be considered to complement our approach. Boc-cignone et al. reconstructed the most likely trajectoriesfollowed by the writer according to good continuity criteriataking into account direction, length, and width of thestrokes [8]. Abuhaiba et al. processed Arabic script andrecovered the temporal information by traversing the graphconstructed from a stroke [9], [10], [11]. Privitera andPlamondon recovered the original motor-temporal infor-maion from the images of handwritten words based on theshape analysis without involving any thinning process [12],[13]. Huang and Yasuhara recovered the temporal informa-tion from static images of single-stroke cursive scriptsaccording to good continuity criteria calculated by use ofthe SLALOM [14], [15] approximation method [16], [17].

938 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 22, NO. 9, SEPTEMBER 2000

. The authors are with the Deptartment of Information Management Science,Graduate School of Information Systems, the University of Electro-Communications, 1-5-1 Chofugaoka, Chofu, Tokyo 182-8585, Japan.E-mail: {kato, yas}@math-sys.is.uec.ac.jp.

Manuscript received 22 Oct. 1998; revised 22 Jan. 2000; accepted 31 Mar.2000.Recommended for acceptance by R. Plamondon.For information on obtaining reprints of this article, please send e-mailto:[email protected], and reference IEEECS Log Number 108107.

0162-8828/00/$10.00 ß 2000 IEEE

Their method evaluated smoothness, not in a local area ofcrossing points, but in the global area of the handwritingimage to overcome difficulties resulting from ambiguities ornoise at the crossing points. JaÈger recovered the temporalinformation by searching the minimum curvature bysolving a traveling salesman problem in a Hamiltonianpath created from the line graph of a skeleton image [18].Bunke et al. recovered the storke order from static hand-written images by searching a graph using the global andthe local criteria [19]. Lallican and Viard-Gaudin recoveredthe stroke order from static gray-scale images by applying aKalman filter to follow the stroke trajectory [20].

The Huang and Yasuhara's method and JaÈger's methodsolved the recovery problem by searching the graphscreated from the skeleton images to find the Eulerian orHamiltonian path having the minimum cost, respectively, asshown in Table 1. Huang and Yasuhara calculated theglobal smoothness based on the spline smoothing andreduced the complexity of the problem by decomposing theproblem into simpler subproblems where possible. JaÈgercalculated the global smoothness by calculating the anglesof intersecting lines. Both methods cause combinatorial

explosions when the scripts become complex. For example,

the script shown in Fig. 2a can be decomposed into four

simple subproblems, but the script shown in Fig. 2b cannot

be decomposed, causing a combinatorial explosion. In

contrast, our method presented in this paper does not

enumerate the possible cases and, therefore, does not cause

a combinatorial explosion. As summarized in Table 1, there

are two weak points in Huang and Yasuhara's work; one is

the lack of PD-line detection (which will be discussed in

Section 4.5.1) and the other is the computational complexity.

One of the purposes of our study is to address and solve

both problems.

3 DRAWING ORDER RECOVERY PROBLEM

In this section, we discuss the drawing order recovery

problem as well as the problems which have to be solved to

recover the drawing order.

3.1 Assumptions

The following assumptions are made to recover the

drawing order of a handwritten script:

KATO AND YASUHARA: RECOVERY OF DRAWING ORDER FROM SINGLE-STROKE HANDWRITING IMAGES 939

Fig. 1. Recovery of a drawing order. (a) An original image is scanned.

(b) The drawing order is recovered from the image.

TABLE 1Comparison of Two Related Methods

Fig. 2. Problem with previous works. (a) A script which can bedecomposed into subproblems and can be solved. (b) A script whichcannot be decomposed into subproblems and will cause a combinatorialexplosion.

1. The script is written in a single stroke as shown inTable 2a. A stroke is a trace of pen-tip movementwhich starts at pen-down and ends at pen-up.

2. The script must have a start point and an end pointas shown in Table 2b.

3. There exists no junction where more than two linesintersect as shown in Table 2c.

With the first assumption, we define the condition of thedrawing (writing). We begin with the recovery of drawingorders of single-stroke scripts and will extend a method formultistroke scripts in the following study. With the secondassumption, we exclude a script such as ª0,º ª6,º or ª8º; wedo not try to cut a loop at an appropriate point to make astart point and an end point. With the third assumption, weallow junctions with two intersecting lines. Although thereis no theoretical limitation to allow the junctions havingmore than two intersecting lines, for the sole purpose ofsimplicity we limit our discussion to the situations whereno more than two lines intersect at a junction.

The recovery method will convert the spacial informa-tion, which is represented by an array of pixels whose ONvalues corresponding to the pen trace, to the temporalinformation, which is represented by a vector of the (x, y)coordinates of the drawing order.

3.2 Unicursal Figure Problem

The recovery of the drawing order of a single-stroke scriptcan be seen as an application of the well-known unicursalfigure problem which can be solved by graph theory. Wedefine an undirected graph G � �V ;E�, where V �fv1; v2; . . . ; vkg is a set of vertices and E � fe1; e2; . . . ; elg is

a set of edges. An edge is represented by an unordered pair

of vertices, �vi; vj�, and vertex vi is said to be adjacent to the

vertex vj and vice versa. The degree of vertex vi is the

number of vertices adjacent to vi. A path from the vertex v1

to the vertex vn is represented by a sequence of edgese1; e2; . . . ; en and its length is nÿ 1. The adjacency matrix A is

a kV k � kV k matrix whose ijth element, A�i; j�, denotes the

number of edges between the vertices vi and vj.If a connected graph G is an Eulerian graph, it is possible

to traverse all of the edges exactly once. The Eulerian graph

G must satisfy the following condition:

��vi� � 0 �mod 2� �1 � i � k�; �1�where ��vi� is the degree of vi.

In the case of our recovery problem, the start vertex v1

and the end vertex vk are distinct, thus, the path v1; v2; . . . ; vkis not closed. G is called a semi-Eulerian graph whichsatisfies the following conditions:

��v1� � 1 �2�

��vi� � 0 �mod 2� �1 < i < k� �3�

��vk� � 1: �4�An example1 is shown in Fig. 3 in which ��v1� � 1,

��v2� � 4, ��v3� � 4, and ��v4� � 1.

3.3 Double-Trace Problem

If a graph is semi-Eulerian it is possible to traverse all of the

edges exactly once by starting at v1 and ending at vk.However, we sometimes have situations in which one or

more lines are drawn twice, as shown in Fig. 4. We call the

line which was traversed twice a double-traced line (D-line).

It is, therefore, necessary to detect all of the D-lines in the

graph and then traverse each of the D-lines twice whentraversing the graph. Hence, in this paper, we assume that

the semi-Eulerian graph be extended to allow multiple D-

lines which can be traced twice but not more than twice.

3.4 Thinning Problems

To recover the drawing order from a given 2D image, we

begin by applying a thinning algorithm to extract skeletal

pixels so that, we can analyze the lineal structure of the image.

Each of the skeletal pixels is called a terminal pixel, a line pixel,


TABLE 2Assumptions to Recover the Drawing

Order from a Handwritten Script

(a) A script must be drawn in a single stroke. (b) A script must have astart point and an end point. (c) A script must have no junctions wheremore than two lines intersect.

Fig. 3. An example of the single-stroke script. This is a semi-Eulerian

graph and can be drawn in a single stroke.

1. A loop edge contributes the degree of the vertex by two.

or a branch pixel depending on whether it is adjacent to one,

two, or more than two other pixels, respectively.However, the thinning process results in various un-

desirable defects and the defects shown in Fig. 5 are to be

resolved from the line trace view point:

. Spurious lines (S-lines),

. Clusters of branch pixels.

An S-line does not exist in the original image but appears

in the skeleton image after thinning. It is usually very short

in length and always exists between two adjacent branch

pixels, making it difficult to trace the stroke because an

originally continuous line is divided into three segments.A cluster of branch pixels makes it difficult to select a

correct path at the branch pixel because multiple branch

pixels are directly adjacent to each other and a similar

selection must be made again and again.We have to solve the thinning problems discussed above

so that we can build an adjacency matrix to trace the graph

correctly.

4 RECOVERY OF DRAWING ORDER

In this section, we present the method to recover the drawing

order from a single-stroke handwritten script. After the input

image is scanned in and preprocessed, we apply the drawing

order recovery method which consists of two major phases:

1. Global analysis phase.2 Globally analyzes the graphconstructed from the skeletal image and labels eachof the edges by its type.

2. Trace phase. Traces the graph from the start vertexto the end vertex by referring to the labelinginformation obtained in the global analysis phase.

The method consists of the following steps; steps 1

through 5 perform preprocessing, step 6 performs the

global analysis, and step 7 performs the trace.

1. Image capture. A handwriting image is scanned byan image scanner. We use an image scanner tocapture an image at 200� 200 dpi which is a

resolution normally used for handwriting characterrecognition.

2. Image smoothing. Possible noise in the input imagemay cause recovery errors, and therefore, needs to beremoved by applying a filtering algorithm to the inputimage. Hence, we apply the filling operations to theimage to remove holes.

3. Image thinning. The skeletal pixels are extracted byapplying a thinning algorithm called Classical Thin-ning Algorithm [21], but our method does not assumethe perfect 8-connected curves[22].

4. Clustering branch pixels. A cluster of multipleneighboring branch pixels is grouped together andreduced to a single vertex.

5. Adjacency matrix creation. The adjacency matrix iscreated by scanning the skeletal pixels and detectingthe terminal and the branch vertices.

6. Global analysis. The graph is analyzed globally;that is, all the D-lines are detected and each of theedges is labeled.

7. Trace. The drawing order is recovered by tracing thegraph from the start vertex to the end vertex byreferring to the labeling information.

However, in some situations the drawing order cannot be

determined uniquely. For example, in Fig. 6a the drawing

starts at the end of the long line, in Fig. 6b it starts at the same

point but takes a different path, in Fig. 6c it starts at the loop

and there is no start point, thereby violating the second

assumption, and in Fig. 6d it starts at the loop and there is a

start point. Thus, the script shown in Fig. 6c will be

interpreted as same as the script shown in Fig. 6a.

4.1 Thinning Problem Resolution

To solve the thinning problems, we apply a clustering

algorithm to reduce neighboring multiple branch pixels to a

single cluster as shown in Fig. 7. The threshold �c, a

parameter determined by the average width of the stroke, is

used for this clustering and the average width of the stroke

is obtained by determining the distances from each skeletal

pixel to the nearest white pixels, dividing the sum of the

distances by the number of skeletal pixels, and multiplying


Fig. 4. An example of the Double-traced line (D-line). (a) A script with a

D-line. (b) A part of the script is drawn twice.

Fig. 5. Thinning problems. We have two Spurious-lines and one cluster

of branch pixels in this example.

2. The global analysis, a new idea introduced here, makes it possible tosuccessfully avoid the combinatorial explosion which is the most criticalproblem of the previous works.

the result by the predetermined constant. If the distance oftwo adjacent branch pixels is shorter than �c, they areconsidered to belong to the same cluster.

In the case of Fig. 7a, the branch pixels, b1, b2, b3, and b4, areconsidered to belong to the same cluster, v, which is shown inFig. 7b. This cluster has four lines li, 1 � i � 4, and is,therefore, of degree four. After the clustering, the S-linebecomes a part of the resultant cluster and is seen as aninternal line of the cluster. By clustering in this manner, eachof the clusters of branch pixels is merged into a single cluster.

4.2 Adjacency Matrix Creation

The following theorem provides the information about thevertices.

Theorem 1. There is no vertex whose degree is other than one,three, or four in the graph constructed from the skeletal imageof the single-stroke handwriting.

Proof. First, for necessity, let p be any one of the pixels inthe skeletal image obtained after the above steps 3 and 4,and let ��p� denote the number of the pixels connected tothe pixel p. To begin with, it follows directly from theassumptions mentioned in the Section 3.1 that:

1. if ��p� � 0, p is an isolated pixel which is assumednever to appear,

2. if ��p� � 1, p is either a start or an end vertex, vand, thus, ��v� � 1,

3. if ��p� � 2, p is not a vertex but a pixel in a lineconnecting two different vertices if the line is nota loop, or an identical vertex if the line is a loop,

4. if ��p� � 3, p is a vertex v of which one of the threeconnected edges must be traced twice so that itsdegree becomes four and, thus, ��v� � 3,

5. if ��p� � 4, p is a vertex where two different linesintersect and, thus, ��v� � 4.

For ��p� � 5,

6. if ��p� � 0 �mod 2�, p is a vertex where ��p�=2,that is three or more, different lines intersect,

7. if ��p� � 1 �mod 2�, p is a vertex v of which at leastone of the��p� edges must be traced more than onceso that ��v� becomes even, implying that three ormore different lines intersect at that vertex.

Both cases 6 and 7 contradict the assumptions stated inSection 3.1 and, thus, must not occur.

Second, for sufficiency, if there is no vertex whosedegree is other than one, three, or four in the graphconstructed, it is easy to prove that the assumptions hold.tu

Hence, after the image is captured, filtered, processed by

the thinning algorithm, and clustered, it is necessary and

sufficient to find all the pixels of degree one, three, and four

to create the graph G and the adjacency matrix A. This

matrix is used not only to record G, but also to record

information such as the vertex types, the edge types, the

distances to the connected vertices, the sequence numbers

to order the edges, and several flags which are to be set

during the global analysis and during the trace.

4.3 Path Selection

At an intersection, we have to select the path which is

considered to be natural when drawing the script. There are

four possible paths at an intersection as shown in Fig. 8a.

However, we can eliminate the case shown in Fig. 8d

because this path does not result in a single-stroke script.

When we compared the paths shown in Fig. 8b and Fig. 8c,

the path shown in Fig. 8a shows good continuity because

there is no directional change. Hence, it can be seen that by

selecting the middle edge at an intersection we can follow

the path which satisfies good continuity.

4.4 Basic Trace Algorithm

We now present a simple trace algorithm named Basic Trace

Algorithm (BTA) which traces a graph constructed from a

single-stroke handwriting image. Given a vertex and one of

the edges connected to the vertex, BTA traces the graph

until a vertex of odd degree is reached. At the vertex v of

even degree, BTA selects the middle edge and traces the

middle edge to the next vertex. BTA repeats the algorithm


Fig. 6. Different drawing orders. A solid arrow indicates the initial

direction and a dotted arrow indicates the second direction. The scripts

shown in (a) and (c) cannot be distinguished.

Fig. 7. Clustering branch vertices. (a) There are four neighboring branch

pixels, b1, b2, b3, and b4, and an S-line pixel. (b) The cluster v has four

lines, l1, l2, l3, and l4.

Fig. 8. The path selection at an intersection. An arrow shows the

possible path.

until a vertex of odd degree is found and finally returns the

path detected.For example, if BTA is applied to the start vertex v1

and its only edge shown in Fig. 9a, BTA traces the edge to

the next vertex. At the vertex v2 shown in Fig. 9b, BTA

selects the middle edge and traces the edge to the next

vertex. At the vertex v3 shown in Fig. 9c, BTA completes

the trace because a vertex of odd degree is reached, and

returns the path detected.The following theorem exists for the BTA.

Theorem 2. Let G be a semi-Eulerian graph. Let vx be a vertex in

G of any degree and e be one of the edges connected to vx.

Then, BTA finds a path from vx to vy such that

��vy� � 1 �mod 2�.Proof. BTA traces e from the vertex vx to find the next

vertex v.If ��v� � 1 �mod 2�, then the algorithm completes with

vy � v.If ��v� � 0 �mod 2�, then v is a vertex where multiple

lines intersect. That is, there are an even number of edgesconnected to v and one of them is e. Because it is anintersection, every line divides each of the rest of linesinto two segments. The total number of line segments, n,produced by the division is given by n � 2� �N ÿ 1�,where N is the number of lines to intersect, and thenumber of divided segments at each side of the line isgiven by n=2. Hence, the edge e0 to be selected at v is themth edge from e in either a clockwise or a counter-clockwise direction where

m � n=2� 1 � �2� �N ÿ 1��=2� 1 � �N ÿ 1� � 1 � N:After the edge e0 is selected, BTA traces e0 to find the

next vertex and continues the trace until it reaches avertex vy such that ��vy� � 1 �mod 2�. Since G is semi-Eulerian there exist at least two vertices of degree one. Ifnone of these two vertices are reached by BTA, itcontradicts the assumption that G is semi-Eulerian; thatis, there is no path from vx to the start or the end vertex.Hence, BTA does find a path from vx to vy such that��vy� � 1 �mod 2�. tu

4.5 Global AnalysisÐGraph Labeling

To detect and label the D-lines in a graph, we provide thecharacterization of the D-lines in the following.

Theorem 3. There exists a D-line either between a vertex of degreeone and a vertex of degree three or between vertices of degree three.

Proof. Let vx and vy be two different vertices other than the

start or end vertex in the graph G and suppose there is a

D-line between vx and vy. Clearly, ��vx� = 1, 3, or 4 and

��vy� = 1, 3, or 4 according to Theorem 1.First, assume ��vx� � 4. Then, one of the edges

connected to vx must be traced twice and the actualdegree of vx becomes five. This does not satisfy (3). Thismeans that there exists no D-line which is connected tothe vertex of degree four.

Second, assume ��vx� � 1 and ��vy� � 1. Then, theonly edge connected to vx and vy must be traced twice,and the actual degree of vx and vy becomes two. As aresult, the path starts at vx and arrives at vy. Then, thereturn path starts at vy and ends at vx. That is, the pathreturns back to the vertex where it started. It is clear thatthere exists no path connecting the start/end vertex andvx=vy. Hence, a D-line cannot exist between ��vx� � 1 and��vy� � 1.

Third, assume ��vx� � 1 and ��vy� � 3. Then, the onlyedge connected to vx must be traced twice, thus, theactual degree of vx becomes two. One of the edgesconnected to vy must be traced twice, thus, the actualdegree of vy becomes four. Hence, both vx and vy satisfy(3) and a D-line can exist between a vertex of degree oneand a vertex of degree three.

Last, assume ��vx� � 3 and ��vy� � 3. Then, one of theedges connected to vx must be traced twice and theactual degree of vx becomes four. The same applies to vy.Hence, both vx and vy satisfy (3) and a D-line can existbetween vertices of degree three.

Conversely, if there is a D-line between vx and vy,where ��vx� � 1 and ��vy� � 3 or ��vx� � 3 and ��vy� � 1,the path is traversed twice resulting in the adjustment ofdegrees such that ��vx� � 2 and ��vy� � 4 or ��vx� � 4and ��vy� � 2, respectively and all other vertices of evendegree on the path remain of even degree, thus satisfying(3). Similarly, if there is a D-line between vx and vy,where ��vx� � 3 and ��vy� � 3, the path is traversed twiceresulting in the adjustment of degrees such that ��vx� � 4and ��vy� � 4 and all other vertices of even degree on thepath remain of even degree, thus satisfying (3).

It follows, therefore, that if there exists a D-line it iseither between a vertex of degree one and a vertex ofdegree three or between vertices of degree three. tu

4.5.1 Classification of D-Lines

The D-lines are classified into the following three types:

. Looped D-line (LD-line) as shown in Table 3a andTable 3b.

. Spurious D-line (SD-line) as shown in Table 3c.

. Proper D-line (PD-line) as shown in Table 3d.

An LD-line can be seen as a degenerated loop and is

relatively straight. The LD-line occurs exclusively between a

vertex of degree one and a vertex of degree three as shown

in Table 3a and Table 3b.An SD-line can be seen as a long S-line as shown in

Table 3c. There are two possibilities with respect to the

drawing order of the SD-line as shown in Fig. 10. The script


Fig. 9. Basic Trace Algorithm. (a) The trace is started at v1. (b) The

middle edge is selected at v2 which is an intersection. (c) The trace is

completed at v3 which is a vertex of degree one.

shown in Fig. 10a can be interpreted as the script shown in

Fig. 10b or the script shown in Fig. 10c.A PD-line has two possibilities as shown in Table 3d. A

loop connecting vx and vy is either crossing or not crossing

the PD-line.We detect all the D-lines by following the steps:

1. detecting all the LD-lines first, then2. detecting all the SD-lines and the PD-lines next.

4.5.2 LD-Line Detection

The problem of detecting the LD-lines can be turned into the

problem of detecting the pair of a start and an end vertices.

That is, if the start and the end vertices are detected, the

remaining vertices of degree one have to be connected to LD-

lines and, therefore, those LD-lines can be detected by

applying BTA to the remaining vertices of degree one. One

possible approach is to choose a pair of vertices of degree one

and calculate the smoothness of the path starting at one end

and ending at the other end by applying the trace algorithm.

We repeat this calculation and select the pair which resulted

in having the smoothest path. However, the total number of

pairs to be evaluated, T , becomes T � tC2, where t is the

number of terminal vertices. For example, the total number of

paris to be evaluated for the script shown in Fig. 11a isT � tC2 � 6C2 � 15. It can be seen that this approach may not

be used unless t is small. Hence, we will select the edge

corresponding to the minimum angle, which is shown in

Fig. 11b, as an LD-line. When selecting the minimum angle,

we take into account the writing behavior of top-to-bottom

and left-to-right and select the upper, left-hand angle even if itis not minimum but is close to minimum. The threshold �l is

used for this decision. The limitation of this approach is that

an LD-line such as shown in Fig. 11c cannot be detected.After a D-line is detected, it is duplicated; all the vertices

on the path are adjusted in their degrees by modifying theadjacency matrix A. For example, if the LD-lines in thescript shown in Fig. 12a are detected, the edges areduplicated as shown in Fig. 12b. The adjacency matrixbefore and after the detection is shown in Table 4. Both theoriginal line and the duplicated line are labeled as D-linesso that they are distinguished from the ordinary lines. Bythis adjustment, a vertex of odd degree becomes of evendegree while a vertex of even degree remains of evendegree; a vertex of degree three becomes of degree fourwhile a vertex of degree four becomes of degree six. It canbe seen, for this example, that there exists no vertex ofdegree greater than six. Finally, we label both vx and vy withthe D-line type.

4.5.3 SD-Line and PD-Line Detection

After all the LD-lines are detected, all the remaining odd-degree vertices must be of degree three (except for thestart and the end vertices) because all the vertices ofdegree one are already changed to degree two after theLD-line detection.


TABLE 3The D-line Types

A D-line exists between vx and vy. (a) and (b) LD-Line. (c) SD-Line.(d) PD-Lines.

Fig. 10. Two possible interpretations of an SD-line. (a) An SD-line.

(b) Crossing. (c) Contact.

Fig. 11. The detection of LD-lines. (a) An LD-line with six end points.

(b) An edge having a minimum angle. (c) An LD-line which cannot be

detected.

Fig. 12. The duplication of D-lines. (a) A script with D-lines. (b) All

the D-lines are detected and duplicated.

To find an SD-line or a PD-line we arbitrarily choose avertex vx of degree three and apply BTA to each of the threeedges connected to vx to find one of the paths to the vertices ofodd degree. There are only three possible cases as shown inFig. 13 in which vertices vx, vy, and vz are assumed mutuallydifferent:

1. The SD-line. Two paths reach the identical vertex vxand one path reaches the vertex vy of degree three asshown in Fig. 13a.

2. The PD-line (no crossing). One path reaches thevertex vz of odd degree and two paths reach thevertex vy of degree three, as shown in Fig. 13b.

3. The PD-line (crossing). One path reaches the vertexvz of odd degree and two paths reach the vertex vy ofdegree three crossing each other as shown in Fig. 13c.

Because there are no other combinations we candetermine the type of the D-line based on this search. Then,we determine which of the three paths is a D-line:

. In the case of the SD-line, the only path between vxand vy is a D-line,

. in the case of the PD-line, the shorter path from vx tovy is a D-line since the D-line is supposed to berelatively straight. (If both paths from vx to vy aresame in length, either one may be selected.)

An example of the PD-line detection is shown in Fig. 14,in which there exist two PD-lines to be detected. Eventually,we have two possible cases depending on which of the four


TABLE 4Vertices and Their Degrees

�0�v� is the original degree and ��v� is the degree after the D-lines areduplicated.

Fig. 13. The SD-line and PD-line detection. vx is a vertex from which aD-line is to be detected, vy is an odd-degree vertex which corresponds tovx, and vz is a vertex of odd degree. (a) An SD-line. (b) A PD-line whoseloop does not cross the PD-line. (c) A PD-line whose loop crosses thePD-line.

Fig. 14. The PD-line detection example. (a) Detection from one end of a

PD-line. (b) Detection from another end of a PD-line.

Fig. 15. A PD-line and a special case of PD-line. (a) A PD-line with

�1 < �� and �2 < ��. (b) A special case of PD-line with �1 < �� and

�2 � ��.

vertices of degree three the vertex vx is supposed to. By

applying BTA, vz is found to be either 1) one of the verticesconnected to another PD-line, or 2) the start or the endvertex. It is easy to see that the two PD-lines are successfullydetected in either one of the cases.

Although there are two possible cases of interpretation of

an SD-line as shown in Fig. 10, we select the crossing pathas shown in Fig. 10b, which is more likely to occur than thecontacting path as shown in Fig. 10c.

After an SD-line or a PD-line is detected, we duplicate

the path, adjust the adjacency matrix, label the D-lines, and

label both vx and vy as type D-line as we did for the LD-

lines. Then, we label the loop line to distinguish it from the

ordinary lines. In the case of a PD-line, we label both vx and

vy to record whether or not the loop crosses the D-line.We repeat this SD-line and PD-line detection until all the

vertices of degree three are processed.


Fig. 16. The trace algorithm. The trace starts at the start vertex v1 and the next edge to select will be determined by the type of the current vertex v. Asolid line shows an ordinary line and a dotted line shows a D-line. A � shows a vertex which has not been visited, a � shows a vertex which hasalready been visited, and a shows a vertex which has been visited or has not been visited. A dotted arrow shows the path from the previous vertexand a solid arrow shows the path from the current vertex v.

4.5.4 Special Case of PD-Line

When compared with the normal PD-line shown in Fig. 15a,the PD-line shown in Fig. 15b needs special considerationbecause it is more natural to make a 180-degree turn than tomake a 90-degree turn while writing. The special PD-linecan be detected by calculating the angles between the twoordinary lines. We check each of the PD-lines to see if theangle at one of the end vertices is greater than or equal to apredetermined threshold �� and if the angle at the other endis smaller than ��. The PD-line which satisfies this conditionis marked ªspecial.º In the trace phase, this mark is tested toselect the appropriate path.

4.6 Trace Algorithm

We can now trace the graph from the start vertex until theend vertex is reached by executing the trace algorithmreferring to the edge type identified as shown in Fig. 16. Thestep to be followed at a D-line can be determined by thestate of the vertex, which is illustrated by �, �, and , andthe type of the edge.

5 EXPERIMENT

The recovery method presented above has been implemen-

ted in Java. To check the utility of the proposed method, we

have prepared a set of more than 100 test cases which are

included to cover possible combinations, which do not

always appear in normal handwritten characters. Some

examples are shown in Fig. 18. The input image can be

selected from a list of prescanned images. One of the

prescanned images is shown in Fig. 17a. The image is then

traced by the algorithm to recover the drawing order. The

recovered drawing order is shown in Fig. 17b which is

originally shown in animation with a moving pencil on a

display screen. The double-traced lines are originally shown

in red while the single-traced lines are in blue on the display

screen. Several recovered examples are shown in Fig. 18.From the above experiments, it was shown that the

method presented in this paper could successfully recover

the drawing order from static 2D images even if there were

multiple D-lines including LD-lines, SD-lines, and PD-lines.

6 DISCUSSIONS

When a script is drawn in a single stroke, the proposed

method can recover a drawing order even if there are

multiple D-lines. When recovering the drawing order, the

method uses a minimum set of heuristics; the clustering

threshold, the LD-line detection angle, the LD-line orienta-

tion, and the special PD-line detection angle. The major

characteristics of the method are as follows:

1. It is based on the graph traversal but does notevaluate all the possible combinations during therecovery.

2. It detects and handles multiple D-lines which can betraced twice but not more than twice.

3. The detection of the S-lines is included in theclustering process; the distinction of an S-line from

a D-line is based on the clustering threshold and anS-line is not treated as a D-line.

4. The detection of the LD-lines is based on the heuristics;the smoothest curvature is selected by examining theangles between two intersecting lines and the orienta-tion of those lines. There is no need to evaluate all thepaths between a combination of two terminal vertices.

5. Once all the LD-lines are detected, other types of D-lines can be detected by applying BTA.

6. The detection of the special case of the PD-lines isbased on the heuristics; a writing behavior of the180-degree direction change is honored.

7. The trace algorithm refers to the labeling informationand is simple because of the duplication of the D-lines.


Fig. 17. An experimental result. (a) An input image is read from aprescanned image file. (b) The drawing order is recovered; a sold circleindicates the starting point, an arrow indicates direction of the drawing, aD-line is highlighted in red, and a moving pencil indicates the drawingorder in animation on the display screen. (c) A part of the recovery resultis shown magnified.

There are cases that cannot correctly be recovered by the

method because of the following:

1. It is assumed that a middle edge at an intersectionsatisfies the good continuity.

2. The intersecting path is always selected for an SD-line.

The analysis of the graph alone is insufficient to

determine the correct SD-line direction. One possibility is

to identify several candidate paths and then calculate the

global smoothness for each paths for final selection.

Another possibility is to use a complementary approach

which applies more heuristics such as the clues at the

intersection [5], [6], [7] or more knowledge about the human

writing behavior.

6.1 Extension to Multistroke Scripts

To extend the method for multistroke scripts, we have to

consider several problems including the following:

1. A D-line may be caused by the contact or thecrossing of two different strokes.

2. Each stroke must be traced separately from otherstrokes.

6.2 Application to Offline Handwriting Recognition

For the application to the offline handwriting recognition of

single-stroke scripts, we have made a preliminary study in

which we recover the drawing order, obtain the vector data,

convert the coordinate data to the chain code data, compute

the distances from the template data, sort the distance data,

and determine the candidates. When computing the


Fig. 18. Several recovery results. (a) A simple script. (b) A script which cannot be decomposed into subproblems. (c) A script with an LD-line. (d) A

script with three LD-lines. (e) A script with a PD-line. (f) A complex script with three LD-lines. (g) A complex script with two SD-lines and a PD-line.

(h) A script with two PD-lines. (i) A script with a special PD-line.

difference between the recovered vector data and eachtemplate data, we take into account the situation that therecovery is made in a reverse direction from the templatedata.

For the offline handwriting recognition for multistrokescripts, we have to consider several additional problemsincluding the following:

1. The recovered drawing order may be different fromthe drawing order of corresponding template data.

2. The number of the strokes may be different. Forexample, the script may be drawn in two strokeswhile the corresponding template data is drawn inthree strokes.

7 CONCLUSION

In this paper, we have presented a new method whichrecovers a drawing order of a single-stroke handwrittenscript from a static 2D image. The problem is an extensionof the unicursal figure problem and allows a script toinclude double-traced lines.

The method is based on graph traversal and consists oftwo phases. In the first phase we globally analyze the graphconstructed from the skeletal image and label the graph bydetermining the type of each edge. In the second phase, wetrace the graph from the start vertex to the end vertex byreferring to the labeling information. There is no need tocalculate the local or global smoothness of the curvatureexcept for the detection of the special double-traced lines.

Unlike previous works this method does not evaluate allthe possible combinations, for example, by solving thetraveling salesman problem and, therefore, does not cause acombinatorial explosion even if the script is very complex.Hence, this method can be used to solve complex scriptrecovery problems very practically.

For further study, we have been working on the problemwhich extends the presented method to recover thedrawing order from multiple-stroke scripts [23].

REFERENCES

[1] J.C. Pan and S. Lee, ªOffline Tracing and Representation ofSignatures,º Proc. Computer Vision and Pattern Recognition, pp. 679-680, 1991.

[2] S. Lee and J.C. Pan, ªOffline Tracing and Representation ofSignatures,º IEEE Trans. Systems, Man, and Cybernetics, vol. 22,no. 4, pp. 755-771, 1992.

[3] V. Govindaraju and S.N. Srihari, ªSeparating Handwritten Textfrom Interfering Strokes,º From Pixels to Features III: Frontiers inHandwriting Recognition, S. Impedovo and J.C. Simon, eds., pp. 17-28, North-Holland, 1992.

[4] V. Govindaraju and R.K. Krishnamurthy, ªHolistic HandwrittenWord Recognition Using Temporal Features Derived from Off-Line Images,º Pattern Recognition Letters, vol. 17, pp. 537-540, 1996.

[5] D.S. Doermann and A. Rosenfeld, ªRecovery of TemporalInformation from Static Images of Handwriting,º Proc. ComputerSoc. Conf. Computer Vision and Pattern Recognition, pp. 162-168, 1992.

[6] D.S. Doermann and A. Rosenfeld, ªTemporal Clues in Hand-writing,º Proc. 11th Int'l Conf. Pattern Recognition, vol. II, pp. 317-320, 1992.

[7] D.S. Doermann and A. Rosenfeld, ªRecovery of TemporalInformation from Static Images of Handwriting,º Int'l J. ComputerVision, vol. 15, pp. 143-164, 1995.

[8] G. Boccignone, A. Chianese, L.P. Cordella, and A. Marcelli,ªRecovering Dynamic Information from Static Handwriting,ºPattern Recognition, vol. 26, no. 3, pp. 409-418, 1993.

[9] I.S.I. Abuhaiba, M.J.J. Holt, and S. Datta, ªProcessing of Off-LineHandwritten Text: Polygonal Approximation and Enforcement ofTemporal Information,º Computer Vision, Graphics, and ImageProcessing, vol. 56, no. 4, pp. 324-335, July 1994.

[10] I.S.I. Abuhaiba, M.J.J. Holt, and S. Datta, ªProcessing of BinaryImages of Handwritten Text Documents,º Pattern Recognition,vol. 29, no. 7, pp. 1,161-1,177, 1996.

[11] I.S.I. Abuhaiba, M.J.J. Holt, and S. Datta, ªRecognition of Off-LineCursive Handwriting,º Computer Vision and Image Understanding,vol. 71, no. 1, pp. 19-38, July 1998.

[12] C.M. Privitera and R. Plamondon, ªA System for Scanning andSegmenting Cursively Handwritten Words into Basic Strokes,ºProc. Third Int'l Conf. Document Analysis and Recognition, pp. 1,047-1,050, Jan. 1995.

[13] R. Plamondon and C.M. Privitera, ªThe Segmentation of CursiveHandwriting: An Approach Based on Off-Line Recovery of theMotor-Temporal Information,º IEEE Trans. Image Processing, vol. 8,no. 1, pp. 80-91, 1991.

[14] Y. Isomichi, ªA New Technique for the AntiquantizationProblem,º Proc. First Symp. Information Theory and Its Applications,pp. 27-32, Nov. 1978 (in Japanese).

[15] Y. Isomichi, ªInverse-Quantization Method for Digital Signals andImages,º IEICE Trans., vol. J64-A, no. 4, pp. 285-292, 1981. (inJapanese).

[16] T. Huang and M. Yasuhara, ªRecovery of Information on theDrawing Order of Single-Stroke Cursive Handwritten Charactersfrom Their 2D Images,º IPSJ Trans., vol. 36, no. 9, pp. 2,132-2,143,Sept. 1995.

[17] T. Huang and M. Yasuhara, ªA Total Stroke SLALOM Method forSearching for the Optimal Drawing Order of Off-Line Hand-writing,º Proc. IEEE Systems, Man and Cybernetics Soc., pp. 2,789-2,794, Oct. 1995.

[18] S. JaÈger, ªRecovering Writing Traces in Off-Line HandwritingRecognition: Using a Global Optimization Technique,º Proc. 13thInt'l Conf. Pattern Recognition, pp. 150-154, 1996.

[19] H.Bunke,R.Ammann,G.Kaufmann,T.M.Ha,M.Schenkel,R.Seiler,and F. Eggimann, ªRecovery of Temporal Information of CursivelyHandwritten Words for On-Line Recognition,º Proc. Fourth Int'lConf. Document Analysis and Recognition, pp. 931-935, 1997.

[20] P.M. Lallican and C. Viard-Gaudin, ªA Kalman Approach forStroke Order Recovering from Off-Line Handwriting,º Proc. FourthInt'l Conf. Document Analysis and Recognition, pp. 519-522, 1997.

[21] T. Pavlidis, Algorithms for Graphics and Image Processing, pp. 199-201 Rockville, Md.: Computer Science Press, 1982.

[22] H. Tamura, ªA Comparison of Line Thinning Algorithms fromDigital Geometry Viewpoint,º Proc. Fourth Int'l Conf. PatternRecognition, pp. 715-719, 1978.

[23] Y. Kato and M. Yasuhara, ªRecovery of Drawing Order fromScanned Images of Multi-Stroke Handwriting,º Proc. Fifth Int'lConf. Document Analysis and Recognition, pp. 261-264, 1999.

Yoshiharu Kato received the BSc degree fromKeio University in 1974 and the MS degree incomputer science from Case Western ReserveUniversity in 1985. He is currently a doctoralstudent in the Department of Information Man-agement Science at the Graduate School ofInformation Systems, the University of Electro-Communications. His research interests includethe pattern recognition and the image proces-sing. He is a member of IEEE Computer

Society, ACM, and IPSJ.

Makoto Yasuhara received the BSc degree fromthe University of Electro-Communications in1963 and the MD and PhD degrees in 1974and 1986, respectively, both from the Universityof Tokyo. He joined the University of Electro-Communications in 1966 and was a professor inthe Department of Computer Science and In-formation Mathematics from 1987 to 1993.Currently, he is a professor in the Departmentof Information Management Science, Graduate

School of Information Systems. His research interests include thestochastic system theory, the control theory, and their applications. He isa member of IEEE, IEICE, IPSJ, and SICE.
