An evaluation of motor models of handwriting

1060 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, VOL. 19, NO. 5 , SEPTEMBER/OCTOBER 1989

An Evaluation of Motor Models of Handwriting

R ~ J E A N PLAMONDON, SENIOR MEMBER, IEEE, AND FRANS J. MAARSE

Abstract -A general method is presented for describing and analyzing biomechanical handwriting models. Using Laplace’s transform theory, a model can be represented in what we call the neural firing-rate domain. Consistent terminology is also proposed to facilitate model evaluation and comparison. An overview of previously published models suggests that they could be described using this method, with second- and third-order linear model representation. Fourteen simplified theoretical models are simulated in an experiment designed to study the parameter domain in which handwriting is controlled by the nervous system to gain insight into which type of model provides the best reconstruction of natural handwriting. Results show that velocity-controlled models produce the best outputs. No significant difference exists between second- and third-order systems. In handwriting, fine motor behavior is, in the first instance, velocity-controlled. These findings agree with other recent automatic signature verification results and are of interest for a number of applications, from pattern recognition to handwriting education.

I. INTRODUCTION

HE PROCESSES involved in handwriting are rather T complex and have been analyzed over the past decades from several perspectives by various groups of researchers: physicists, engineers, computer scientists, experimental psychologists, neurologists, cognitive scientists, and graphoanalysts. Because it is one of the basic human- specific slulls, handwriting has been analyzed to study motor behavior, control and learning, to detect neurologi- cal disease, to develop psychodiagnostic tests for remedial teaching and even to design anthropomorphic robots [1]-[4]. Being one of the natural channels used by humans to communicate, handwriting has also been analyzed to develop new man-computer interfaces, character recognizers, signature verifiers, text authenticators, and handwriting generators [5]-[8].

These studies and applications may, however, be so specialized that researchers in one area are unaware of developments in related areas. In fact, most of these pro-

Manuscript received November 3, 1987; revised February 12, 1989. This work was supported in part by CRSNG of Canada under Grant OGP-0000915, in part by FCAR, Quebec, Canada, under Grand AS- 2240F, and in part by a grant from The Netherlands Organization for the Advancement of Pure Research.

R. Plamondon is with the Laboratoire Scribens, DCpartement de Genie Electrique, Ecole Polytechnique de MontrCal, C.P. 6079, Succursale “ A , MontrCal, PQ, Canada H3C 3A7.

F. J. Maarse is with the Nijmegen Institute for Cognition Research and Information Technology, P. 0. Box 9104, 6500 HE Nijmegen, The Netherlands.

IEEE Log Number 8927729.

jects rely directly or indirectly upon a basic knowledge of the handwriting processes themselves. From the time work began in th s field, handwriting modelization has been the foundation upon which some of the basic research projects and development applications have been built. Nowadays, several models are used in a large spectrum of projects, from fundamental studies aimed at understanding the biomechanical or neuropsychological systems to immediate applications at the feature extraction level in the design of pattern recognizers. Some models are more oriented toward handwriting analysis, others toward handwriting generation.

Since these models were not all developed in the same context nor with a common goal, it is not always easy to analyze their differing perspectives. Comparing one model with another often means coping with problems of basic vocabulary. Neither is it always evident how to evaluate and discuss the impact of the simplifying assumptions associated with each model.

The purpose of this paper is first to provide a general context within which most of the biomechanical models proposed to date can be analyzed and reviewed. This study is limited to physical models, that is, mathematical models that can be used directly to analyze or generate a piece of handwriting. Conceptual models, more oriented toward the understanding of high-level motor programming, are not included here. Nevertheless the expectation is that the terminology used here will be consistent with the terminology used in this important category of research work on modelization.

Secondly, its purpose is to gain further insight into what variable is controlled by nervous system during handwriting. Stein [9] has suggested the following control variables: force, velocity, length (spatial target), stiffness, viscosity, more than one of the previously mentioned, or none of the previously mentioned. By using simplified models, three control variables are tested in this study by means of computer simulation [lo]: acceleration, velocity, and spatial target (length).

Finally, we are hopeful that t h s overview will shed significant light on the basic problems related to developing the computer applications of handwriting. For example cursive script recognition [ll] and signature verification [5] systems are still limited owing to inadequate answers to fundamental pattern recognition questions.

0018-9472/89/0900-lO60$01.00 01989 IEEE

PLAMONDON AND MAARSE: AN EVALUATION OF MOTOR MODELS OF HANDWRITING 1061

Top-down approaches

Motor program: grapheme level allograph level

parameter level

Activation mechanisms

Motor program extraction

J

! I

Nerves Information

transmission

Muscles Movement activation

Pen/paper Trajectory

Memorization

I I

Bottom-up approaches

NeNe-muscle interface > 1 5' order system

Hand-paper interface (2"! order system)

Fig. 1. Study of handwriting as motor behavior

Using a proper representation space for signals and having a good model to describe their generation can lead to more efficient segmentation protocol, feature selection, signal coding, and compression. Similarly, better definition and representation of strokes can be a great help in design- ing software for (remedial) teaching of handwriting and also in developing a measure of fluency and ballisticity in handwriting, for use in psychophysiological tests and ex- perimen t s.

Starting with a commonly accepted view of the biologi- cal processes involved in handwriting, a standardized classification of the previously published models is proposed according to the Laplace transform theory of system description. In this way, the models may be classified in terms of the order of the system used to generate the movement. Consistent terminology is also proposed to facilitate model evaluation and comparison. The practical interest of this standard approach and terminology is demonstrated in the last part of this paper, where several models are simulated [lo] to gain further insight in which variable is controlled by the nervous system.

11. HANDWRITING PROCESSES

A . General Overview

Several conceptual, physical, and empirical models have been developed for studying and understanding handwriting. Although the models may differ greatly in their description of the phenomena, depending on the context and purposes for which they have been developed, most can be viewed as emanating from a commonly accepted view of the processes involved.

The center column of Fig. 1 is a block diagram of the organizing functions involved in handwriting. This schema also reflects some basic hypotheses generally assumed when one studies these biomechanical processes. Like any highly skilled motor process, fast handwriting is considered a ballistic phenomenon, that is, a motion controlled without instantaneous position feedback, the product of a learned motor program [12]-[14]. At the beginning of a writing segment, the whole tt.ajectory of that movement is defined. No extra control is applied during execution. According to this model, some central nervous system mechanisms withn the brain fire, with a predetermined intensity and duration, the nerve network which activates the proper muscles in a predetermined order. The motion of the pen on the paper, resulting from muscle contraction/relaxation, leaves a partial trace of the pen-tip trajectory.

Although there is no clear-cut boundary between fast and slow handwriting movements (a slower writing process is probably a matter of position and visual feedback [15]), t h s representation has resulted in a stepwise analysis, involving several stages, from movement planning to muscle activation. Several models have been proposed to study these mechanisms and, depending on the emphasis placed on the behavior of the brain or of the hand, two comple- mentary approaches have been followed to study handwriting: top-down and bottom-up. This complementarity is illustrated in the left- and right-hand parts of Fig. 1.

The top-down approach has been developed mainly by psychologists and researchers interested in the study and application of the motor program itself: the fundamental unit of movement coding, code sequency and retrieval, movement control and learning, and task complexity [lo],

1062 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, VOL. 19, NO. 5 , SEPTEMBER/OCTOBER 1989

[16]. In this context, research is concentrated on high-level representation and utilization of information. The biophysical phenomena are grouped within a black box dedicated to movement generation, which in turn activates the proper nerve and muscle pairs to generate the movement corre- sponding to the grapheme/allograph/parameter representation of the programmed information.

Indeed, according to such a conceptual approach, handwriting would be produced in at least three steps reflected by the additive effects on reaction time experiments [17], [18]: first the abstract motor program is retrieved from long-term motor memory, then parameters like actual size, accuracy, and speed are fed into th s program to make it more concrete, and finally, this information (program and parameters) is translated for the recruitment and activation of the proper muscles.

The bottom-up approach has been used by those, mainly physicists and engmeers, interested in the analysis and synthesis of the biomechanical processes. Their goal was to produce handwriting forms and not to simulate the psychomotor process. Several models of the muscle-activated hand motion have been proposed [13], [14], [19]-[27]. In their simplest form, these models represent the study of a point mass under the action of muscular forces. Some refinements have incorporated, at least theoretically, the viscous force [13], [14], [19], [22], the external friction force [14], the elastic force [20], [21], [24], and also, to some extent, nerve-muscle interface phenomena [22], [26].

A study of these models reveals that research efforts are concentrated on the mathematical description of the nerve-muscle-pen-paper system: choice of differential equations, parameter extraction, movement analysis, and synthesis. This system is assumed to be controlled by a program that can be described as a sequence of input stimuli.

B. Basic Equations

Although both top-down and bottom-up models have been found helpful in many studies on handwriting, only the bottom-up approaches are considered here. These rely on mathematical description of the different subsystems depicted in the right-hand part of Fig. 1.

The hand-pen-paper system is represented by a point mass M whose motion along one linear direction r can be described by

M d 2 r ( t ) dt

+ A T d r ( t ) + K r ( t ) + f , N ( t ) = F , . ( t ) (1)

equivalent mass of the hand-pen system, intrinsic viscosity coefficient of the hand, stiffness coefficient of the hand, extrinsic friction coefficient between the pen-tip and the writing surface; component of the writing pressure normal to the writing surface,

q ( t ) muscular force applied to the equivalent point mass.

At least two of these equations are needed to produce two-dimensional handwriting movements.

In many studies, simple second-order equations are used for describing and simulating handwriting. A nerve-muscle interface of zero order is thus assumed in this case. In more fundamental studies, the nerve-muscle interface has been thoroughly studied and several models have been proposed. In its simplest representation, this interface can be described by a first-order system 1281, [29]:

where

F,

a , b g,( t ) activation level, u ( t )

This system acts as a pure force generator if the constant

maximal isometric force in the operating-length region of the muscle, constants specific to the muscle under study,

contraction velocity of the muscle.

b is substantially greater than the contraction velocity:

F , ( t ) = F,g,(t), ifb>> u ( t ) . (3)

Since it is generally assumed that the activation level g , ( t ) can be described at least as a first-order response to the neural firing rate [29], the following is obtained for the nerve-muscle interface:

(4)

where u r ( t ) is the neural firing rate and C is the fitting constant.

The brain-nerve interaction has been studied in the context of motor program representation. This top-down approach has not resulted in any mathematical description of the mechanisms involved and it is not clear thus far if such a representation can be developed or would be significant.

Moreover this basic knowledge, although expressed mathematically, should not mask the various hypotheses from which it has been developed. It is assumed, for example, that the coefficients of these differential equations are constants, at least over one handwriting stroke, and that the complex set of muscles involved in handwriting production along a principal direction behaves like a single nerve-muscle interface. In addition all these subsystems are assumed to be linear and stationary.

111. CLASSIFICATION AND STANDARDIZATION

As will be seen in Section V, several other simplifying assumptions have been proposed by other authors, to make this theoretical description more practical for specific applications. Several models have been documented to date, and, to understand their relevance better, a stan-


Motor program

1 i domain

I I

L ______________________ J (b)

Fig. 2. (a) General form of handwriting models. (b) Third-order approximation.

dard representation might be helpful. Fig. 2(a) summarizes the basic principles upon which the standardization proposed in this paper is based.

1) A generation channel can be described by an nth- order linear system using Laplace transform representation, the overall system being described by a cascade of m simpler subsystems to be studied separately or as a whole, according to the equivalence of the cascaded linear system.

2) Whatever the domain of analysis for which it has been used, the model is transposed to a mathematical representation where the system output is in the space versus time domain and the input motor program U ( S ) is represented by a sequence of abstract input step functions (V,( S ) ) whose amplitude reflects the neural firing rate:

where A , is the amplitude of the step function, K , = + 1 or - 1, to take into account positive and negative stimuli, and U , ( S ) = K , A , e - T S / S is the unit step function occurring at T,.

The choice of the input function and the output domain of analysis has been guided by established practice and convenience. Indeed, early works on handwriting modeling use the step function as the system input, and the ultimate test for any model is whether the proper pen-tip trajectory can be generated in the space-time domain. In this context, the order of a model is defined as the order of the differential equation used to describe it, when it is fired with a standard abstract input U ( S ) to produce the proper displacement r ( S ) . The system order must be viewed as a relative definition and not as an absolute one, since there

is no way of knowing what the real input function is that represents a motor program within the brain. Substituting, (4) and (3) into (1) and using Laplace transform representation (with no initial conditions), the minimal order of the equation theoretically representing the behavior of the handwriting generation system is obtained:

This (6) makes it possible to represent any motor program in what we will henceforth call the abstract neural firing- rate domain [ U( s)]. It also shows that at least a third-order linear system would be necessary to code the handwriting movement in that abstract domain, if the other simplifying assumptions made in this study are justified. Fig. 2(b) shows, for one generation channel, a block diagram of this minimum theoretical model described by the following equation:

1 . (7) -- - r ( 4

u(s) [ s + 4 a s * + BS + y + S / r ( s ) ]

For simplicity, in this equation the gain of the system is fixed at unity since there is no way of measuring it. A first-order nerve-muscle system is also assumed. In a more simplified approach, this interface can be supposed to be of zero order, so that the equation would therefore be reduced to a second-order system.

IV. TERMINOLOGY

The standard representation proposed in the previous section for biomechanical handwriting models also leads to some practical definitions [30].

1064

Axis I

Axis 2 -

A X I S 3 (pressure)

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, VOL. 19. NO. 5. SEPTEMBER/OCTOBER 1989

MOTOR PROGRAM REPRESENTATION

-==I I

* Word b

Fig. 3. Terminology resulting from motor program representation as function of time.

1 ) Stimulus: A stimulus is any change of state in an input function U( S). This change may be characterized by its time of occurrence and its signed variation in amplitude. The overall gain of the system is included in t h s amplitude.

2) Motor Program: A motor program is a set of stimulus sequences that can generate handwriting when fed into a proper biomechanical model of the process under study. As shown in Fig. 3, a program is at least composed of three sequences, two for producing the plane movement of the pen-tip and one for the generating of pressure or pen-up/pen-down signals.

3) Stroke: A stroke is a sequence of stimuli which results in the production of a fundamental unit of handwriting movement.

4) Component: A component is a sequence of strokes produced during a continuous pen-down signal. It may be, for instance, a letter or an allograph.

5) Word: A word is a sequence of components resulting in a handwritten discontinuous curve normally having a few specific semantic values. At this level, handwriting is analyzed in a linguistic context.

In Fig. 3, a schematic representation of a motor program is given. The timing is represented, in addition to the amplitude of stimuli and for the three movement generation systems (axes 1-3). The first two axes refer to stimuli that might be needed to generate plane movement (coded in an X - Y Cartesian reference system, or in any other which might be more powerful [26]). Stimuli for movements in the thrd dimension, those resulting in pen- up/pen-down movements and pen pressure, are contained in axis 3. The definitions of “component” and “word” derive from this third dimension. These three motor pro-

grams, when fed into three movement generation subsystems, result in a typical handwriting output.

V. REVIEW OF HANDWRITING MODELS

With t h s representation and terminology, all the bottom-up models proposed to date may be classified according to the order of the system used to describe the biomechanical generators. It must be remembered however that these models were not necessarily developed for the same purpose and that the proposed classification in a biomechanical context is introduced only for purposes of comparison. In spite of the differences, most of them may be interpreted from a physiological point of view using this standard method. Some models dealing with a representation and segmentation of handwritten images have also been proposed, mainly for off-line character recognition [31]. Since these studies were not concerned with temporal simulation, they are not included in this review.

A. Second-Order Systems

Table I summarizes the properties of the second-order models that have been published to date, using the standard terminology defined in Fig. 2. The first model in t h s table was proposed by Denier van der Gon et al. [32] and is the product of experimental studies in handwriting simulation based on four principles: 1) fast handwriting is a ballistic phenomenon, 2) movements are caused by two independent and perpendicular groups of muscles, 3) these muscles apply forces to hand and pencil which are considered as a mass with some internal friction, and 4) once applied, the force increases to a certain fixed value and then remains more or less constant, the duration of its


TABLE I LIST OF SECOND-ORDER MODELS

REMARKS AUTHORS STANDARDIZED MODEL

1 no external friction I negligible elasticity

I I Denier Van der Gon I I 1 I Thuring and Strackee I ?*I ~ I-> X(S) or y(s) 1 two identical axes (1962 1 1 ux(S) 1 as2 + BS ] 1 subsystem

)or uY(s) I orthogonal model I I

I Eden I (1962) I

I

Eden I (1964 1 I

I I

(1981) I I I I I i

Mermelstein and I

Hollerbach

1 external and internal friction I neglected I predominance of elasticity I phenomena I orthogonal model I horizontal movement divided into I oscillatory and trend components I I I I I I I I

Dooijes ( 1983

I I

I external friction I internal friction I also neClected in some I reconstruction experiments I

I I I I I I I I I I

I non-orthogonal model (X' , Y' 1

I horizontal movement - I 1 I I

-%I ~ h uXl(S) 1 as2 + BS I I divided into oscillatory I > X'(S) 1 and trend components - I

-&I __ + I UxZ(S) I s I I

I

I 1 I 4- I I

'

Plarnondon ( 1987-88)

I I I I I I I I I I

1 external friction 1 neglected I invariable with rotation n

1 1 I I

I s I 1 - r(S) 1 velocity generators

1 independent of any fixed I reference axis I I I

1066 IEtE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, VOL. 19, NO, 5 , SEPTEMBER/OCTOBER 1989

application being related the magnitude of the movement. In this model, strokes of different lengths were produced by different force timings, the amplitude of the forces being constant for a specific writing size. Using our standard approach, t h s model becomes a zero-order system for the nerve-muscle interface and is thus a second-order system where external friction and elasticity effects considered negligible.

Three authors have proposed undamped versions of the second-order system. Their general view is based on a harmonic oscillator description of the muscle action [20], [21], [24] and uses velocity equations to describe momentum impulses or strokes. Eden [21] proposed the first of these models. Sinusoids were used as the input function to an integrator. With the appropriate difference in oscillation frequency of the orthogonal X and Y axes, plus a proper phase shft, handwriting could be generated, provided also that a linear trend was added to the horizontal displacement. Mermelstein and Eden [20] modified th s representation to incorporate a different velocity amplitude for the rise and fall of input excitation. They found that in this case handwriting could be better simulated with the help of a larger set of fitting parameters: X and Y velocity amplitude, frequency and phase shift between X and Y velocities, and timing of velocity changes. Hollerbach [24], using a similar approach, carried out an extensive study of the practical interest of the oscillatory model in controlling the shape, height, and slant of handwriting. He analyzed the basis of such a representation in the context of a muscle-spring model, as well as studying the effects of a nonorthogonal writing axis, a variable spring constant, internal friction, etc. In the context of our general representation, all these models can be schema- tized by a second-order undamped system, with an integrator added in the x branch to superimpose a linear trend movement.

Two additions were proposed for the Denier van der Gon model by Dooijes [23], [29]. The first is the assumption that the principal directions of motion, i.e., those directions emerging if either of the antagonistic muscle pairs are independently excited, form an oblique reference system instead of an orthogonal one. The orientation of these two principal axes (X’-Y’) may be determined with the help of two procedures based either on the Lissajous transformation or on Hilbert transform pairs. The second is the assumption that the movement along the horizontal direction is the results of two independent mechanisms: a uniform trend upon which a second-order generator is superimposed. The time-based component was extracted by linear regression analysis. In his analysis Dooijes also neglected internal friction terms and was dealing with a purely ballistic model.

Plamondon [26], [27] recently proposed model based on intrinsic representation of handwritten curves. Using differential geometry, the handwriting process is biomechani- cally represented, without reference to any fixed-axis system. In its simplest form, the generation of handwriting is thus reduced to the production of two types of displace-

ment: a linear displacement ( a ) of the pen-tip along the curvilinear abscissa and an angular displacement that results in the proper change of direction of the pen-tip. The whole system behaves like a speed generator and the brain has to control the magnitude and the orientation of the pen-tip velocity. The trajectory is integrated via the action of pen-paper contact.

B. Third-Order System

Using the same approach, Table I1 depicts the third- order models. Apart from a single-axis model developed by Plamondon and Lamarche [22], these models use two sets of similar equations for both X and Y displacement. Therefore, one axis is described here, identified by r ( S ) .

The first two of these are direct modifications of the model proposed by Denier van der Gon [13], and were designed to improve its performance in simulating handwriting. MacDonald [19], using a similar set of equations, has suggested the use of trapezoidal acceleration pulses to feed a second-order system, the slopes of the pulse corre- sponding to the rate of change of muscular force per unit of time. Yasuhara [14] has demonstrated that exponential transitions for the stimulus pulse are even more powerful. This model to a certain extent incorporates external friction components.

Looking at the general representation proposed in this paper, these improvements can be seen as adding a first- order system in front of the second-order model previously proposed by Denier van der Gon [13]. Indeed starting from abstract rectangular step function in the nerve firing-rate domain, a first-order system produces an exponential muscle force function. A trapezoTda1 one is obtained if the time constant of t h s stage is assumed to be null. These changes, that is, incorporation of the nerve- muscle interface in the biomechanical interpretation of the processes, clearly constitute an ingenious way of working with higher order systems.

In a very different context, Morass0 et al. [25] have developed a model that incorporates some dynamic information to allow a complete generation of a handwritten trajectory from basic segments. Basic strokes (described by curved segments of given length, tilt angle, and angular change) are used to reconstruct human handwriting performance with the constraint that each stroke was generated with a symmetric bell-shaped velocity profile, centered at a given instant of time. Assuming a proper mechanism for transforming spatial sequences into angular/muscular sequences, a motor program coded with these segments can generate handwriting, provided there is a proper overlap between two successive segments.

T h s space-oriented model may be considered to be halfway between the muscle-oriented model and the conceptual top-down model. Their representation in the neural firing-rate domain is not straightforward. A third-order transfer function, which can be used to generate a bell- shaped velocity profile from a step input, is shown in Table 11. The output of this first stage, when integrated,

PLAMONDON AND MAARSE: AN EVALUATION OF MOTOR MODELS OF HANDWNTING 1067

TABLE I1 LIST OF THIRD-ORDER MODELS

I

AUTHORS I STANDARDIZED MIDEL I

RDIARKS I

MacDonald (1966)

1 no external friction

I trapezofdal acceleration patterns

I -' I 1-4 - H-

I 1 1 1 1 I I negligible elasticity e r(S) I

~~ ~

I I

1 external friction I -- Yasuhara I I 1 1 1 1 I 1 partly integrated in internal (1975) 1-4 - H- +-> r(S) 1 friction term

I ur(S) I S + B I I as2 + BS I I negligible elasticity 'U I exponential acceleration patterns I

I

I

I orthogonal model I I -- I segment defined with bell-shaped

I velocity profile and specific Morass0 and I I U2S2 + 4 - 7 5 I I 1 I Mussa Ivaldi I -4 ~ +> r(S) I length, tilt angle and angular (1982) 1 Ur(S) I S(S2 + 4 - 7 5 I 1 s I I change

I -- I I -- I

I Plamondon I Lamarche I ---->-I--- H -

I

1 external friction I neglected 1 simplified to a second-order

I 1 1 1 1 I +> r(S)

( 1986 I ur(S) I as2 + BS I I S 1 I system for test experiments U- I single axis model

Maarse (1987

I

I purely ballistic model -- 1 1 1 1 1 1 I

+> r(S) 1 orthogonal model ->_I - H - Ur(S) I s 1 I as2 1 I 'U I triangular acceleration patterns

I I several other input 1 patterns also studied I

I I

results in segment generation. Proper geometric parameters could be controlled by initial conditions, timing, and amplitude parameters.

Yet another environment has been used by Plamondon and Lamarche [22] to develop a model. Applying the transfer function of the dc motor used in the Vredenbregt and Koster's handwriting simulator [33], they described the handwriting process with a speed generator system fed with a rectangular pulse voltage that directly represents the neural firing rate. Unidirectional movement was analyzed in various simulation experiments with a simplified second-order version of this model. It was demonstrated that

firing-rate domain. This scheme is used in the following section to perform some control variable analysis.

Higher order models have also been proposed recently [34] that can be fitted to the same general scheme using fifth- and seventh-order systems. Their interpretation, in terms of the peripheral psychomotor model of Fig. 1, led to the use of third- or fifth-order systems to describe the nerve-muscle interface. These models, based on the dynamic minimization of the jerk or snap, allow reconstruction of the kinematic movement from the shape of the pen-tip trajectory only.

at least a third-order linear system was necessary to simulate human handwriting performance. VI. HANDWRITING SIMULATION EXPERIMENT

Finally, Maarse [lo] in a model comparison experiment, has used a cascade of integrators to process triangular acceleration pulses. This approach is equivalent to assuming a purely ballistic third-order system in the neural

The formal mathematical representation described in the previous sections is very helpful in comparing, theoretically, the various models proposed to date but may also be helpful in studying the variable used by the brain to

1068 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, VOL. 19. NO. 5 , SEPTEMBER/OCTOBER 1989

Simulation I 2 3 4 5 6

Fig. 4. Displacement, velocity, and acceleration used for simulations 1-6. Acceleration inputs were set. Velocity and displacement were computed from these inputs.

control these systems and in evaluating which model is the best for producing and reconstructing natural handwriting. Indeed, computer simulations can possibly answer this question. To implement all the different models with opti- mal values for each specific parameter would be a lengthy task. Maarse [lo] showed that simplified pure ballistic models with different inputs can produce good reconstructions of natural handwriting, although (external) friction and elasticity were neglected (which is the case for many of the models presented in Tables I and 11). Provided that handwriting models are linear, system parameters can be transferred to the input of a pure ballistic linear system, and rather reliable simulations can be performed by choos- ing adequate inputs. As previously mentioned, the sum of the equivalent order of the inputs and the number of integrators used yields to the order of the whole model.

In t h s context, inputs fed to two integrators represent strokes in the acceleration domain and are called force impulses. Inputs or strokes defined in the velocity domain are the so-called momentum impulses. When there is no integrator at all, the inputs represent strokes in the spatial domain or represent pieces of handwriting and are called segments [30]. This domain of analysis has been used, for example, by Morass0 [25] who reconstructs handwriting from circular segments.

By defining inputs in the three domains mentioned, an experiment can be performed to study what type of input yields the best reconstruction. If inputs defined in a certain domain give significantly better results than others, it may be an indication that handwriting is controlled by variables in that domain. Or in other words, insight into the control variables (force, velocity, and spatial target or lengths) can be gained. Let us look at a more detailed description of the experiment performed, the reconstruction techniques, the inputs, and the results [lo].

A. Experimenl

Reconstructions were made of handwriting samples using 14 different simulations. Differences between these reconstructions and the original samples were computed to discover whch input(s), most accurately reconstructed natural handwriting.

The handwriting samples were recorded on a Calcomp 9240 X - Y tablet connected to a PDP-11/45 computer. The handwriting signal was sampled with a sampling frequency of 105.2 Hz. The pen was equipped with a standard ball-point refill. Four subjects were required to copy, in their normal handwriting, a text of eight lines on a sheet of paper. The horizontal writing direction was coincident

PLAMONWN AND MAARSE: AN EVALUATION OF MOTOR MODELS OF HANDWRITING 1069

Ve l o c i t y

1.6 m/s2 - - - - - - -

0.0 ‘r ]li p: - - - - t/ - - - f-l$l- ii_ A c c e l e r a t i o n -1.6 m/s2- - - - - - _ -

1 250 S 1

S i m u l a t i o n 7 E 9 10 I1 12

Fig. 5. Displacement, velocity, and acceleration used for simulations 7-12. Velocity inputs were set. Acceleration and displacement were computed from these inputs.

with the horizontal X direction of the X - Y tablet. Follow- ing data acquisition and filtering with a low-pass finite- impulse response (FIR) filter with a cutoff frequency of 10 Hz and a transition band of from 10 to 25 Hz, the simulations were then done on a VAX-11/750 computer connected to an interactive graphics display and a plotter.

B. Reconstruction Techniques I ) Finding Moments in Time and Length of Strokes: In

the 14 simulations performed, three types of strokes were derived from the original handwriting specimen. In the acceleration domain, strokes or force impulses were found for X, as well as for Y, by minima in the absolute acceleration in the X and Y directions, respectively (simulations 1-6). In the velocity domain, the momentum impulses were derived from minima in the absolute velocity for X and Y independently (simulations 7-12). The strokes or segments in spatial simulation 13 were derived from the minima in the absolute velocity. In simulation 14, minima as well as maxima in the absolute velocity were used. For the three types of strokes, the length and the velocity at the beginning of a stroke were computed from the original handwriting. For the momentum impulse, the initial velocity was, of course, zero.

2) Znputs Used: Based on stimulus timing, stroke length, and starting velocity, a new synthetic stroke was computed. For the force impulses, the shapes of the six inputs used are given in Fig. 4 (simulations 1-6). Simulation 1

shows a rectangular force impulse, as used by Denier van der Gon [30] and Dooijes [28]. Simulation 2 with its trapezoidal shape was inspired by MacDonald [13]. The exponential function of simulation 3 was suggested by Yasuhara [12]. Simulations 4-6, with sinus, bell, and trian- gle shapes, respectively, were introduced by Maarse [ 101. The same shapes were used as input for a first-order system for the simulations in the velocity domain (7-12 in Fig. 5 ) as those used for simulations 1-6.

In simulation 13, it was assumed that the segments are circular. The velocity distribution was supposed to be bell-shaped. By an overlap in time of 50 percent of the successive segments, a fluent course of the reconstructed trace was obtained. The curvature of a segment is derived from the pen position at the beginning, at the middle, and at the end of the original segment.

Simulation 14 consists of two linear systems working independently, one for long segments between minima of the absolute velocity and one for short segments between its maxima. Here again the velocity profiles were bell- shaped for both systems. With the two systems working simultaneously, the final output had a fluent course.

C. Performed Simulations and Computing Errors

In this pilot experiment, reconstructions of one-minute handwriting samples were made from the data obtained from four subjects, using the 14 simulation protocol previ-

1070

O r i g i n a l

1

3

5

7

9

11

13

Fig. 6.

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, VOL. 19, NO. 5, SEPTEMBER/OCTOBER 1989

p-l

2

4

6

0

10

12

14

Original handwriting and 14 reconstructions of dutch word fuif (party) written by one subject.

ously described. The partial results of the reconstruction are shown in Fig. 6 (only one word appears here for purposes of clarity). Note that the quality of the recon-

i output differs ,from one experiment to another. An .ojective spatial measurement as defined by Maarse [lo] was used to compute the error between the original and the reconstructed stroke. This measurement computes, for every stroke, the area between the original and the reconstructed stroke. By dividing this surface area by the square of the length of the stroke, a measurement independent of the size of the handwriting was obtained. For all handwriting specimens, the average value of this measurement was computed.

D. Results

Table I11 records the average error for the whole experiment for the four subjects. It will be noted that the simulations in the acceleration domain yield more or less the same results, with the exception of simulation 5 . A bell-shaped force impulse seems to produce a simulation which is significantly less realistic than the others. In the velocity domain, it is evident that simulation 7 yields the worse result. Indeed rectangular momentum impulses are unrealistic. Simulations 8-12 do not differ significantly. For the simulations in the spatial domain, the output with long and short segments seems to be less realistic.

With the exception of these three poor results (reconstructions 5 , 7, and 14), all the other reconstructions yield consistent results. In general, reconstructions in the velocity domain are significantly better, or, in other words, handwriting reconstructions controlled by momentum impulses are the best ones. However as will be discussed

later, the conclusion that handwriting is controlled by velocity impulses only is not evident from these results, but in these kinds of simulations they are superior to force impulses and segments. No significant difference in performance between second- and third-order linear systems was actually observed with this approach.

VII. CONCLUSION

The main purpose of this analysis was to propose a general mathematical context for analyzing handwriting models and to give an overview of handwriting models previously published. The proposed model definition seems to be helpful not only for mathematical modeling and comparing biophysical handwriting systems, but also for higher level handwriting modeling. It gves clear definitions of parameters like stimulus, stroke, and components, and offers the potential for standardizing models. For representation of the system, Laplace transformation theory is used. In the proposed description of handwriting models, the order of a system is defined between the final product on paper and an arbitrary step function as input. Theoretically if a first-order nerve-muscle interface is assumed, the minimum order of a system, between the abstract neural firing-rate domain and the spatial domain, should be at least three. When t h s assumption is not made, the order is at least two. The models are assumed to be linear and stationary.

The overview of previously published handwriting models indicates that all systems are of second and third order and are linear and stationary within one stroke. Only for simulation purposes were first- and fourth-order systems studied. It appears to be possible to describe all the previously published models with the proposed mathematical method.

The secondary purpose of this analysis was to gain further insight into the psychomotor aspects of handwriting. From the simulations performed, it seems that second- and third-order models with stimuli defined in the velocity domain yield the best results. Solely on the basis of these simulations, it cannot be concluded that the nervous system controls the velocity of handwriting movements, but it is evident that for simulation, regeneration and generation purposes, momentum impulses will yield the best results. It is clear that in these simulations, a first-order system cannot adequately describe a handwriting system. In general, in the acceleration domain, thrd- and fourth-order systems do not yield better results than do second-order systems. In the velocity domain, the type of input used does not lead to significant differences between second- and third-order systems, although a previous study [22] suggests that third-order overdamped systems would be superior to second-order overdamped models. These third-order overdamped models were not studied here.

Handwriting may be seen as a motor task producing a certain spatial output within relatively stringent time lim- its. Controlling velocity is, or seems to be, the simplest way to perform such a task. By generating momentum impulses handwriting can be divided into strokes with relatively less


TABLE I11 DISTANCE BETWEEN ORIGINAL NATURAL HANDWRITING OF FOUR SUBJECTS AND 14 RECONSTRUCTIONS

SUBJECT NUMBER SIMULATION SHAPE OF EQUIVALENT

NUMBER INPUT SIGNAL ORDER OF THE SYSTEM 1 ' 2 3 4 Mean

% % % 4 0. --- Acceleration 1 Rectangular 2 4 . 8 4 . 3 5 . 8 5 . 1 5 . 0

2 Trapezoidal 3 5 . 2 6 . 0 5 . 3 7 . 0 5.9

domain

3 Exponential 3 5.9 5.5 5.9 6 . 7 6 . 0

4 Sinusoidal 4 5 . 6 6.5 5 . 5 7 . 5 6 . 3

5 Bell-shaped 4 11.0 11.4 1 0 . 5 1 2 . 0 11.2

6 Triangular 3 6 . 2 7.1 5 . 9 8 . 0 6 . 8

Velocity domain

Spatial domain

7 Rectangular 1 7 . 1 6 . 8 1 0 . 8 6 . 4 7 . 8

0 Trapezofdal 2 4 . 0 3 . 7 7 . 5 4 . 1 4 . 8

9 Exponential 2 3.3 3.3 7 . 0 4 .7 4 . 6

10 Sinusoidal 3 3 . 4 3 . 1 6.7 3 . 9 4 .0

11 Bell-shaped 3 3 . 4 3 .6 5 . 7 4 .3 4 . 3

12 Triangular 2 3 . 2 3 . 2 7 . 0 4 . 1 4 . 4

13 Circular strokes 3 5.0 4 . 8 5 .6 5 . 3 5 . 2

14 Long and short 3 7 . 3 7 . 3 9.5 8 . 1 8 . 1

strokes

activity at the beginning and at the end. These points can be seen more or less as spatial targets, and it is easy to develop motor programs based on such spatial targets. In a training phase these targets can be used as feedback. If handwriting is assumed to be force-impulse controlled, the feedback mechanism is more complex. The beginning and the end of a force stroke coincide more or less with velocity extrema.

Moreover other indirect data also support velocity as a control variable. Indeed, in a recent comparison experiment, it has been suggested that the velocity domain seems to be the best representation space for automatic signature verification, as compared to the position and acceleration domains, for systems using a X-Y tablet as input. Com- bining results from three different signal comparison algo- rithms, this study showed that smaller error rates were obtained when X - Y signals were used to represent a

two-dimensional signature signal [35]. If fast fine-motor human behavior is controlled in the first instance in the velocity domain, as suggested here, this conclusion could be of great importance in handwriting education and in other fine-motor training programs. The task to be performed must contain easily recognizable target points. During training, these points can be used intentionally for segmentation in strokes or other small parts of movements. Well-chosen target points may ultimately be expected to reduce the motor program development procedure.

For example, interactive systems are currently being developed to help children learn handwriting. These systems display characters on a screen and the student is asked to reproduce them. A character-recognizer module gives the proper feedback about their performance [36]. This approach has the advantage of teaching not only how to write the image of a character but also the proper stroke

1072 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, VOL. 19, NO. 5, SEPTEMBER/OCTOBER 1989

order. Our conclusion suggests that using stroke definition

characters would result in the most realistic patterns in terms of the dynamic visualization of the whole generation

handwriting: A computational model,’’ Biol. Cybern., vol. 45, pp. 131-142, 1982.

handwriting generation?” in Proc. 3rd Int. Symp. Handwriting and Computer Applications, Montreal, PQ, Canada, 1987, pp. 11-13.

[27] -, “A handwriting model based on differential geometry,” in Computer Recognition and Human Production of Handwriting,

and representation in the domain to generate [26] R, plamondon, “What does differential geometry tell us about

process.

REFERENCES

A. J. M. W. Thomassen, P. J. G. Keuss, and G. P. van Galen, “Motor aspects of handwriting,” Acta Psychologica, vol. 54, p. 354. H. S. R. Kao, G. P. Van Galen, and R. Hoosain, Graphonomics Contemporun, Research in Handwriting. Amsterdam, The Nether- lands: North Holland, 1986, p. 403. R. Plamondon, C. Y. Suen, J. G. Deschenes and G. Poulin, Proc. 3rd Int. Svmp. Handwriting and Computer Applications, Edition &ole Polytechnique de Montreal, PQ, Canada, 1987, p. 196. M. Martlew, The Psvchology of Written Language: Developmental and Educational Perspectives. R. Plamondon and G. Lorette, “Automatic signature verification and writer identification: The state of the art,” Pattern Recognition, vol. 22, no. 2, pp. 107-131, 1989. D. Charraut, J. Duvernoy and L. Hay, “L’analyse automatique de I’tcriture,” Lu Recherche, no. 184, Jan. 1987. C. Y . Suen, M. Berthod, and S. Mori, “Automatic recognition of handprinted characters: The state of the art,” Proc. IEEE, vol. 68, Apr. 1980, no. 4. C. Y. Suen and R. De Mori, Computer Analysis and Perception: Vol. 1. Visuul Signals, Boca Raton, FL: CRC Press, 1982, p. 157. R. B. Stein, “What muscle variable(s) does the nervous system control in limb movements?” Behavioral Brain Sci., vol. 5, pp. 535-577, 1982. F. J . Maarse, The Studv of Handwriting Movement: Peripheral Models arid Signal Processing Techniques. Lisse, The Netherlands: Swets & Zertlinger. 1987, p. 160. C. C. Tappert, C. Y . Suen, and T. Wakahara “On-line handwriting recognition: A survey,” in Proc. 9th Int. Conf. Puttern Recogni- tion, Rome, Nov. 1988, pp. 1123-1132. K. S. Lashley, “The accuracy of movements in the absence of excitation from the moving organ,” Amer. J . Physiol, vol. 20, p. 169, 1917. J. J. Denier van der Con and J. P. Thuring, “The guiding of human writing movements.” Kvhernetik, vol. 4, no. 2, pp.145-148, Feb. 1965. M. Yasuhara, “Experimental studies of handwriting process,” Rep. Univ. Electro-Comm., vol. 25, pp. 233-254, 1975. K. U. Smith and R. Schappe, “Feedback analysis of the movement mechanisms of handwriting,” J . Exp. Education vol. 38, no. 4, pp, 61-68, 1970. D. I. Margolin, “The neuropsychology of writing and spelling: Semantic, phonological, motor, and perceptual processes,” Quart. J . Expen‘menrul Ps.ychol., 36A, pp. 459-489, 1984. G. P. van Galen, “Structural complexity of motor patterns: A study on reaction time and movements of handwritten letters,” Psychol. Res., vol. 46, 1984. G. P. van Galen and H. L. Teulings, “The independent monitoring of form and scale factors in handwriting,” Motor Aspect of Hand- writing. Acta Psychologica. Amsterdam, The Netherlands, North Holland, 1983. J. S. MacDonald, “Experimental studies of handwriting signals,” Ph.D. dissertation, Mass. Inst. Technol., Cambridge, Sept. 1984. P. Mermelstein and M. Eden, “Experiments on computer recognition of connected handwritten words,” Inform. Contr., vol. 7, pp. 255-270. 1964. M. Eden, “Handwriting and pattern recognition,” IRE Trans. Inform.. Theorv, vol. IT-8, pp. 160-166, 1962. R. Plamondon and F. Lamarche, “Mcdelization of handwriting: A system approach,” in Graphonomics: Contemporary Research in Hundwriring. H. S . R., Kao, G. P. Van Galen, and R. Hoosain, Eds. Amsterdam, The Netherlands: Elsevier Sci., North Holland, 1986, pp. 169-183. E. H. Dooijes, “Analysis of handwriting movements,” Acta Psycho- logicu, vol. 54, pp. 99-114, 1983. J. M. Hollerbach. “An oscillation theory of handwriting,” Biol. Cvhern.. no. 39. nn. 139-156. 1981.

New York: Wiley, 1983, p. 428.

R. Plamondon,C. Y. Suen, and M. Simner, Eds. Hong Kong: World Scientific, pp. 179-192, 1989. J . T. Stern, “Computer modelling of gross muscle dynamics,” J . Biomech., vol. 7, p. 411, 1974. E. H. Dooijes, “Analysis of handwriting movement,” Ph.D. dissertation, Univ. of Amsterdam, Amsterdam, The Netherlands, 1983. R. Plamondon and F. J. Maarse, “A neuron oriented representation to compare biomechanical handwriting models,’’ in Proc. 3rd Int. Symp. Handwriting and Computer Applications, Montreal, PQ, Canada, 1987, pp. 2-4. S. Kondo and B. Attachoo, “Model of handwriting process and its analysis,’’ in Proc. R f h Int. Conf. Pattern Recognition, Paris, France,

J. J . Denier van der Gon, J. P. Thuring, and J. Strackee, “A handwriting simulator,” Phys. Med. Biol., vol. 6, pp. 407-414, 1961. J. Vredenbregt and W. G. Koster, “Analysis and synthesis of handwriting,” Philips Tech. Rev., vol. 32, pp. 73-78, 1971. S. Edelman and T. Flash, “A model of handwriting,” Biol. Cybern., vol. 56, pp. 25-36, 1987. R. Plamondon and M. Parizeau, “Signature verification from position, velocity and acceleration signals: A comparative study,” in Proc. 9Ih Int. Conf. Pattern Recognition, Rome, Italy, Nov. 1988,

H. Kamada, Y. Yamaguch, Y. Kijima, and T. Fujita, “A cal system based on Kaiyi recognition for learning Japanese,” in Proc. 3“‘ Int. Conf. Hundwriting and Computer Applications, Montreal, PQ. Canada, July 1987, pp. 188-190.

[28]

[29]

[30]

[31]

1986, pp. 562-565. [32]

[33]

[34]

[35]

pp. 260-265. [36]

Rejean Plamondon (M’79-SM85) received the B.Sc. degree in physics and the M.Sc.A. and Ph.D. degrees in electrical engineering from Uni- versitt Laval, Quebec, PQ, Canada, in 1973,1975. and 1978. respectively.

In 1978, he joined the staff of the Ecole Poly- technique, Universitt de Montreal. Montreal, PQ, where he is currently an Associate Professor. From 1985 to 1986, he was involved in several research projects while a guest of the Computer Science Department, Concordia University,

Montreal, the Motor Behavior Laboratory, University of Wisconsin- Madison, the Department of Experimental Psychology, University of Nijmegen, The Netherlands, and the Laboratoire de Genie Electrique de Crtteil, Universitk de Paris Val-de-Marne, France. His research interests deal with the computer applications of handwriting: biomechanical models, neural and motor aspects, character recognition, signature verification, signal analysis and processing, computer-aided design via handwriting, forensic sciences, software engineering, and artificial intelligence. He is the founder and director of the Laboratoire Scribens at the Eo le Poltyechnique de Montrtal, a research group dedicated exclusively to the study of these topics. He is the author or coauthor of numerous publica- tions and technical reports.

Dr. Plamondon is a member of the board of the International Grapho- nomics Society and President of the IAPR technical committee on text processing applications.

Frans J. Maarse received the M S degree in electncal engineenng from the Technical Univer- sity of Delft, The Netherlands in 1970 and the Ph D degree in social sciences in 1987

He joined the Psychologcal Department of the Umversity of Nijmegen in 1971 He first promoted the use of computers in expenmental psychology and was for ten years Head of the Computer Section of the Psychological Depart- ment He is now involved in psychomotor and handwriting research at the Nijmegen Institute ~, . .., rr ~. ~... ~.

[25] P: Morass0 and F. A. Mussa Ivaldi, “Trajectory formation and for Cognition Research and Information Technology.

Documents

An evaluation of motor models of handwriting