IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 5, NO. 6, JUNE 1996 1073

[9] H. J. A. M. Heijmans, “Theoretical aspects of gray-level morphology,” ZEEE Trans. Patt. Anal. Machine Zntell., vol. 13, no. 6 , pp. 568-582,

on the observation of the BM, BBM, and input matrices. Analysis of the BM arid BBM shows that these matrices are skew-svmmetric and have many redundant entries. It is also shown that most entries in the successive input matrices at two adjacent time indices are identical. By eliminating these redundancies, a fast recursive formula for the

1991. [lo] D. Wang and J. Ronsin, “Bounded gray-level morphology and image

representation,” Tech. Rep.. INSA de Rennes, France, 1994.

Fast Recursive Algorithms €or Morphological Operators Based on the Basis Matrix Representation

Sung-Jea KO, Aldo Morales, and Kyung-Hoon Lee

Abstruct- A real-time implementation method for the most general morphological system, the so-called grayscale function processing (FP) system is presented. The proposed method is an extension of our previous works [5], [6] using the matrix representation of the FP system with a basis matrix (BM) and a block basis matrix (BBM) composed of grayscale structuring elements (GSE). In order to further improve the computational efficiency of the basis matrix representation, we propose recursive algorithms based on the observation of the BM and BBM. The efficiency of the proposed algorithms is gained by avoiding redundant steps in computing overlapping local maximum or minimum operations. It is shown that, with the proposed scheme, both opening and closing can be determined in real time by 2 N - 2 additions and 2 N - 2 comparisons, and OC and CO by 4 N - 4 additions and 4N - 4 comparisons, when the size of the GSE is equal to N . It is also shown that the proposed recursive opening and closing require only 3 N - 3 memory elements.

I. INTRODUCTION

Morphological operations can be classified into three types of operations: the set-processing (SP) operation. the function and set processing (FSP) operation, and the most general morphological operation, the function-processing (FP) operation. A variety of im- plementation algorithms developed for the order statistic and stack filters can be utilized for implementation of morphological FSP and SP operations. However, these algorithms can not be directly applied to Fp operations using GSE that do not obey threshold decomposition. Moreover, FP openingklosing using the cascade representation (erosioddilation followed by dilatioderosion) requires memory storage for the first erosioddilation outputs, and delays are introduced.

In this paper, we present efficient real-time implementation algo- rithms for the most general grayscale morphological FP operations [1]-[4]. The proposed method is based on our previous work using the matrix representation of the FP system with the basis matrix (BM) [5] and block basis matrix (BBM) [6], which is an extension of the morphological basis representation theory [7] , [SI. A procedure was proposed for the derivation of the BM for FP opening and closing, and the BBM for FP open-closing (OC) and clos-opening (CO) from any GSE [5] , [6]. It was also shown that the most general morphological operations including opening, closing, OC, and CO are accomplished by a local matrix operation with the BM and BBM rather than cascade combinations of erosion and dilation, eliminating delays and requiring less memory storage.

In order to further improve the computational efficiency of the basis matrix representation, we propose recursive algorithms based

Manuscript received November 29, 1994; revised August 30, 1995. S.-.I. KO and K.-H. Lee are with the Department of Electronics Engineering,

A. Morales is with the College of Engineering, Pennsylvania State Univer-

Publisher Item Identifier S 1057-7149(96)04220-0.

Korea University, Seoul, Korea (e-mail: sjko (3 dali.korea.ac.kr).

sity, DuBois, PA 15801 USA.

proposed matrix Operations can be obtained, which can significantly reduce the required computations. The efficiency of the proposed algorithms is gained by avoiding redundant steps in computing overlapping local maximum or minimum operations. To evaluate the computational efficiency of the proposed scheme, the required numbers of operations for each morphological operator including opening, closing, OC, and CO are calculated. It is shown that, with the proposed scheme, both opening and closing can be determined in real time by 2N - 2 additions and 2 N - 2 comparisons, and OC and CO by 4 N - 4 additions and 4N - 4 comparisons, when the size of the GSE is equal to h’. It is also shown that the proposed recursive opening and closing require only 3N - 3 memory elements.

The organization of this paper is as follows. In Section 11, grayscale morphological operations are defined and the procedure to derive the BM and local matrix operations for opening and closing are briefly reviewed. The fast recursive implementation algorithms for opening and closing operators are proposed in Section 111. In Section IV, the matrix representation of OC and CO using the BBM are explained and the fast recursive implementation algorithms for OC and CO are presented. Finally, the concluding remarks are presented in Section V.

11. BASIS MATRIX REPRESENTATIONS FOR OPENING AND CLOSING

In this section, we first briefly review the basis matrix representa- tion, since the proposed recursive: algorithms are an extension of our previous works [5] , [6] .

The basic morphological operations are defined as follows: Let the grayscale input signal be denoted by f , and a GSE of size N by k, where the domain of k is IC. The FP dilation and erosion operations of f by k, respectively, are defined by

g<i(n) = (f @ k)(n) =: max[f(n - z) + k ( z ) ]

g , ( n ) = (f 3 k)(n) =: min[f(n + z ) - k ( z ) ] .

(1)

(2)

The FP opening and closing operations of f by k, respectively, are defined by f o k = (f 8 k) @ k and f 0 k = (f @ k) 0 k.

Using the max-midmin-max representation for grayscale morpho- logical openingklosing, the opening and closing can be represented as follows.

Proposition 1: The FP opening and closing of f by k are equiv- alent to

Z € K

=€I<

min[f(n - z + z’) + b ( z , z ’ ) ] } (34

gc(n) = min m$x[f(n + z - z ’ ) - b ( z , z ’ ) ] z + c

where b ( z , z ’ ) = k ( z ) - k ( z ’ ) , i E A’ and z’ E IC. This property is also a direct consequence of the morphological

representation theory (Theorem 3 in [SI). The max-midmin-max representations using this property for grayscale morphological open- ingklosing, although more complex, are faster than the cascade representations since openingklosing can be accomplished by a local operation of neighborhood input samples.

Proposition 1 can be expressed in matrix form [5] using the input matrix and the basis matrix as follows: The N x N input matrix F ( n ) consists of 2N - 1 input samples, { f ( n - AJ+ l), . . . , f (n+ N - l)},

1057-7149/96$05.00 0 1996 IEEE

I074 lEEE TRAhSACTIONS ON IMAGE PROCESSING, VOL 5, NO 6, JUNE 1996

Fig. I . The local matrix operalion for opening with the GSE of size three

and i s defined by

(4)

The :V x 1V basis matrix I3, whose elements consist of { b ( i . j ) } , is defined by

b ( 0 , 0) b ( 0 . 1 ) ' . ' b(O..\. - 1)

b( 1. .\- - 1) I i 5 )

= [bx:oj b ( - Y i 1.1) I:: b .3 - - 1. j .\- - 1)

h ( 1; 1) . . '

where b ( 2 . j ) = k ( i ) - k ( j ) . It i s interesting to observe several properties of the BM. First, each row represents a basis function as defined by 171 and [SI. Second, since h ( i . , j ) = k ( i ) - k ( , j ) = - ( k ( j ) - k ( i j ) = -b( . j . j ) a n d b ( i . i ) = 0, theBMis skew-symmetric and the total number of distinct elements in the BM can be reduced from .\-' down to [ -IT2 - A-)/2. Moreover, the total number of linearly independent elements in the BM can be reduced from 2.1- - 1 down to Y - 1.

The output of opening g0(7i j can be obtained by finding the maximum of the minima of each rows from the matrix F ( n ) + L?. The output of closing, q , . ( r i j , i s the minimum of the maxima of each column of the matrix F ( n ) + I?. It can be easily shown that these local matrix operators require -IT2 - A\- additions and :Ira - 1 comparisons for an openingiclosing output. To clarify the basis matrix representation, a signal flow graph of the local matrix operation for opening with the GSE of size three is presented in Fig. 1.

In fact, the cascade implementation requires 2.Y - 2 additions and 2 Y - 2 comparisons for each output of opening/closing. However, the main drawback of this implementation method is that it cannot calculate the opening/closing outputs in real time. To obtain the output of openingklosing, an input signal is eroded/dilated, and the inter- mediate result needs to be stored for the following dilatioderosion operation. Therefore, the cascade implementation method introduces

Fig. 2. of size three.

The fast recursibe implementation method of opening with the GSE

delays and requires a large amount of memory storage for the first erosion/dilation output.

Although the matrix operations described above are more complex than the cascade implementation method, they can directly calculate the outputs from the input samples and, thus can eliminate delay. Moreover, the local matrix operators can reduce memory requirement since they do not need to store the intermediate erosioddilation result. Since there are ( :V2 - n ' ) /2 distinct elements in the EM, the same number of memory elements are required. The storage for the intermediate columnwise maximdlow-wise minima from F ( n ) + B requires T memory elements. Therefore, the total number of memory elements for a local matrix opening/c~osing operator is equal to (LV2 + .Y ) / 2 , Although a parallel architecture of the cascade implementation also can reduce the delay, it requires additional hardware for the concurrent calculations of the first erosion/dilation stage.

In the next section, we will propose fast recursive algorithms in order to further improve the computational efficiency of the basis matrix representation.

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 5, NO. 6, JWNE 1996 1075

OC The local matrix operation for OC with GSE of size two. Fig. 3.

111. FAST RECURSIVE OPENING AND CLOSING

In this section, fast recursive opening and closing are proposed based on the basis matrix representation.

As described in the previous section, the outputs of opening and closing, respectively, are determined by using the row-wise minima and the column-wise maxima of the matrix F ( n ) + L3. Expressing the opening and closing operations in terms of the row-wise min and column-wise max operations gives

where R,(n) and C,(n) , respectively, denote the minimum of the ith row and the maximum of the ath column of the matrix F ( n ) + L3, i.e.

The proposition stated below indicates that the outputs of the opening and closing operators at time n can be recursively obtained from the previous operation results, {R;(n> - l), 0 5 i 5 N - 2) and {C,(72 - 1); 1 5 j 5 1V - 1). Proposition 2: The ith row-wise minimum R,(n) and the j t h

column-wise maximum C,(n) of F ( n ) + L3 can be obtained using the following recursive formula:

n,(7!,) = R,-,(n - 1) + b ( i , i - l),

C, (n ) = C]+1(7L - 1) + b ( j + L j ) ,

i = 1 ,2 , . . . , N - 1 (Sa)

j = O , l , ..., N - 2 . (8b)

Proo) For i = 1,2 , . . . , N - 1

R,(n,) = min[f (n - i ) + b ( i , O), f ( n - i + 1) + b ( i , l ) , . . . , f ( n - i + M - 1) + b ( i , N - I ) ]

f ( n - i + 1) $- b ( i - 1 , l )

+ b ( i - 1, N -- 1) + b ( i , i - 1)]

+ b ( i - 1, I) , . . . , f ( n - i + 1v - 1)

= min[f(n - i ) + b ( i - 1 , O ) + b ( i , i - I),

+ b ( i , i - l), . . . , f(n, - i + N - 1)

= min[f(n - i ) + b ( i - 1, O), f ( n - i + I)

+ b ( i - 1, N - l)] + b ( i , i - 1) = R,-l (n - 1) + b ( i , i - 1). (9)

In a similar way, C,(n) can be obtained using C,+,(n. - 1) recursively.

Using this proposition, the opening and closing operations can be redefined as

g , (n ) := max[Ro(n),Ro(n-- l ) + b ( l , O ) , ..., R N - Z ( n - 1)

+ b(AJ - 1, N - 2)]

+ b ( N - l,A'-2),Civ--l(n,)]. (lob)

(104 gc(n,) = min[Cl(n - 1) + b(l,O), . . . , C.v-1 ( n - I),

These equations represent a fast implementation method of the opening and closing operations. The calculation of RO (n) requires N - 1 additions and fV - 1 comparisons. The calculation of { R, (n - 1) + b ( i , i - I), 0 5 i 5 N -- 2) requires N - 1 additions. The maximum operation for yo ( n ) in1 (loa) requires N - 1 comparisons. Thus, the total number of operations required to determine opening and closing at each point is equal to 2 N - 2 additions and 2fV - 2 comparisons. Therefore, the opening and closing operations using Proposition 2 is computationally more efficient than the opera- tions without using Proposition 2 (Ai2 - N additions and N 2 -

1076 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 5 , NO. 6, JUNE 1996

1 comparisons). Moreover, the proposed recursive algorithms can significantly reduce the memory requirement. Consider the opening operation in (loa). Ro(n), {R , (n - 1): 0 5 i 5 Y - 2) and ( b ( 1 , O ) , b ( 2 , I), . . . , b ( N - 1,IV - a)}, respectively, require /I- - 1 memory elements. Therefore, the total number of memory elements for a recursive opening operator becomes 313' - 3.

To illustrate the computational efficiency of the proposed fast implementation method, a signal flow graph of the proposed recursive operator for opening of (loa) is presented in Fig. 2. Note that this recursive structure has significantly fewer computations than the implementation method of Fig. 1 without using Proposition 2.

Iv. FAST RECURSIVE ALGORITHMS FOR OC AND CO

Before we propose our fast recursive algorithms for CO and OC, it is worthwhile to review the basis matrix representation using the N 2 x N 2 input matrix F(n) and the M2 x !IT2 block basis matrix C [6] as follows:

F(7z) F (.) F(7z + 1) . . . F ( n + -7- - 1)

F(n? - 1) F(n ' ) . . . F ( n + -1- - 2 )

r(" - N + 1) F(" - 1v + 2) . . . =i (1 1) and

C(0, l ) . . . C(0, .j- - 1) C( 1.0) C(1,1) . . . C(1, -7- - 1)

C('V ~ 1 , O ) C ( N - 1.1) . . . C ( S - l..Y ~ 1)

(12)

where F ( n ) denotes the input matrix of (4) and C ( i . j ) = B + [ b ( i , j ) ] with [ b ( i , , j ) ] representing a matrix whose elements are all equal to b ( i , j ) . Thus, the ( k , l)th component C"'( i , j ) of the ( i . j ) th submatrix C ( i , j ) is C k 2 ' ( i , j ) = h ( k , Z ) + b ( i , j ) . Note that this block matrix is skew-symmetric.

The output of the OC operator at time n is obtained in real time according to the following steps: Add matrices in (11) and (12), and at each submatrix of F(n) + C find the row-wise minimum. The maximum of the row-wise minima at each submatrix is the element of the resultant 1V x 1Y matrix. From this resultant matrix, find the maximum of each column. The output of the OC operator is the minimum of the column-wise maxima. Conversely, the output of the CO operator can be obtained by substituting maximum for minimum and column-wise for row-wise operations in the above procedure. This calculating procedure is illustrated in Fig. 3 using the signal flow graph. Once the BBM is constructed, the outputs of OC and CO can be obtained in real time by !IT4 - .Vz additions and N4 - 1 comparisons from the set of input samples. Next, we propose recursive algorithms for the basis matrix implementation to improve the computational efficiency of the basis matrix representation.

In the calculating procedure for an OC/CO output, the intermediate results obtained from each of the submatrices of F(n)+C correspond to the outputs of the openingklosing operation. For example, for the CO operation. the output of the ( i , j ) th submatrix becomes g , ( n ~ i + j) + b ( z , j ) . Once all the output of the submatrices are obtained, the output of CO is determined by selecting the maximum of the row-wise minima as follows:

g,,(n) = max[ilo(17,),fi1(n);. . . . R W - l ( 7 l ) ] (13) where l?,(n) denotes the minimum of the ith row of the matrix

I 1 b(W)

oc Fig. 4. two.

The fast recursive implementation method of OC with GSE of size

G,(n) + B , i.e.

k , ( n ) = min[g,(n - i + j ) + b ( Z , j ) ] . (14) ]ti<

Proposition 2 can give a recursive formula for RL ( n ) as follows:

fiz(n) = f i t - l (n - 1) + b ( i , i - 1). 1 5 i 5 N - 1. (15) Using (15), the .I-- 1 elements in (13), {k~(n). &(n), . . . , ~ N - I }

can be obtained from the previous computation results by !V - 1 additions. The only unknown value in (13), Ro(ri), is given by

&(n) = miri[g,(n) + b ( O . O ) . g , ( n + 1) + b(0. I);. . . ,g,(lc + *V - 1) + b ( O , N - 1)). (16)

Notice that & ( n ) and - 1) have N - 1 closing operation resultsin common, which are {g.(n),g,(n+l), . . . , g c ( n + N - 2 ) } . These N - 1 overlapped closing operation results do not need to be calculated for Ro(TL). In (16), the first N - 1 elements,

2)}, can be obtained by N - 2 additions, since b(0 ,O) is equal to zero. To determine the last element, y,(n + N - 1), 21V - 2 additions and 2 N - 2 comparisons are needed when the fast implementation algorithm described in Section 111 is exploited. Thus, 3 S - 3 additions and 3 N ~ 3 comparisons are required to compute &o(n) . In (13), the max- operation requires N - 1 comparisons and { ( n ) , Rz ( n ) , . . . . R1v- I } does N - 1 additions. Therefore, the total calculation required for each of the outputs of FP CO is 4-I- - 4 additions and 4N - 4 comparisons. In a similar way, a fast implementation method for the FP OC operation can be obtained.

{ g c ( n ) + b ( O ; O ) ; g , ( n + l ) + b ( O , l ) , . . . , g , ( ' ~ ~ + $ i \ i - 2 ) + b ( 0 , N -

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 5 , NO. 6, JUNE 1996 1077

The signal flow graph for the fast recursive implementation method of OC with the GSE of size two is presented in Fig. 4. The comparison of Fig. 3 with Fig. 4 shows that the fast recursive structure in Fig. 4 requires significantly fewer computations for an

V. CONCLUSION Efficient real-time implementation methods for the FP nnorphologi-

cal operators were presented by extending our previous work 151, [6]. It was shown that the proposed recursive algorithms can improve the computational efficiency of the basis matrix implementation method by avoiding the redundant steps in computing overlapping min/max operations. It was also shown that, with the proposed recursive algorithms, both opening and closing can be determined in real time by 2N - 2 additions and 2N - 2 comparisons, and bolh OC and CO by 4 N - 4 additions and 41V - 4 comparisons when the size of the GSE is equal to N . Moreover, the proposed recursive algorithms can reduce the memory requirement further than the basis matrix representations.

output of oc.

REFERENCES G. Matheron, Random Sets and Image Geometry. New York Wiley, 1975. J. Serra, Image Analysis and Mathematical Morphology. New York: Academic, 1982. C. R. Giardina and E. R. Dougherty, Morphological Methods in Image and Signal Processing. R. M. Haralick, S. R. Stemberg, and X. Zhuang, “Image analysis using mathematical morphology,” IEEE Trans. Pattern Anal. Machine Intell.,

S. J. KO, A. Morales, and K. H. Lee, “Matrix representation of composite morphological function processing systems,” in Proc. 36th Midwest Symp. Circuits Syst., Aug. 1993. -, “Block basis matrix implementation of the morphological open-

Englewood Cliffs, NJ: Prentice Hall, 1988.

vol. PAMI-9, pp. 523-550, July 1987.

closing and clos-opening,” IEEE Signal Processing Lett., vol. 2,- pp. 7-9, Jan. 1995. P. Maragos and R. W. Schafer, “Morphological filters-Part I: Their set-theoretic analysis and relations to linear shift-invariant filters,” IEEE Trans. Acoust., Speech, Signal Prucessing, vol. ASSP-35, pp. 1153-1 169, Aug. 1987. P. Maragos, “A representation theory for morphological image and signal processing,” IEEE Trans. Pattern Anal. Machine Intell., vol. 11, pp. 586-599, June 1989.

Motion Estimation Using Higher Order Statistics

Elisa Sayrol, Antoni Gasull, and Javier R. Fonollosa

Abstract-The objective of this paper is to introduce a fourth-order cost function of the displaced frame difference (DFD) capable of estimating motion even for small regions or blocks. Using higher than second-order statistics is appropriate in case the image sequence is severely corrupted by additive Gaussian noise. Some results are presented and compared to those obtained from the mean kurtosis and the mean square error of the DFD.

Manuscript received December 15, 1994; revised October 3, 1995. This work was supported by the Spanish Ministry of Education under Contract TIC 95-1022-CO5-04. Portions of this paper were presented at EUSIPCO-94, Edinburgh, Scotland, September 1994.

The authors are with the Universitat Politkcnica de Catalunya, Depart- ment of Signal Theory and Communications, Barcelona, Spain (e-mail: elisam gps.tsc.upc.es).

Publisher Item Identifier S 1057-7149(96)04173-5.

I. INTRODUCTION There is a growing interest in applications involving the estimation

of 2-D motion or velocity field between consecutive image frames 1141. There are some situations where motion has to be estimated in the presence of noise. These include motion estimation or motion compensation applications, as in images from surveillance cameras or medical images such as echographics with speckle noise. In such cases, most existing methods do not work properly, and more robust techniques are necessary. On the other hand, noise can be realistically described as a colored Gaussiam process. In such circumstances, higher order statistics (HOS) may offer some advantages since cumulants of Gaussian processes are asymptotically zero.

HOS-based methods have already begun to be used in motion estimation. In [7] the displacement vector is obtained by maximiz- ing a third-order statistics criterion. In [2] several algorithms are developed based on a parametric cumulant method, a cumulant- matching method, and a mean kurtosis error criterion. The latter is an extension of the quadratic pixel-recursive method by Netravali and Robbins [lo]. Other improved extensions to this algorithm were given by Walker and Rao [I81 and Biemond et al. 131. On the other side, iterative solutions that involve additional constraints to compute optical flow have been studied by Horn and Shunk [5] and latter by Nagel 191 using, in both cases, smoothness constraints. See [17] for a review on motion estimation techniques.

In this correspondence, we propose an alternative criterion that exploits HOS 1121, [13]. However, our goal is to obtain a low- variance cost function to reduce the problems associated with the estimation of HOS for small blocks of data. Our method is based on an adaptive algorithm that was proposed in [I] for the estimation of fourth-order cumulants. The motivation behind this approach is to use previous frames and previously estimated displacements.

This work is organized as follows. The problem formulation is introduced in Section 11. In Sections I11 and IV, cost functions based on the variance and the kurtosis of the DFD are revised. In Section V, a new class of HOS-based cost. functions is derived. In Section VI, a recursive version of the new cost function is presented. Simulation results are provided in Section VI1 and, finally, Section VI11 is devoted to conclusions and final remarks.

11. PROBLEM FORMULATION The problem of motion estimation can be stated as follows: “Given

an image sequence, compute a representation of the motion field that best aligns pixels in one frame of the sequence with those in the next” [IO]. This is formulated a’s

g k - 1 (mj = fk-L(mj + 1L.k-1 (m)

g k ( m ) = . f i ( m ) -k w ( m ) == f k - l ( m - &(m)j + %(m) (1)

where m = ( I I L , n) denotes spatial image position of a point; yk (m) and gk-l(m) are observed image intensities at instant I; and IC - 1, respective1y;fk (m) and fk--l (m) are noise-free frames; nk(m) and 7Lk- l (m) are assumed to be spatially and temporally stationary, zero- mean image Gaussian noise sequences with unknown covariance; and dg(m) is the displacement vector of tht. object during the time interval [ k - 1, k ] . The noise-free signals are assumed to be zero- mean non-Gaussian random fieldis that are statistically independent of the noise. In this formulation, the basic assumption is intensity constancy, as follows:

fn(m) = fk -1 (WL - &(mj). (2)

1057-7149/96$05.00 0 1996 IEEE