Recursive soft morphological filters

IEEE TRANSACTIONS ON IMAGE PROCESSING. VOL. 4, NO. I, JULY 1995 1027

Figs. 5 and 6 and is compared with mean reconstruction, in which all available adjacent blocks are evenly weighted, and with optimal reconstruction, obtained by generating the weights as if the lost blocks are known. In “Lena,” PSNR performance is approximately halfway between optimal and mean reconstruction, while “Couple’s’’ PSNR performance is closer to that of the mean. ”Lena” contains larger homogeneous areas than the more detailed “Couple,” and hence more blocks in “Lena” can be accurately reconstructed as a sum of their neighbors. However, both reconstructed images maintain good visual quality. Thus, while the algorithm may not reproduce lost blocks exactly, it generates fills that are both smooth and adequate. This algorithm has also been successfully applied to reconstruct lost blocks in intraframes in MPEG-like coded video [lo], and a variation that requires less than 10% transmission overhead to generate higher quality reconstructions has also been proposed [l].

V. CONCLUSION

A general image reconstruction technique for block based transform coded images is proposed that reconstructs lost transform coefficients by forming a linear combination of the same coefficients in available adjacent blocks subject to a smoothing constraint. This allows the local image characteristics to determine the structure of a reconstructed block, thereby producing visually pleasing reconstructed images. A maximum of only four weights is required to reconstruct any or all lost coefficients in each damaged block, and decoder computation for each entirely lost block requires less than 2.2 times the computations to perform an inverse DCT. Good visual performance is observed under loss rates as high as 30%.

REFERENCES

[l] M. Ghanbari, “Two layer coding of video signals for VBR networks,” IEEE J. Select. Areas Commun., vol. 7, no. 5, pp. 771-781, June 1989.

[2] G. Morrison and D. Beaumont, “Two layer video coding for ATM networks,” Signal Processing: Image Communication, vol. 3, no. 2-3, pp. 179-195, June 1991.

[3] N. Shacham and P. McKenney, “Packet recovery in high-speed networks using coding and buffer management,” in Proc. IEEE lnfocom ’90, Los Alamitos, CA, vol. 1, 1990, pp. 12&131.

[4] N. MacDonald, “Transmission of compressed video over radio links,” BTTechnol. J., vol. 11, no. 2, pp. 182-185, Apr. 1993.

[5] R. H. J. Veldhius, “Adaptive restoration of unknown samples in discrete- time signals and digital images,” Ph.D. thesis, Katholieke Univerisiteit te Nijmengen, The Netherlands, 1988.

[6] Y. Wang and Q.-F. Zhu, “Signal loss recovery in DCT-based image and video codecs,” in SPIE Con$ Visual Commun. Image Processing, Nov. 1991, vol. 1605, pp. 667478.

[7] JPEG-9-R7: Working draft for development of JPEG CD, Feb. 1991. [8] S. Sheng, A. Chandrakasan, and R. W. Brodersen, “A portable

multimedia terminal,” IEEE Communications Mag., vol. 30, no. 12, pp. 64-75, Dec. 1992.

[9] K. R. Rao and P. Yip, Discrete Cosine Transform. San Diego: Aca- demic, 1990.

[lo] S. S. Hemami and T. H.-Y. Meng, “Spatial and temporal video reconstruction for nonlayered transmission,” in Proc. 5th Int. Workshop Packet video, Berlin, Germany, Mar. 1993.

[ l l ] S. S. Hemami and R. M. Gray, “Image reconstruction using vector quan- tized linear interpolation,” in Proc. ICASSP ’94, Adelaide, Australia, vol. 5, 1994, pp. 629432.

Recursive Soft Morphological Filters

Frank Y. Shih and Padmaja Puttagunta

Absfract- In this correspondence, we present properties of recursive soft morphological filters that use previously filtered outputs as their inputs, cascade combinations of these filters, and the idempotent recursive soft morphological filters. The development allows problems in the imple- mentation of cascaded recursive soft morphological filters to be reduced to the equivalent problems of a single recursive standard morphological filter.

I. INTRODUCTION

Mathematical morphology, which is based on set-theoretic concept, extiacts object features by choosing a suitable structuring shape as a probe [4], [lo], [ I l l , [14]. The applications of morphological filters in image processing and analysis are numerous, which include shape recognition [l], [13], industrial parts inspection [12], [17], nonlinear filtering [3], [8], [9], and biomedical image processing [lo], [16]. The structuring element in morphological filters can be regarded as a template that is translated to each pixel location in an image. These filters can be implemented in parallel due to the fact that each pixel’s value in the transformed image is only a function of its neighboring pixels in the given image [5]. Also, the sequence in which the pixels are processed is completely irrelevant. Thus, these parallel image operations can be applied to each pixel simultaneously if a suitable parallel architecture is available. Parallel image transformations are also referred to as nonrecursive transformations.

In contrast to nonrecursive transformations, a class of recursive transformations are also widely used in signal and image processing-for example, in sequential block labeling, predictive coding, and adaptive dithering [5], [lo]. The main distinction between these two classes of transformations is that in the recursive transformations, the pixel’s value of the transformed image depends upon the pixel’s values of both the input image and the transformed image itself. Due to this reason, some partial order has to be imposed on the underlying image domain so that the transformed image can be computed recursively according to this imposed partial order. In other words, a pixel’s value of the transformed image may not be processed until all the pixels preceding it have been processed.

Koskinen et al . [6], [7], introduced soft morphological filters that possess the desirable property of being less sensitive to additive noises and to small variations in the shape of the objects to be filtered. The structuring element in soft morphological filters is divided into two parts: one being the “hard center” and the other being the “soft boundary.” Soft morphological filters can also be viewed as a special class of the weighted order statistic filters [2], [8], where only two weights are given to the two parts of the structuring element. In this correspondence, we introduce recursive soft morphological filters and their properties. A novel idea of reducing the cascaded recursive soft morphological filters to a single recursive standard

Manuscript received January 8, 1993; revised July 6, 1994. Supported by the National Science Foundation under Grant W-9109138 and the New Jersey Institute of Technology under Grant 421770. The associate editor coordinating the review of this paper and approving it for publication was Prof. Nikolas P. Galatsanos.

The authors are with Computer Vision Laboratory, Department of Computer and Information Science, New Jersey Institute of Technology, Newark, NJ 07102 USA.

IEEE Log Number 941 1856.

1057-7149/95$04.00 0 1995 IEEE

1028 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 4, NO. 7, JULY 1995

morphological filter is presented. A new class of idempotent recursive soft morphological filters and their properties are also presented.

This correspondence is organized as follows. In Section 11, the definitions of recursive soft morphological transformations are given. In Section 111, the properties of recursive soft morphological filters are discussed. In Section IV, the new class of idempotent recursive soft morphological filters are presented. In Section V, we describe several properties for cascade combinations of recursive soft morphological filters and a significant reduction of the cascade combination into a single recursive standard morphological filter.

Example 1: Let B = (-1.0, I), A = (0), and k = 2. Let the input signal f = (4 7 2 9 6 8 5 4 7 ) . We have

f eT [B,A,2] = (4 7 7 9 8 8 5 5 7 )

fe, [B,A,2] = (4 4 2 6 6 6 5 4 7 )

where “$,” and ‘‘0,” denote the recursive soft morphological dilation and erosion, respectively.

Fig. ](a) shows a 125 x 125 gray-level image, and Fig. l(b) shows the same image but with added Gaussian noise having zero mean and standard deviation 20. The mean-square signal-to-noise

11. DEFINITIONS OF RECURSIVE SOFT MORPHOLOGICAL FILTERS Let A and B be two finite convex sets in the N-dimensional Eu-

clidean space E N . The constraint on the two sets is A 2 B and B is divided into two subsets: the hard center set A and the soft boundary set B\A, where “\” denotes the set difference. Also, let k be a positive integer, such that 1 5 k 5 min{Card(B)/2, Card(B\A)}, where Card(B) denotes the cardinality of the set B. The translation of a set A by a vector 2 E E N is defined by A, = {a + 2 I a E A}.

A collection set of pixels where repetition is applied is called a multiset. The repetition k times of f(a) is represented by { k o f ( u ) } = {f(u),f(u),...,f(a)} (k times). The detailed definitions and properties of soft morphological transformations can be referred to [6, 7, 151. Let f denote the input gray scale image and A and B denote the structuring element sets such that A C B. Note that there is no gray value assigned to the structuring element, i.e., the structuring element is flat on the top.

Definition 1: The soft morphological dilation of f by [B. A, k ] is

(f @ [ B , A, k ] ) ( z ) = kth largest of multiset

{k 0 f(.) I a E AL) U I b E (B\A)z l . (1 )

Definition 2: The soft morphological erosion of f by [B, A, k ] is

(f 8 [ B , A, k ] ) ( z ) = kth smallest of multiset

{kof (a) 1 a E A,} U { f ( b ) 1 b E (B \A)z j . (2)

Note that when k = 1, the soft morphological operations are equivalent to the standard morphological operations. Since soft morphological filters adopt order statistics [2] and mathematical morphology [4], [IO], [I l l , they can be viewed as weighted order statistic filters that apply set union operation and more weights are given to the pixels in the hard center than to the pixels in the soft boundary. In general, recursive structures usually provide better smoothing capabilities and take less computational time even though at the expense of increased detailed distortion [51, [17].

Recursive filters are the filters that use previously filtered outputs as their inputs. Let zz and yz denote the input and output values at location i , respectively, where i = ( 0 , l . . . . , N - l}. Let the domain of the structuring element be (-L,...,-l,O,l,...,R) , where L is the left margin and R is the right margin. Hence, the structuring element has the size of L + R + 1. Start up and end effects are accounted for by appending L samples to the beginning and R samples to the end of the signal sequence. The L appended samples are given the value of the first signal sample; similarly, the R appended samples receive the value of the last sample of the signal.

Definition 3: The recursive counterpart of a nonrecursive filter Q given by

(3) y2 = 9 ( 2 , - ~ , . . . , 2 * - 1 , 2 2 , 2 , + 1 ....,x~+ ~)

is defined as

yz = *(Yz-L,’..,Yz-l,22,~~+I,...,~*+R (4)

by assuming that the values of y z - ~ , . . . , yt- l are already given.

ratio denoted SNR is used. Figure l(b) has SNR = 40.83. Fig. l(c) shows the result of applying a recursive soft morphological dilation with B = ((O,-l),(-l,O),(O,O),(l,O),~O,l~), A = ( (O.O)) , and k = 3, where SNR = 64.65. Fig. l(d) shows the result of applying a recursive soft morphological erosion on Fig. l(c) (i.e., closing) with the same structuring elements and rank order, where SNR = 101.45.

Fig. 2(a) shows a 512 x 432 gray-level image, and Fig. 2(b) shows the same image but with added Gaussian noise having zero mean and standard deviation 20. Fig. 2(b) has S N R = 35.54. Fig. 2(c) shows the result of applying a standard morphological closing with a structuring element of size 3 x 3, where S N R = 25.15. Fig. 2(d) shows the result of applying a recursive soft morphological closing with B = ((0, -l), (-l,O), (0,0), ( l ,O), (0. l)), A = ( (O,O)) , and k = 3, where SNR = 69.41.

111. PROPERTIES OF RECURSIVE SOFT MORPHOLOGICAL FILTERS

Definition 4: A filter Q is said to be idempotent if Q( *(f)) = Q(f) for any input signal f.

Definition 5: A filter Q is said to be extensive if 9 ( f ( 2)) 2 f (z) for every z. Otherwise, if 9 ( f (z ) ) 5 f(x), the filter is said to be anti-extensive.

Definition 6: A filter Q is said to be increasing if for any two input signals f and g , such that f(z) 5 g(z) for every z, the resultant outputs satisfy the relationship 9 ( f (z ) ) 5 Q(g(z)).

We prove that recursive soft morphological filters are increasing by first proving that if a soft morphological filter is increasing, then its recursive counterpart is also increasing.

Theorem 1: If a soft morphological filter is increasing, then the recursive soft morphological filter is also increasing.

Pro08 Let two input signals be f(z) = { Z O , ~ : . . , X , \ - I }

and g ( d ) = { 5;. x; , . . . , zh - ) and have the ordering relation of f(x) 5 g(z’) for every 1c and d . We need to prove that for every yZ and YI

Y Z = q ( Y Z - L , . .. , Y t - 1 , Z,, Z t + l r ’ ’ ’ , Z Z + R ) I I I I *(Y:- L , . ’ . , Y z - - l r T z , Z ~ + l r . . . . ~ . I + R ) = Y:.

That is, yt 5 y: for every i . We prove the theorem by induction. Since the recursive and nonrecursive filters of yo and yh only depend upon the values of the input pixels { 20, . . . , z ~ } and {zh, . . . , &}, respectively, which have the ordering of * ( f (z)) 5 9 (g (2’ )), the initial condition of i = 0 is satisfied. Assume that the condition is true for i = L - 1. That is the output values at locations { 0, . . . , L - 1) satisfy the condition. Now for i = L, we have

Y L = *(yo,. . . . yL-1, Z L , 2 L + 1 . ... , S L + R ) I I I 5 , YL-1 , X L , X L + l > . . .,&+RI = YL

simply because the output values at locations (0.. . . . L - l} and the input values at locations {L, . . . , L + R} both satisfy the condition. This implies that the recursive counterpart of * also has the increasing property. 0

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 4, NO. 7, JULY 1995 1029

Fig. 1 . soft morphological erosion (i.e., closing)

(a) Original image; (b) image corrupted by Gaussian noise; (c) result of a recursive soft morphological dilation; (d) followed by a recursive

Based on the above theorem and the fact that the soft morphological dilation and erosion are increasing [6], [7], [E], we conclude that the recursive soft morphological dilation and erosion are also increasing.

Theorem 2: Recursive soft morphological dilation and erosion are increasing.

Theorem 3: Recursive soft morphological dilation is extensive, and recursive soft morphological erosion is anti-extensive.

Pro08 According to the definition of soft morphological dilation, the values of the multiset { k o f ( a ) I a E A,}U{f(b) I b E (B\A),} are sorted in the descendent order and the kth largest is selected. If f(z) is the maximum value in the set B , it is selected as the output after the repetition k times. If f(z) is not the maximum value in B,

the selected kth largest value must be greater than or equal to f(z) after f(z) is repeated k times. This implies that for every z, soft morphological dilation is extensive. We can similarly derive that the

0 From the definition of recursive soft morphological dilation and the

properties of scalar multiplication, the result obtained by multiplying the dilated value of the input signal by a positive constant is equal to the result obtained by initially performing a recursive soft morphological dilation and then multiplying it by a positive number. The proof is straight-forward and therefore is skipped.

Property 1: Recursive soft morphological filters are scaling- invariant.

recursive soft morphological erosion is anti-extensive.

1030 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 4, NO. 7, JULY 1995

Fig. 2. morphological closing.

(a) Original image; (b) image corrupted by Gaussian noise; (c) result of a standard morphological closing; (d) result of a recursive soft

Iv . IDEMPOTENT RECURSIVE SOFT MORPHOLOGICAL FILTERS

An idempotent filter in Definition 4 maps an arbitrary input signal into an associated set of root sequences. Each of these root signals is invariant to additional filter passes. The standard morphological opening and closing have the property of idempotency. In contrast, recursive soft morphological opening and closing are not idempotent in the general case. However, when the filter is designed in a specific way, recursive soft morphological opening and closing are also idempotent. We will present the idempotent recursive soft morphological filters in one dimension for simplicity. Two or more dimensional filters can be similarly derived.

If B = (-n.-n + 1 : ~ ~ , - 1 , 0 . 1 ; ~ ~ , r i - 1.n) and A = (-n + 1:. . , -1,O. 1.. . . . n - l), where n 2 1, then we denote the structuring element [ B , A. k] by [n. n - 1, k].

Property 2: Recursive soft morphological dilation and erosion are idempotent for the structuring element: B of length three, A the central point, and k = 2. That is the structuring element is [l. 0.21.

Proof: Let f( .r) = { a h c d e}, where a. b , c, d. and e are arbitrary numbers. The number of ordering relationships of the five variables is 5! = 120. By applying a recursive soft morphological dilation on f by [l, 0, 21 gives the same result as we apply the dilation again on the result. The same property holds true for any size o f f Similarly,

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL 4. NO 7, JlJ1.Y 1995 1031

the procedures can be applied to the recursive soft morphological 0

The proof of Property 3 can be derived similarly and is therefore skipped.

Property 3: Recursive soft morphological closing and opening by [ I L , n - 1, k] are idempotent, where k = 1 or 2.

Corollary I : If the kth largest (smallest) value selected during the scanning of the first pixel of the input signal for a recursive soft morphological dilation (erosion) happens to be the maximum (minimum) positive impulse in the input signal, then the filter is idempotent.

erosion and yield the same property.

v. CASCADED RECURSIVE SOFT MORPHOLUGICL FILTERS Cascaded weighted median filters were introduced by Yli-Harja et

al. [ 181 in which several interesting properties of weighted cascaded filters were discussed. We now present some properties of cascaded recursive soft morphological filters. By cascade connection of filters F and G, we mean that the original input signal f is filtered by the filter F to produce an intermediate signal g. Then, g is filtered by the filter G to produce the output signal h. Cascaded filters F and G can also be presented as a single filter H which produces the output h directly from the input f . We now discuss the properties of cascaded recursive soft morphological filters.

Property 4: The cascaded recursive soft morphological filters are not commutative.

Property 5: The cascaded recursive soft morphological filters are associative.

Note: The output of recursive soft morphological filters depends upon two factors, namely, the order index k and the length of the hard center. Although the technique to combine any two cascaded filters F and G into a single filter H is the same, no direct formula can be given to produce the combined filter H for different cascade combinations. Due to the many variations present, we can only present the essential idea in constructing the combined filter and illustrate it on some examples.

Given two filters [ B I , A I , k] and [ B z . .Az, k ] of length five and a hard center of length three, where k 2 2. Let B1 = I32 = (-2 - 1 0 1 2) and A1 = -42 = (-1 0 1). Denote the input signal f to be {s~,.?:l,.....r,~-~}, the output of the first filter to be {yo. y1, . . . , y ~ - 1 1, and the output of the second filter to be { . Z O . Z ~ ; . . , Z N - ~ } . Since k 2 2, we have yo = kth largest of multiset { k o T O , k o T I } , where the multiset need not consider 22. It means that yo depends upon (LO, r1 ). Again, yl = kth largest of multiset { k o yo. k o 2 1 . k o L Z } . It means that yl depends upon ( y 0 , z l . T Z ) . Proceeding in this way, we obtain y L to be yz = kth largest of multiset { k o ~ ~ - 1 , k o s,, k o s,+]}. It means that y, depends upon ( y z - - l r ~ z , ~ l + ~ ) .

According to the definition of cascaded filters, the intermediate output {yo, y1. . . . , yt , . . . , yn; - 1 } is used as the input further processed by the filter [&, Az, k ] . Similarly, we obtain the final output 2, to be zL = kth largest of multiset { z ~ - - I . ~ ~ , ~ ~ + I } . It means that 2 , depends upon (zt--1,zZ, sz+l, z,+z). The above derivation states that the result of the cascade combination of two aforementioned filters can be equivalently obtained by a single recursive standard morphological filter of (- 1 0 1 2). Other examples of reducing the cascaded filters into a single recursive standard morphological filter can be similarly derived.

Note that the number of computations required also reduces significantly when we use the combined filter obained by the above result. For example, if we have two filters of size five each having a hard center of length three and the order index k , then the number of computations required for an input signal of length N is equal to

TABLE I DIFFERENT CASCADE COMBINATION AND THEIR COMBINED FILTERS

(FOR I; 2 2)

N k ( 5 k + 3 ) , whereas according to Table I in the case of the combined filter of size four, we have only 3 N .

The following property provides the reduction for the cascaded filters when k = 1, which are equivalent to standard morphological filters. Because of the page limit, the proof is omitted.

Property 6: If the cascade combination of two filters of any lengths, denoted by BI = ( - L I S . . - 1 0 l . . . R i ) , BZ = ( - L z . . . - 1 0 l . . . R ; ? ) and any A1 and Az, and the order index k = 1, their combined recursive standard morphological filter will be (-1 0 l . . . ( R 1 + Rz) ) .

VI. CONCLUSION

In this correspondence, several properties of recursive soft morphological filters have been presented. Similar to the cascade combination of median filters, the cascade combination of recursive soft morphological filters can be combined into a single recursive standard morphological filter with a smaller window size. This crucial property reduces the number of computations significantly. A new class of idempotent recursive soft morphological filters has also been introduced.

REFERENCES

[ l ] T. R. Crimmins and W. M. Brown, “Image algebra and automatic shape recognition,” IEEE Trans. Aerosp. Electron. Syst., vol. AES-21, pp. 6049 , Jan. 1985.

[2] H. David, Order Statistics. New York Wiley, 1981. [3] E. Dougherty and R. Haralick, “The logical context of nonlinear

filtering,” in Proc. SPIE Symp. Image Algebra Morphological Image Processing, vol. 1658, 1992, pp. 234-244.

[4] C. R. Giardina and E. R. Dougherty, Morphological Methods in Image and Signal Processing.

[5] R. M. Haralick and L. G. Shapiro, Computer and Robot Vision, vol. 1. Reading, MA: Addison-Wesley, 1992, pp. 157-26.

[6] L. Koskinen, J. Astola, and Y. Neuvo, “Soft morphological filters,” in Proc. SPIE Symp. Image Algebra Morphological Image Processing, vol. 1568, 1991, pp. 262-270.

[7] L. Koskinen and J. Astola, ”Statistical properties of soft morphological filters,” in Proc. SPIE Symp. Nonlinear Image Processing, vol. 1658, 1992, pp. 25-36.

[8] P. Maragos and R. Schafer, “Morphological filters-Part II: Their relations to median, order-statistics, and stack filters,” IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-35, pp. 117Cb1184, Aug. 1987.

[9] P. Maragos and R. Ziff, “Threshold superposition in morphological image analysis systems,’’ IEEE Trans. Pattern Anal. Machine Intell., vol. 12, no. 5, pp. 498-504, May 1990.

London: Aca- demic, 1982.

London: Academic, 1988.

Englewood Cliffs, NJ: Prentice-Hall, 1988.

[ 101 J. Serra, Image Analysis and Mathematical Morphology.

[ 111 -, Image Analysis and Mathematical Morphology, vol. 2.

F. Y. Shih and 0. R. Mitchell, “Industrial parts recognition and inspection by image morphology,” in Proc. IEEE In?. Con$ Robotics Automation, Philadelphia, PA, Apr. 1988, pp. 1764-1766. -, “Automated fast recognition and location of arbitrarily shaped objects by image morphology,” in Proc. IEEE Con$ Comput. Vision Pattern Recognition, Ann Arbor, MI, June 1988, pp. 774-779. -, “Threshold decomposition of grayscale morphology into binary morphology,” IEEE Trans. Pattern Anal. Machine Intell., vol. 11, no. 1, pp. 3142, Jan. 1989. F. Y. Shih and C. C. Pu, “Threshold decomposition of soft morphological filters,” in Proc. IEEE Conf: Comput. Vision Pattern Recognition, New York City, NY, June 1993, pp. 672-673. S. R. Stemberg, “Biomedical image processing,” Computer, vol. 16, no.

S. R. Stemberg and E. S. Stemberg, “Industrial inspection by morphological virtual gauging,” in IEEE Workshop Comput. Architecture Pattern Anal. Image Database Management, Oct. 1983, pp. 237-247. 0. Yli-Harja, J. Astola, and Y. Neuvo, “Analysis of the properties of median and weighted median filters using threshold logic and stack filter representation,” IEEE Trans. Signal Processing, vol. 39, no. 2, pp. 395410, 1991.

1, pp. 22-34, Jan. 1983.

Adaptive Postprocessing Algorithms for Low Bit Rate Video Signals

Tsann-Shyong Liu and Nikil Jayant

A h f r a t - A new adaptive postprocessing algorithm to enhance the quality of a noisy video sequence is presented. The algorithm recognizes that the visibility of noise depends on local signal characteristics. It therefore classifies the video signal into different classes and uses separate nonlinear filters matched to each class. The most general version of the algorithm employs motion-compensated frame averaging to improve picture quality in a first stage. A classification algorithm subsequently divides subblocks of pixels in the averaged frame into four classes: edge, smooth, nonsmooth with motion and nonsmooth without motion. Spatial algorithms that perform multilevel median mering, double median wer ing , and median mering are used for pixels belonging to edge, smooth, and nonsmooth with motion categories. Pigels in the nonsmooth, unmoving category are left unfiltered to preserve corresponding image texture. In a simpler version of this four-class system, the motion cues and motion-compensated frame averaging are eliminated, and the purely spatial filtering is based on a three-class algorithm. When used at the output of a 3-D subband coder at 384 kbps, the spatial postfilter was shown to provide a consistent gain in subjectively evaluated picture quality. ’benty-five viewers participated in an experiment involving three coded sequences. In a painvise comparison of postfiltered and unfiltered sequences, the postfiltered version was judged to be better in 63 out of 75 instances.

I. INTRODUCTION The importance of good quality for low-bit rate video coding is

growing with the increasing use of video over existing and future networks including ISDN. Noise introduced in the video sequence at

Manuscript received August 8, 1993; revised August 23, 1994. The associate editor coordinating the review of this paper and approving it for publication was Prof. Nikolas P. Galatsanos.

T.-S. Liu is with the Institute of Computer Science, National Tsing Hua University, Hsinchu, Taiwan, R.O.C. He is also with the Telecommunication Labs., Chung-Li, Taiwan, R.O.C.

N. Jayant is with AT&T Bell Laboratories, Signal Processing Research Department, Murray Hill, NJ 07974 USA.

IEEE Log Number 941 1866.

the receiver owing to quantization error is inevitable, especially in low bit rate video coding (384 kbps or less). In order to reduce the noise, postprocessing of the reconstructed video sequence is useful and particularly appropriate for current low bit rate algorithms that are known to result in inadequate picture quality. This correspondence proposes a spatio-temporal postfilter with separable spatial and temporal components. Informal subjective testing has shown the benefits of both the full-fledged algorithm and the somewhat less powerful, purely spatial version. Formal subjective testing has quantified the benefits of the spatial postfilter.

A. Spatio-Temporal Postjiltering

The intensities of the pixels making up a video sequence are spatially and temporally correlated. Temporal correlation is very important for interframe coding, where temporal filtering is frequently used. The simplest and most straightforward temporal filter is a frame-averaging algorithm. However, this tends to degrade moving objects. Hence, motion compensation is combined with the temporal filter to improve the quality of video sequence. In [l] and [2], a motion estimation algorithm is applied to a noisy image sequence to estimate the motion trajectories, i.e., locations of the pixels that correspond to each other at a predetermined number of successive image frames. Then, the value of a particular pixel at a certain frame is estimated using the noisy image sequence values that are on the motion trajectory transversing that pixel. The algorithm segments the video into moving and stationary components. Then, an adaptive temporal filter is applied. In a real scene, motion can be a complex combination of translation and rotation. Such motion is difficult to estimate and may require a large amount of processing. In [3] and [4], the motion compensation is formed by using block matching. Subsequently, frame averaging with motion compensation is applied in [3], and median filtering with motion compensation is applied in

Since some artifacts such as blocking and contouring last for a few frames temporally, it is very difficult to reduce those artifacts by using only a temporal filter. In [5 ] , a spatial filter to reduce staircase effects is presented. The algorithm in [5] uses edge detection to classify subblocks of pixels into two classes: edge and nonedge. Subsequently, the median filter and the so-called D-filter [6] are applied to edge and non-edge pixels, respectively.

Based on the above observations, this correspondence combines temporal and spatial filters to reduce noise and artifacts in video coding, with particular focus on 3-D subband coding. A simple but effective subsystem of the saptio-temporal filter is a purely spatial postfilter.

141.

B. 3-0 Subband Coding

Subband coding was first introduced to code speech signals [7]. It has since been used to code images, initially in 2-D [8], [9] and later in 3-D [lo]-[12]. The subband coding concept is based on the decomposition of the image into different frequency subbands and the coding of each subband separately according to its statistics. The advantages of such an approach are the confinement of coding errors to individual subbands (if the quantization of subband signals is fine enough), and the noise spectrum shaping due to varying bit assignment in the subbands. A very effective 2-D subband coder for still image compression has been developed based on perceptual modeling [13]. One example of a 3-D subband coder for full motion video is that presented in [ l l ] , [12], and [18]. The coder begins

1057-7149/95$04.00 0 1995 IEEE

Documents

Recursive soft morphological filters