Unequal-arm Adaptive Rood Pattern Search withEarly Terminations For Fast Block-matching

Motion Estimation on H.264Bin Li and Kai-Kuang Ma*

School of Electrical and Electronic EngineeringNanyang Technological University

Singapore*ekkma@ntu.edu.sg

Abstract— Based on a rood-shaped pattern of the motion-vector displacement distribution inherently resulted fromblock-matching motion estimation, a series of adaptive roodpattern search (ARPS) algorithms have been introduced.Among them, the unequal-arm ARPS has achieved superiorperformance on many accounts while maintaining a fairly closerate-distortion performance compared with that of the fullsearch. In this paper, an enhanced version of the unequal-armARPS is introduced by incorporating early terminations atboth the initial search and the refined local search stages.We compare its performance with that of the UMHexagonSfast block-matching motion estimation algorithm based on theJM-9.3 software encoder. The experimental results show thatthe unequal-arm ARPS with early terminations has yieldedsignificant search efficiency gain while consistently maintaininga fairly close rate-distortion performance compared with thatof the UMHexagonS. Furthermore, the proposed algorithm isfairly simple on implementation.

Keywords—H.264, block-matching motion estimation, fastsearch, unequal-arm adaptive rood pattern search, earlytermination.

I. INTRODUCTION

For all the video coding standards, block-matching motionestimation (BMME) is exploited to remove temporal redundancyamong the adjacent video frames. It is the most time-consumingprocess in the encoder if exhaustive search is applied due to largenumber of blocks and search points required to be evaluated. FastBMME algorithm is developed to reduce computational complex-ity significantly while achieving close rate-distortion performancecompared with that of the full search. The key philosophy ofdeveloping a practical BMME algorithm is to strike an optimumbalance among multiple considerations—computational gain (orspeed-up ratio), coding efficiency, video quality, and implementa-tion cost/complexity; the optimal trade-off intimately depends onthe application’s objectives and requirements.

Among the existing fast BMME algorithms, the diamond search(DS) [1] and the motion vector field adaptive search technique(MVFAST) [2] have been adopted by the MPEG-4 standard in[3] and [4], respectively. As an enhanced version of the DS, theadaptive rood pattern search (ARPS) [5] has been proposed basedon the DS and the star search (SS) [6]. In the ARPS, a noveladaptive rood pattern (ARP) is exploited at the initial searchstage on each macroblock search, followed by using a unit-sizerood pattern (URP) (i.e., the small diamond search pattern, SDSP)repeatedly at the refinement (or local) search stage until the motion

vector of the current macroblock is found. In [5], the ARPS hasbeen implemented and incorporated into the MPEG-4 VerificationModel (VM) encoder for demonstrating its superior performance tothat of the DS on both computational gain (two times, generally)and the peak-signal-to-noise ratio (PSNR). By further relaxing therood’s arm lengths from equal to unequal for a better adaptionof video contents, the unequal-arm ARPS algorithm [7] has beenproposed and shown further performance gain.

With the release of the H.264 JM-9.3 software encoder [8], it istimely to make a performance comparison between the unequal-armARPS algorithm and the UMHexagonS [9] algorithm. Furthermore,early terminations are incorporated to enhance the performance ofthe unequal-arm ARPS.

II. THE H.264 MOTION ESTIMATION FRAMEWORK

Compared with the existing video coding standards, H.264provides more varieties of block sizes (denoted as modes) (16×16,8 × 16, 16 × 8, 8 × 8, 8 × 4, 4 × 8 to 4 × 4 pixels) as wellas the so-called multi-frame motion-compensated prediction [10]to achieve better coding efficiency. Furthermore, in the currentH.264 JM software encoder, the rate-constrained motion estimationis utilized, where the criterion to find the optimum motion vectoris to minimize a Lagrangian cost function:

J(m, λM ) = SAD(s, c(m)) + λM × R(m − p) (1)

where m = (mx, my)T is the current motion vector (MV),p = (px, py)T is the predicted MV, and λM is the Lagrangianmultiplier. The SAD stands for the sum of absolute difference,which is used for the distortion measurement, and the rate termR(m − p) represents the number of bits required to encode thedifference between the current MV m and the predicted MV p.

Instead of exploiting the SAD as the distortion measurementmetrics for fast BMA, the Lagrangian cost functional J(m, λM )is used as the yardstick to find the optimum MV in H.264. This isbeneficial to the very low bitrate video compression cases, wherethe MVs occupy a significant portion of the compressed data.Since the Lagrangian formulation can be viewed as imposing asmoothness constraint to regularize the motion displacement field[11], the value of R(m − p) would be directly affected by thedistribution of the MVs and could yield noticeable video qualitychanges.

III. A BRIEF REVIEW OF THE UNEQUAL-ARM ARPS

A. Prediction of the target MVIn H.264, two kinds of prediction methods are introduced for

conducting the differential coding for the MVs: the median predic-tion and the directional segmentation prediction [12] as shown in

402

1-4244-0387-1/06/$20.00 c©2006 IEEE

E

D B C

A

(a)

8 16 16 8x x

(b)

Fig. 1. Motion vector prediction methods. (a) Median prediction; (b)Directional segmentation prediction.

Fig. 1. Generally speaking, the latter is used when the block sizeis 16× 8 or 8× 16; otherwise, the median prediction will be used.

B. Unequal-arm ARP for the initial search

Predic ted MV2

-4

1

-3

-5

-6-1 0 3 4 5 6 721

-2

-1

0

M V1

M V5

M V4

M V3

M V2 M V0

Fig. 2. The unequal-arm adaptive rood pattern

The unequal-arm ARP employs the search pattern as illustratedin Fig. 2. The center point of the unequal-arm ARP is placed onthe predicted position, because the true motion vector has a highcorrelation with the predicted MV.

Among the MVs available from the neighboring blocks, themaximum and the minimum values of MVs can be considered asconservative measurement of the amount of deviations from thepredicted MV. Therefore, they can be used as a reliable estimationof the arm length to represent the dynamic range in their respectivedirections. That is, the coordinates of the four motion vectors are:

−−→MV 1 = [max(MVx), MVpredicted y]−−→MV 2 = [min(MVx), MVpredicted y]−−→MV 3 = [MVpredicted x, max(MVy)]−−→MV 4 = [MVpredicted x, min(MVy)]

(2)

where MVx and MVy are the horizontal and vertical componentsof all the MVs in the neighboring blocks, respectively. Operators“max” and “min” are used to find the maximum and the minimumvalues among all the available MVs in the neighboring blocks,respectively. For the robustness of the algorithm, the [0, 0] point isalso included in the unequal-arm ARP.

C. URP for the refined local searchWith the new search center obtained in the initial search stage,

a small search pattern is applied to refine local search. For that, theunit-size rood pattern (URP) [5] (i.e., the small diamond searchpattern of DS [1] as shown in Fig. 3) is also used in this paper.Note that the ARP could be shrunk to the URP, if all four armlengths are less than 2, or even to the center point itself, if alllengths are 0.

4

-2

0

1

2

3

-1

-3

-4-4 -3 -2 -1 0 1 2 3 4

Fig. 3. The unit-size rood pattern (URP) [5] (i.e., the small diamond searchpattern (SDSP) as used in the diamond search [1]).

D. Summary of the unequal-arm ARPS search algorithm• Step 1: [Prediction] The median prediction based on the

neighboring blocks as shown in Fig. 1 is performed for findingthe predicted MV.

• Step 2: [Initial Search] Place the center of the unequal-armARP on the position of the predicted MV found in the Step1. In the unequal-arm ARP, the locations of the four vertexpoints are calculated by (2). Compute the J(m, λM ) of allfour vertex points of the unequal-arm ARP. (This step isexecuted only once.)

• Step 3: [Refined Local Search] Place the center point of theURP on the position that incurs the minimum J(m, λM )found in the Step 2 and evaluate J(m, λM ) for each point. Ifthe new minimum J(m, λM ) is not incurred at the center ofthe current URP, repeat this step; otherwise, the MV is foundas the minimum matching error (MME) point.

IV. ENHANCED VERSION WITH EARLY TERMINATION

To further reduce unnecessary search, the threshold-based earlytermination (ET) (another term, zero motion prejudgement, ZMP)has been well proven as an effective methodology, especially for“talking-head” type of sequences with large still background. It hasbeen widely exploited in fast motion estimation search algorithms(e.g., adaptive irregular pattern search (AIPS) with ZMP [13]). Inthis paper, the threshold-based ET or ZMP is incorporated into theunequal-arm ARPS for enhancing its performance; for that, we haveobserved two search stages, where the use of ET could be highlyeffective in obtaining additional efficiency gain — one at the initialsearch stage and the other at the refined local search stage, withdetails as follows.

A. Early termination at the initial searchFirst, note that ET or ZMP only takes effect for (nearly)

stationary blocks; that is, for those blocks with zero motion ora very small motion (below the imposed threshold). For the large-magnituded motion block (i.e., above the threshold), the ET wouldbe effective provided that it can be applied at the position of thepredicted MV as the origin point. This is due to the fact that thepredicted MV tends to be very close, if not totally identical, to thefinal MV found. Hence, using the predicted MV as the origin point,it has an effect of eliminating the non-stationarity of block motion.

B. Early termination at the refined local searchA refined local search should be considered unnecessary, if

the block-matching residuals resulted are already quite small.Especially for almost uniform search regions, the local search mayconsume a significant amount of computational power for gaining

APCCAS 2006 403

negligible PSNR enhancement. Therefore, it would be beneficial toinvoke early termination at the refined local search.

C. Mode adaptive thresholding

Due to the various block sizes (so-called modes) in H.264, theblock dimensions (block size x in the horizontal direction andblock size y in the vertical direction) are involved in the thresholdsetting to make it mode adaptive for both the initial search and therefined local search. From our experiment, the PSNR drop will beless than 0.01 dB, if the following threshold is used at both initialsearch and refined local search:

Threshold = 2 × block size x × block size y (3)

D. Summary of the unequal-arm ARPS with early termina-tions

• Step 1: [Prediction (unchanged)] The median prediction basedon the neighboring blocks as shown in Fig. 1 is performedfor finding the predicted MV.

• Step 2: [Early Termination for the Initial Search] Calculatethe Lagrangian cost function at the location of the predictedMV and perform threshold-based early termination. If thetermination condition is satisfied, the MV search will beterminated.

• Step 3: [Initial Search (unchanged)] Place the center of theunequal-arm ARP on the position of the predicted MV foundin the Step 1. In the unequal-arm ARP, the locations ofthe four vertex points are calculated by (2). Compute theJ(m, λM ) of all four vertex points of the unequal-arm ARP.(This step is executed only once.)

• Step 4: [Early Termination for the Refined Local Search]Check position that incurs the minimum J(m, λM ) found inthe Step 3 with threshold-based early termination condition.If the termination condition is satisfied, the MV search willbe terminated.

• Step 5: [Refined Local Search (unchanged)] Place the centerpoint of URP on the position that incurs the minimumJ(m, λM ) found in the Step 3 and evaluate J(m, λM ) foreach point. If the new minimum J(m, λM ) is not incurredat the center of the current URP, go back to the Step 4;otherwise, the MV is found as the minimum matching error(MME) point.

V. EXPERIMENTS AND RESULTS

The unequal-arm ARPS algorithm and the unequal-arm ARPSwith early terminations algorithm are implemented on theH.264 JM-9.3 software encoder platform to demonstrate theirperformance. The experimental results are compared with theUMHexagonS in terms of luminance PSNR measurement (in dB),bitrate, normalized computational cost (NCC), and motion estima-tion execution time.

A. Test conditions

The experimental conditions specified as follows are used toevaluate the performance of the concerned motion estimation al-gorithms. The motion vector search range is 16 pels around thepredicted MV, and 5 reference frames are used for all the testsequences. Sequence format, quantization parameter QP , framerate and the number of frames are listed in Table I. For all thetest sequences, the frames are coded using a GOP size of 12 thatincludes I and P frames only (that is, one I frame followed byeleven P frames). The rate control is disabled, and all the searchmodes are turned on.

TABLE I

VIDEO SEQUENCES FOR TESTING.

Sequence Size QP Frame rate frames

Container QCIF 28 30 fps 300 framesStefan QCIF 28 30 fps 300 frames

Table tennis QCIF 28 30 fps 300 framesHall Monitoring QCIF 28 30 fps 300 frames

Mobile QCIF 28 30 fps 300 framesMother and daughter QCIF 28 30 fps 300 frames

Silent QCIF 28 30 fps 300 framesStefan CIF CIF 28 30 fps 300 frames

B. Normalized computational cost (NCC)Previous MPEG standards have the same block size, called

macroblock (i.e., 16×16), for motion estimation. Due to the variousblock sizes (or modes) and multi-frame-based motion estimation,normalized computational cost (NCC) is proposed in [7] in H.264to have a fair computational gain comparison. For each blocksearch, the total NCC is defined as follows:

NCC =Npts × W (mode)

Nmode × Nref(4)

where Npts denotes the total number of search points, W (mode)denotes the weight of computational cost incurred for the currentblock mode. Nmode is the total number of modes involved (inH.264, Nmode = 7), and Nref is the number of reference frames.Since the computation of SAD is directly proportionate with theblock size and occupies the majority of computational load incomputing J(m, λM ), the computational costs per search point fora 16×16 block is thus defined as one unit cost. Hence, the weightW (mode) becomes the ratio of the area of the current block modeto that of the 16 × 16 mode. That is, for the 16 × 16 mode, theweight is 1; accordingly, W = 1

2for the 16×8 and 8×16 modes,

W = 14

for the 8×8 mode, W = 18

for the 8×4 and 4×8 modes,and W = 1

16for the 4×4 mode. This NCC definition is consistent

with the search point comparison documented in the literature.

C. Motion-estimation execution timeIn the past, the actual motion-estimation execution time was

seldom used for motion estimation comparison due to the issuessuch as machine dependency and program optimization. However,if we consider the software-based applications, the execution time isactually the most relevant indicator that engineers want to referencefor block-matching algorithm comparison. The actual executiontime also gives a direct indication regarding the block-matchingalgorithm complexity. Based on our simulation experiments, thespeed variations from the same search algorithm on the same PCwithout much interference from other softwares (i.e., disconnectingto network and closing all application processes) is only around2% (due to interferences from system processes). Therefore, theactual execution time is quite informative to be used as a referencefor block-matching algorithm comparison using the same PC withthe same compiler settings (release mode with optimization inMicrosoft Visual C++ 6.0).

D. Experimental resultsThe experimental results are shown in Tables II, III, IV and V on

the aspects of PSNR (in dB), bitrate, NCC and the actual motion-estimation execution time, respectively. The relative improvementsare shown inside the parenthesis in the respective column for theease of comparison. Based on these results as shown, it can beconcluded that the unequal-arm ARPS with early terminations hasachieved appreciable performance improvement (mainly for “talk-ing head” types of sequences) over the unequal-arm ARPS (without

404 APCCAS 2006

early terminations) and substantial performance improvement overthe UMHexagonS, thus, lower power consumption and cost.

VI. CONCLUSION

The unequal-arm ARPS [7] and the proposed unequal-armARPS with early termination are implemented on the H.264 JM9.3 software encoder. Both of them show superior performancein terms of normalized computational cost and motion-estimationexecution time compared with that of the UMHexagonS algorithm.In particular, experimental results show that the unequal-arm ARPSwith early terminations has achieved 2.62 times (on average, andconsistently more than 2.4 times) of the actual execution timespeed-up compared with that of the UMHexagonS algorithm whilemaintaining about the same PSNR and bitrate. Furthermore, thealgorithmic complexity of the proposed algorithm is much lowerthan that of the UMHexagonS.

REFERENCES

[1] S. Zhu and K.-K. Ma, “A new diamond search algorithm for fast block-matching motion estimation,” IEEE Trans. on Image Processing, Vol.9, no. 2, pp. 287-290, Feb. 2000.

[2] K.-K. Ma and P. I. Hosur, “Core experiment results of fast mo-tion estimation based on new test conditions (Q4a),” ISO/IECJTC1/SC29/WG11, M4934, Vancouver, Canada, Jul. 1999.

[3] ISO/IEC JTC1/SC29/WG11, “MPEG-4 video verification model,” ver-sion 14.0, N2932, pp. 301-303, Oct. 1999.

[4] ISO/IEC JTC1/SC29/WG11, “Text of ISO/IEC 14496-7 CD registrationand consideration (Optimized Code for MPEG-4 Visual Standards),”N3325, Noordwijkerout, Netherlands, Mar. 2000.

[5] Y. Nie and K.-K. Ma, “Adaptive rood pattern search for fast block-matching motion estimation,” IEEE Trans. on Image Processing, Vol.11, Issue 12, Dec. 2002.

[6] S. Zhu and K.-K. Ma, “A new star search algorithm for fast block-matching motion estimation,” Workshop on Very Low Bitrate Coding(VLBV), pp. 173-176, Oct. 8-9, 1998.

[7] K.-K. Ma and G. Qiu, “Unequal-arm adaptive rood pattern search forfast block-matching motion estimation in the JVT/H.26L ,” Proceedingsof the International Conference on Image Processing, Vol. 1, 14-17,Sept. 2003.

[8] H.264 Joint Model 9.3 (JM-9.3), available from ftp://imtc.org/jvt-exports/reference software/

[9] Zhibo Chen, Peng Zhou and Yun He, “Fast integer pel and fractionalpel motion estimation in for JVT”, JVT-F017r1.doc, Joint Video Team(JVT) of ISO/IEC MPEG & ITU-T VCEG, 6th meeting, Awaji, Island,Japan, 5-13 December, 2002.

[10] T. Wiegand and B. Girod. Multi-frame motion-compensation predic-tion for video transmission, Kluwer Academic Publishers, USA, 2001.

[11] G. J. Sullivan and T. Wiegand, “Rate-distortion optimization for videocompression,” IEEE Signal Processing Magazine, Nov. 1998.

[12] Joint Video Team (JVT) of ISO/IEC MPEG and ITU-T VCEG, “Jointfinal committee draft (JFCD) of joint video specification (ITU-T Rec.H.264—ISO/IEC 14496-10 AVC),” JVT-D157, Klagenfurt, Austria, 22-26 July 2002.

[13] Y. Nie and K.-K. Ma, “Adaptive irregular pattern search withmatching prejudgment for fast block-matching motion estimation,”IEEE Trans. on Circuits and Systems for Video Technology, Vol. 15,Issue 6, Jun. 2005.

TABLE II

AVERAGE PSNR (DB) OF LUMINANCE COMPONENT.

Unequal-arm Unequal-armSequence UMHexagonS ARPS ARPS+ET

Container 36.590 36.586 (-0.004) 36.587 (-0.003)Stefan 34.561 34.525 (-0.036) 34.517 (-0.044)

Table Tennis 36.423 36.411 (-0.012) 36.408 (-0.015)Hall Monitoring 37.858 37.860 (+0.002) 37.861 (+0.003)

Mobile 33.539 33.533 (-0.006) 33.534 (-0.005)Mother & daughter 37.916 37.917 (+0.001) 37.917 (+0.001)

Silent 36.333 36.327 (-0.006) 36.332 (-0.001)Stefan CIF 35.784 35.740 (-0.044) 35.732 (-0.052)

TABLE III

AVERAGE TOTAL BITS PER P FRAME.

Unequal-arm Unequal-armSequence UMHexagonS ARPS ARPS+ET

Container 915.8 919.2 (+0.37%) 915.20 (-0.06%)Stefan 12175 12619 (+3.65%) 12618 (+3.64%)

Table Tennis 3810.3 3898.1 (+2.30%) 3893.8 (+2.19%)Hall Monitoring 1556.1 1554.6 (-0.10%) 1556.0 (-0.01%)

Mobile 10950 10939 (-0.10%) 10964 (+0.13%)Mother & daughter 1209.1 1210.5 (+0.12%) 1212.2 (+0.25%)

Silent 2467.3 2505.1 (+1.53%) 2504.6 (+1.51%)Stefan CIF 38387 41132 (+7.15%) 41299 (+7.59%)

TABLE IV

AVERAGE NORMALIZED COMPUTATIONAL COST (NCC) FOR EACH

BLOCK-MATCHING.

Unequal-arm Unequal-armSequence UMHexagonS ARPS ARPS+ET

Container 88.8385 5.2419 (16.95) 4.4340 (20.04)Stefan 86.6164 10.2938 (8.41) 9.7365 (8.90)

Table Tennis 80.6799 7.6573 (10.54) 7.2710 (11.10)Hall Monitoring 88.8063 5.2736 (16.84) 3.7842 (23.47)

Mobile 89.0937 7.7620 (11.48) 7.7251 (11.50)Mother & Daughter 88.5678 6.0220 (14.71) 4.5672 (19.39)

Silent 88.4431 6.5910 (13.42) 6.1817 (14.31)Stefan CIF 85.6888 11.3487 (7.55) 10.5128 (8.15)

TABLE V

AVERAGE MOTION-ESTIMATION EXECUTION TIME PER P FRAME (MS).

Unequal-arm Unequal-armSequence UMHexagonS ARPS ARPS+ET

Container 362.9 132.0 (2.7501) 128.3 (2.8286)Stefan 414.1 162.7 (2.5450) 156.5 (2.6458)

Table Tennis 377.8 145.6 (2.5956) 146.3 (2.5830)Hall Monitoring 328.2 131.1 (2.5009) 124.9 (2.6284)

Mobile 397.5 148.8 (2.6714) 145.9 (2.7244)Mother & daughter 354.6 144.4 (2.4561) 130.2 (2.7233)

Silent 370.4 147.1 (2.5182) 153.4 (2.4146)Stefan CIF 1622.2 656.3 (2.4719) 663.5 (2.4449)

APCCAS 2006 405