1004 IEEE COMMUNICATIONS LETTERS, VOL. 13, NO. 12, DECEMBER 2009

A Memory-Efficient Parallel String Matching forIntrusion Detection Systems

HyunJin Kim, Hyejeong Hong, Hong-Sik Kim, and Sungho Kang, Member, IEEE

Abstract—As the variety of hazardous packet payload contentsincreases, the intrusion detection system (IDS) should be ableto detect numerous patterns in real time. For this reason,this paper proposes an Aho-Corasick algorithm based parallelstring matching. In order to balance memory usage betweenhomogeneous finite-state machine (FSM) tiles for each stringmatcher, an optimal set of bit position groups is determined.Target patterns are sorted by binary-reflected gray code (BRGC),which reduces bit transitions in patterns mapped onto a stringmatcher. In the evaluations of Snort rules, the proposed stringmatching outperforms the existing bit-split string matching.

Index Terms—Computer network security, finite state ma-chines, site security monitoring, and string matching.

I. INTRODUCTION

BEYOND the generalized firewall in the network layer,IDS is necessary to recognize harmful packet payloads

in the application layer. Due to the variety of maliciouspackets, the number of rules used by IDS has increased.An IDS can have multiple string matchers to detect alltarget patterns from packet payloads. As shown in [1], theautomaton-based approach is common for the string matchingin IDS. Depending on an input symbol, the deterministic finiteautomaton (DFA)-based string matcher provides the determin-istic transition between states. Therefore, the DFA-based stringmatcher improves both regularity and scalability with lowertime complexity [1]. However, the memory requirements areproportional to the numbers of states and input symbols.

In order to reduce the memory requirements for the DFA-based string matching, the bit-split string matching using Aho-Corasick algorithm [2] was proposed in [3]. The bit-splitstring matching partitions target patterns into subgroups witha list of the lexicographically sorted target patterns. For asubgroup of the partitioned target patterns, each string matcherconsists of homogeneous FSM tiles that take a fixed set of bitposition groups as input symbols. In the existing bit-split stringmatching, however, bit transitions between successive patternsmapped onto a string matcher reduce the number of sharedcommon prefixes. In addition, due to the biased bit transitionsfor each bit position group, the memory usage between FSMtiles in a string matcher could be unbalanced.

This paper proposes an algorithm for a parallel bit-splitstring matching using Aho-Corasick algorithm. The bit tran-sitions are reduced using the sorted target patterns basedon BRGC values; therefore, the possibility that the common

Manuscript received December 31, 2008. The associate editor coordinatingthe review of this letter and approving it for publication was M. Ma.

The authors are with the Department of Electrical and ElectronicEngineering, Yonsei University, Seoul, 120-749, Korea (e-mail:hyunjin2.kim@gmail.com, hjhong@soc.yonsei.ac.kr, {hongsik,shkang}@yonsei.ac.kr).

Digital Object Identifier 10.1109/LCOMM.2009.12.082230

prefixes of successive patterns are shared increases. In addi-tion, the proposed algorithm determines the optimal set of bitposition groups in order to map as many patterns as possibleonto each string matcher. As a result, the proposed algorithmreduces the number of adopted DFA-based string matchers,compared with the existing bit-split string matching in [3].

II. PROPOSED PARALLEL STRING MATCHING

A. Architecture of String Matcher

The architecture of the string matcher is based on the stringmatching engine in [3], which is summarized as follows: Ina string matcher, each homogeneous FSM tile takes 𝑛 bits ofone character (or one byte) as an input per cycle. In a stateof each FSM tile, pattern identifications are stored as a partialmatch vector (PMV), where the 𝑖−th bit represents whetherthe 𝑖−th pattern is matched or not in the state. A matchedpattern can be recognized after the logical AND operation ofPMVs in all FSM tiles. Each state in an FSM tile has 2𝑛

pointers for the next state according to 𝑛-bit input. Therefore,the memory size of a string matcher is given by:

8

𝑛× 𝑠× (2𝑛 × ⌈𝑙𝑜𝑔2𝑠⌉+ 𝑝) , (1)

where 𝑠 and 𝑝 are the number of states available in an FSMtile and the size of a PMV, respectively. The values of 𝑠 and 𝑝are predetermined, depending on the structure of the adopteddistributed memory. The main difference of the proposedstring matcher from the string matching engine in [3] is that𝑛 bits for an FSM tile input are selected among the inputbits of one character (eight bits) using eight 8:1 multiplexersto support the bit position grouping. Using the multiplexers,the bit position group for each FSM tile can be selected. Inaddition, 𝑛 input bit positions for each FSM tile are stored toselect input bits for the 8:1 multiplexers.

B. Gray Code-Based Sorting

Target patterns are sorted based on BRGC values to reducebit transitions between successive patterns. For example, itis assumed that there are two string matchers, where eachstring matcher can contain only two patterns among foursingle-character patterns: ‘a,’ ‘b,’ ‘c,’ and ‘d.’ If the sortingis performed lexicographically based on ASCII character codevalues, patterns ‘a’ and ‘b’ and patterns ‘c’ and ‘d’ are mappedonto string matchers, respectively. The total number of bittransitions between patterns ‘a’(01100001) and ‘b’(01100010)and between patterns ‘c’(01100011) and ‘d’(01100100) is five.In the sorting based on BRGC values, there are two partitionsof patterns ‘a’ and ‘c’ and of patterns ‘b’ and ‘d’. In this case,the total number of bit transitions is only three. Let us assumethat there are eight FSMs for each input bit, where each FSM

1089-7798/09$25.00 c⃝ 2009 IEEE

KIM et al.: A MEMORY-EFFICIENT PARALLEL STRING MATCHING FOR INTRUSION DETECTION SYSTEMS 1005

Fig. 1. Example of FSMs according to bit position groups.

contains an initial state. In the sorting based on BRGC values,the total number of states in two string matchers is 35; thetotal number of states is 37 in the lexicographical sorting.The bit transition reduction could increase the possibility thatthe common prefixes of successive patterns are shared. In thiscase, if the states in a small depth from the initial state arenot shared, the states in a large depth cannot be shared. Whenthe character code values in the prefixes of target patternsare not evenly distributed, the effectiveness of the gray code-based sorting is restricted. However, as the number of targetpatterns increases by updating a rule, the prefixes could havemany different character code values, so that the successivecode values can be distributed. The gray code-based sortingincreases the possibility that the prefixes of the target patternsare to be mapped onto a string matcher can be shared or havesuccessive gray code values. Therefore, a memory-efficientstring matching is provided at a rule level.

C. Bit Position Grouping

By determining the optimal set of bit positions groups forFSM tiles, the memory usage between FSM tiles is balancedin a string matcher. The bit position grouping maps as manypatterns as possible under the number of states available inhomogeneous FSM tiles. Therefore, a memory-efficient bit-level string matching can be provided. Let us assume thata string matcher has four FSM tiles with two input bits. Inaddition, “he,” “has,” “his,” and “hers” are assumed to be thepatterns to be mapped. Fig. 1 shows FSMs according to bitposition groups. The number in a circle denotes the indexof a state. A double-circled index indicates an output state,where patterns rounding by braces are matched. For clarity,the failing pointers are not shown. For all string matchers in[3], a set of bit position groups for four FSM tiles is fixed as{(8, 7), (6, 5), (4, 3), (2, 1)}, where the number representsa bit position of one character from the LSB. Fig. 1 (a) and(b) describe FSMs for bit position groups (8, 7) and (4, 3),respectively. Because there are no bit transitions in three MSBsof all patterns, the numbers of states for the bit position groups(8, 7) and (4, 3) are unbalanced. After grouping the MSBpositions with other bits, an optimal set of bit position groupscan be {(8, 4), (7, 3), (6, 5), (2, 1)}. In Fig. 1 (c) and (d), thenumber of states is only seven for bit position groups (8, 4)and (7, 3).

For each string matcher, the proposed bit position grouping

Fig. 2. Pseudocode of the proposed bit position grouping.

searches for the optimal set of bit position groups. Thepseudocode of the bit position grouping for a string matcher isdescribed in Fig. 2. Initially, 𝑝 front patterns from the sortedand unmapped target patterns 𝑢 are considered, where 𝑝 is thesize of a PMV. In this case, 𝜔 is set as 𝑝. In the first step of theouter loop, with 𝜔 front patterns from the target patterns 𝑢,a function Build DFAs generate eight DFAs 𝑑𝑓𝑎𝑠1 for eachbit position. If the maximum number of states in 𝑑𝑓𝑎𝑠1 isgreater than the number of states available in an FSM tile 𝑠,FSMs for multiple input bit positions cannot be built either.Therefore, the number of front patterns 𝜔 decreases by one,and the outer loop is repeated; otherwise, the inner loop isexecuted with 𝜔 patterns. In the inner loop, for each set ofbit position groups 𝑔, the proposed algorithm checks whetherall DFAs 𝑑𝑓𝑎𝑠2 can be built under 𝑠. In particular, the failingpointer addition is not performed to reduce the processingtime. If all DFAs can be built with any set of bit positiongroups, the iteration of the inner loop is stopped; otherwise,𝑑𝑓𝑎𝑠2 is set as 𝜙 or empty. Then, if 𝑑𝑓𝑎𝑠2 is not 𝜙, Aho-Corasick algorithm is applied to add failing pointers, and theDFAs 𝑑𝑓𝑎𝑠2 are returned. If 𝑑𝑓𝑎𝑠2 is 𝜙, the number of theadopted front patterns 𝜔 decreases by one, and the outer loopis repeated.

Considering the obtained DFAs and 𝜔 patterns, the contentsof PMVs are generated by the construction in [3]. In this case,the information of the multiple patterns to be matched in eachstate can be obtained. The number of all possible sets of bitposition groups, the size of a PMV 𝑝, and the number of statesavailable in an FSM tile 𝑠 are predetermined, irrespectiveof the number of total target patterns 𝑇 . Therefore, the bitposition grouping for a string matcher has the constant timecomplexity of O (1). When all target patterns to be mappedonto multiple string matchers, the time complexity can be O(𝑇 ). On the other hand, the time complexity of pattern sortingcan be O (𝑇 𝑙𝑜𝑔2𝑇 ). However, due to the large constant factorof the bit position grouping complexity, if the number of targetpatterns 𝑇 is not sufficiently large, the pattern sorting will notbe dominant.

III. PERFORMANCE EVALUATION

Target patterns were extracted from Snort v2.8 rules [4].Considering design analysis in [3], an FSM tile was assumed

1006 IEEE COMMUNICATIONS LETTERS, VOL. 13, NO. 12, DECEMBER 2009

TABLE IPERFORMANCE COMPARISONS FOR FOUR SNORT RULE SETS.

TABLE IIPERFORMANCE COMPARISONS FOR total RULE SET.

to take two bits of one character as an input. Table I sum-marizes the numbers of adopted string matchers for fourrule sets by varying the size of a PMV 𝑝 when the numberof states available in an FSM tile s was 256. The numberof target patterns and the average number of characters intarget patterns are denoted as #𝑢 and 𝑎𝑣𝑔, respectively. Asshown in the third column of Table I, in order to show theeffectiveness of the gray-code based sorting, the bit positiongrouping was performed with the lexicographical sorting. InTable I, the number of adopted string matchers was reducedon average by 4.44%, in comparison with the existing bit-splitstring matching in [3]. The number of reduced string matchersslightly increased with 𝑝. For the backdoor and spyware-putrule sets, when 𝑝 was small, the number of adopted stringmatchers could not be reduced due to the small 𝑎𝑣𝑔 of therule set. In other cases, the proposed algorithm outperformedthe existing bit-split string matching. Considering the memorysize of each FSM tile calculated by Eq. (1), the optimal 𝑝 thatminimizes total memory requirements was different for eachrule set. For example, the optimal values of 𝑝 for the web-client and backdoor rule sets were 16 and 48, respectively.For the optimal 𝑝 of each rule set, the proposed algorithmalways reduced the number of adopted string matchers. Onthe other hand, compared with the evaluations when only thebit position grouping was performed, the proposed algorithmreduced the number of string matchers. Therefore, the gray-code based sorting was valid for minimizing total memoryrequirements with the proposed bit position grouping.

In Table II, the comparison was performed by varying boththe number of states available in an FSM tile 𝑠 and the sizeof a PMV 𝑝. For all patterns of Snort rule sets, total rule setwith 7766 unique patterns was obtained, where the averagenumber of characters in target patterns was 18.6. For all casesin Table II, the proposed algorithm reduced the number ofadopted string matchers; the number of total unused states in

TABLE IIISUMMARY OF NUMBERS OF STRING MATCHERS THAT DID NOT ADOPT

THE FIXED SET OF BIT POSITION GROUPS.

all FSM tiles was reduced on average by 13.46%. In particular,the reduced number of total unused states showed that memoryusage in homogeneous FSMs could be balanced. The statesleft unused showed that the number of patterns mapped ontoeach string matcher could be various. The unused states couldbe reduced by distributing target patterns into string matchersglobally, which was beyond the scope of our paper.

In order to prove the effectiveness of the bit positiongrouping, Table III summarizes the numbers of string matchersthat did not adopt the fixed set of bit position groups in [3].When a string matcher did not adopt the fixed set of bitposition groups, the proposed algorithm mapped more targetpatterns onto the string matcher than the method in [3]. Forthe backdoor, deleted, and spyware-put rule sets, when 𝑝 and𝑠 were 16 and 256, no string matchers adopted the fixed set ofbit position groups due to the small 𝑎𝑣𝑔 of the rule sets. Onaverage, when 𝑠 was 128, the number of string matchers thatdid not adopt the fixed set of bit position groups increased.However, the web-client rule set did not increase the number ofstring matchers due to the large 𝑎𝑣𝑔. In addition, the number ofstring matchers that did not adopt the fixed set of bit positiongroups could be saturated when 𝑝 was large. In Table III, theratio of the string matchers that did not adopt the fixed setof bit position groups was up to 33.33%. Therefore, it wasconcluded that the proposed bit position grouping increasedthe number of mapped target patterns in many string matchers.

Considering the performance enhancements, the proposedparallel string matching is useful for reducing memory costswithout losing regularity and scalability of the string matching.

REFERENCES

[1] P.-C. Lin, Y.-D. Lin, T.-H. Lee, and Y.-C. Lai, “Using string matchingfor deep packet inspection,” IEEE Computer, vol. 41, no. 4, pp. 23-28,2008.

[2] A. V. Aho and M. J. Corasick, “Efficient string matching: an aid tobibliographic search,” Commun. ACM, vol. 18, no. 6, pp. 333-340, 1975.

[3] L. Tan, B. Brotherton, and T. Sherwood, “Bit-split string-matchingengines for intrusion detection and prevention,” ACM Trans. Archit. andCode Optimization, vol. 3, no. 1, pp. 3-34, Mar. 2006.

[4] [Online]. Available: http://www.snort.org.