Fundamental Data Structures

PDF generated using the open source mwlib toolkit. See http://code.pediapress.com/ for more information. PDF generated at: Tue, 29 May 2012 17:21:42 UTC

ContentsArticlesIntroductionAbstract data type Data structure Analysis of algorithms Amortized analysis Accounting method Potential method 1 1 9 11 17 19 21 23 23 27 33 36 52 56 85 87 90 100 100 103 104 117 118 122 124 128 130 138 140 145 146

SequencesArray data type Array data structure Dynamic array Linked list Doubly linked list Stack (abstract data type) Queue (abstract data type) Double-ended queue Circular buffer

DictionariesAssociative array Association list Hash table Linear probing Quadratic probing Double hashing Cuckoo hashing Hopscotch hashing Hash function Perfect hash function Universal hashing K-independent hashing Tabulation hashing

Cryptographic hash function

149 156

SetsSet (abstract data type) Bit array Bloom filter MinHash Disjoint-set data structure Partition refinement

156 161 165 177 180 184 186 186 191 194 200 202 208 213 216 221 223 223 231 242 245 248 250 253 257 272 276 290 313 319 330 335 335

Priority queuesPriority queue Heap (data structure) Binary heap d-ary heap Binomial heap Fibonacci heap Pairing heap Double-ended priority queue Soft heap

Successors and neighborsBinary search algorithm Binary search tree Random binary tree Tree rotation Self-balancing binary search tree Treap AVL tree Redblack tree Scapegoat tree Splay tree Tango tree Skip list B-tree B+ tree

Integer and string searchingTrie

Radix tree Directed acyclic word graph Suffix tree Suffix array van Emde Boas tree Fusion tree

342 347 349 354 357 361

ReferencesArticle Sources and Contributors Image Sources, Licenses and Contributors 365 370

Article LicensesLicense 373

1

IntroductionAbstract data typeIn computer science, an abstract data type (ADT) is a mathematical model for a certain class of data structures that have similar behavior; or for certain data types of one or more programming languages that have similar semantics. An abstract data type is defined indirectly, only by the operations that may be performed on it and by mathematical constraints on the effects (and possibly cost) of those operations.[1] For example, an abstract stack data structure could be defined by three operations: push, that inserts some data item onto the structure, pop, that extracts an item from it (with the constraint that each pop always returns the most recently pushed item that has not been popped yet), and peek, that allows data on top of the structure to be examined without removal. When analyzing the efficiency of algorithms that use stacks, one may also specify that all operations take the same time no matter how many items have been pushed into the stack, and that the stack uses a constant amount of storage for each element. Abstract data types are purely theoretical entities, used (among other things) to simplify the description of abstract algorithms, to classify and evaluate data structures, and to formally describe the type systems of programming languages. However, an ADT may be implemented by specific data types or data structures, in many ways and in many programming languages; or described in a formal specification language. ADTs are often implemented as modules: the module's interface declares procedures that correspond to the ADT operations, sometimes with comments that describe the constraints. This information hiding strategy allows the implementation of the module to be changed without disturbing the client programs. The term abstract data type can also be regarded as a generalised approach of a number of algebraic structures, such as lattices, groups, and rings.[2] This can be treated as part of subject area of artificial intelligence. The notion of abstract data types is related to the concept of data abstraction, important in object-oriented programming and design by contract methodologies for software development .

Defining an abstract data type (ADT)An abstract data type is defined as a mathematical model of the data objects that make up a data type as well as the functions that operate on these objects. There are no standard conventions for defining them. A broad division may be drawn between "imperative" and "functional" definition styles.

Imperative abstract data type definitionsIn the "imperative" view, which is closer to the philosophy of imperative programming languages, an abstract data structure is conceived as an entity that is mutable meaning that it may be in different states at different times. Some operations may change the state of the ADT; therefore, the order in which operations are evaluated is important, and the same operation on the same entities may have different effects if executed at different times just like the instructions of a computer, or the commands and procedures of an imperative language. To underscore this view, it is customary to say that the operations are executed or applied, rather than evaluated. The imperative style is often used when describing abstract algorithms. This is described by Donald E. Knuth and can be referenced from here The Art of Computer Programming.

Abstract data type Abstract variable Imperative ADT definitions often depend on the concept of an abstract variable, which may be regarded as the simplest non-trivial ADT. An abstract variable V is a mutable entity that admits two operations: store(V,x) where x is a value of unspecified nature; and fetch(V), that yields a value; with the constraint that fetch(V) always returns the value x used in the most recent store(V,x) operation on the same variable V. As in so many programming languages, the operation store(V,x) is often written V x (or some similar notation), and fetch(V) is implied whenever a variable V is used in a context where a value is required. Thus, for example, V V + 1 is commonly understood to be a shorthand for store(V,fetch(V) + 1). In this definition, it is implicitly assumed that storing a value into a variable U has no effect on the state of a distinct variable V. To make this assumption explicit, one could add the constraint that if U and V are distinct variables, the sequence { store(U,x); store(V,y) } is equivalent to { store(V,y); store(U,x) }. More generally, ADT definitions often assume that any operation that changes the state of one ADT instance has no effect on the state of any other instance (including other instances of the same ADT) unless the ADT axioms imply that the two instances are connected (aliased) in that sense. For example, when extending the definition of abstract variable to include abstract records, the operation that selects a field from a record variable R must yield a variable V that is aliased to that part of R. The definition of an abstract variable V may also restrict the stored values x to members of a specific set X, called the range or type of V. As in programming languages, such restrictions may simplify the description and analysis of algorithms, and improve their readability. Note that this definition does not imply anything about the result of evaluating fetch(V) when V is un-initialized, that is, before performing any store operation on V. An algorithm that does so is usually considered invalid, because its effect is not defined. (However, there are some important algorithms whose efficiency strongly depends on the assumption that such a fetch is legal, and returns some arbitrary value in the variable's range.) Instance creation Some algorithms need to create new instances of some ADT (such as new variables, or new stacks). To describe such algorithms, one usually includes in the ADT definition a create() operation that yields an instance of the ADT, usually with axioms equivalent to the result of create() is distinct from any instance S in use by the algorithm. This axiom may be strengthened to exclude also partial aliasing with other instances. On the other hand, this axiom still allows implementations of create() to yield a previously created instance that has become inaccessible to the program.

2

Abstract data type Preconditions, postconditions, and invariants In imperative-style definitions, the axioms are often expressed by preconditions, that specify when an operation may be executed; postconditions, that relate the states of the ADT before and after the execution of each operation; and invariants, that specify properties of the ADT that are not changed by the operations. Example: abstract stack (imperative) As another example, an imperative definition of an abstract stack could specify that the state of a stack S can be modified only by the operations push(S,x), where x is some value of unspecified nature; and pop(S), that yields a value as a result; with the constraint that For any value x and any abstract variable V, the sequence of operations { push(S,x); V pop(S) } is equivalent to { V x }; Since the assignment { V x }, by definition, cannot change the state of S, this condition implies that { V pop(S) } restores S to the state it had before the { push(S,x) }. From this condition and from the properties of abstract variables, it follows, for example, that the sequence { push(S,x); push(S,y); U pop(S); push(S,z); V pop(S); W pop(S); } where x,y, and z are any values, and U, V, W are pairwise distinct variables, is equivalent to { U y; V z; W x } Here it is implicitly assumed that operations on a stack instance do not modify the state of any other ADT instance, including other stacks; that is, For any values x,y, and any distinct stacks S and T, the sequence { push(S,x); push(T,y) } is equivalent to { push(T,y); push(S,x) }. A stack ADT definition usually includes also a Boolean-valued function empty(S) and a create() operation that returns a stack instance, with axioms equivalent to create() S for any stack S (a newly created stack is distinct from all previous stacks) empty(create()) (a newly created stack is empty) not empty(push(S,x)) (pushing something into a stack makes it non-empty) Single-instance style Sometimes an ADT is defined as if only one instance of it existed during the execution of the algorithm, and all operations were applied to that instance, which is not explicitly notated. For example, the abstract stack above could have been defined with operations push(x) and pop(), that operate on "the" only existing stack. ADT definitions in this style can be easily rewritten to admit multiple coexisting instances of the ADT, by adding an explicit instance parameter (like S in the previous example) to every operation that uses or modifies the implicit instance. On the other hand, some ADTs cannot be meaningfully defined without assuming multiple instances. This is the case when a single operation takes two distinct instances of the ADT as parameters. For an example, consider augmenting the definition of the stack ADT with an operation compare(S,T) that checks whether the stacks S and T contain the same items in the same order.

3

Abstract data type

4

Functional ADT definitionsAnother way to define an ADT, closer to the spirit of functional programming, is to consider each state of the structure as a separate entity. In this view, any operation that modifies the ADT is modeled as a mathematical function that takes the old state as an argument, and returns the new state as part of the result. Unlike the "imperative" operations, these functions have no side effects. Therefore, the order in which they are evaluated is immaterial, and the same operation applied to the same arguments (including the same input states) will always return the same results (and output states). In the functional view, in particular, there is no way (or need) to define an "abstract variable" with the semantics of imperative variables (namely, with fetch and store operations). Instead of storing values into variables, one passes them as arguments to functions. Example: abstract stack (functional) For example, a complete functional-style definition of a stack ADT could use the three operations: push: takes a stack state and an arbitrary value, returns a stack state; top: takes a stack state, returns a value; pop: takes a stack state, returns a stack state; with the following axioms: top(push(s,x)) = x (pushing an item onto a stack leaves it at the top) pop(push(s,x)) = s (pop undoes the effect of push) In a functional-style definition there is no need for a create operation. Indeed, there is no notion of "stack instance". The stack states can be thought of as being potential states of a single stack structure, and two stack states that contain the same values in the same order are considered to be identical states. This view actually mirrors the behavior of some concrete implementations, such as linked lists with hash cons. Instead of create(), a functional definition of a stack ADT may assume the existence of a special stack state, the empty stack, designated by a special symbol like or "()"; or define a bottom() operation that takes no arguments and returns this special stack state. Note that the axioms imply that push(,x) In a functional definition of a stack one does not need an empty predicate: instead, one can test whether a stack is empty by testing whether it is equal to . Note that these axioms do not define the effect of top(s) or pop(s), unless s is a stack state returned by a push. Since push leaves the stack non-empty, those two operations are undefined (hence invalid) when s = . On the other hand, the axioms (and the lack of side effects) imply that push(s,x) = push(t,y) if and only if x = y and s = t. As in some other branches of mathematics, it is customary to assume also that the stack states are only those whose existence can be proved from the axioms in a finite number of steps. In the stack ADT example above, this rule means that every stack is a finite sequence of values, that becomes the empty stack () after a finite number of pops. By themselves, the axioms above do not exclude the existence of infinite stacks (that can be poped forever, each time yielding a different state) or circular stacks (that return to the same state after a finite number of pops). In particular, they do not exclude states s such that pop(s) = s or push(s,x) = s for some x. However, since one cannot obtain such stack states with the given operations, they are assumed "not to exist".

Abstract data type

5

Advantages of abstract data typing Encapsulation Abstraction provides a promise that any implementation of the ADT has certain properties and abilities; knowing these is all that is required to make use of an ADT object. The user does not need any technical knowledge of how the implementation works to use the ADT. In this way, the implementation may be complex but will be encapsulated in a simple interface when it is actually used. Localization of change Code that uses an ADT object will not need to be edited if the implementation of the ADT is changed. Since any changes to the implementation must still comply with the interface, and since code using an ADT may only refer to properties and abilities specified in the interface, changes may be made to the implementation without requiring any changes in code where the ADT is used. Flexibility Different implementations of an ADT, having all the same properties and abilities, are equivalent and may be used somewhat interchangeably in code that uses the ADT. This gives a great deal of flexibility when using ADT objects in different situations. For example, different implementations of an ADT may be more efficient in different situations; it is possible to use each in the situation where they are preferable, thus increasing overall efficiency.

Typical operationsSome operations that are often specified for ADTs (possibly under other names) are compare(s,t), that tests whether two structures are equivalent in some sense; hash(s), that computes some standard hash function from the instance's state; print(s) or show(s), that produces a human-readable representation of the structure's state. In imperative-style ADT definitions, one often finds also create(), that yields a new instance of the ADT; initialize(s), that prepares a newly created instance s for further operations, or resets it to some "initial state"; copy(s,t), that puts instance s in a state equivalent to that of t; clone(t), that performs s new(), copy(s,t), and returns s; free(s) or destroy(s), that reclaims the memory and other resources used by s; The free operation is not normally relevant or meaningful, since ADTs are theoretical entities that do not "use memory". However, it may be necessary when one needs to analyze the storage used by an algorithm that uses the ADT. In that case one needs additional axioms that specify how much memory each ADT instance uses, as a function of its state, and how much of it is returned to the pool by free.

ExamplesSome common ADTs, which have proved useful in a great variety of applications, are Container Deque List Map Multimap

Multiset Priority queue Queue

Abstract data type Set Stack String Tree

6

Each of these ADTs may be defined in many ways and variants, not necessarily equivalent. For example, a stack ADT may or may not have a count operation that tells how many items have been pushed and not yet popped. This choice makes a difference not only for its clients but also for the implementation.

ImplementationImplementing an ADT means providing one procedure or function for each abstract operation. The ADT instances are represented by some concrete data structure that is manipulated by those procedures, according to the ADT's specifications. Usually there are many ways to implement the same ADT, using several different concrete data structures. Thus, for example, an abstract stack can be implemented by a linked list or by an array. An ADT implementation is often packaged as one or more modules, whose interface contains only the signature (number and types of the parameters and results) of the operations. The implementation of the module namely, the bodies of the procedures and the concrete data structure used can then be hidden from most clients of the module. This makes it possible to change the implementation without affecting the clients. When implementing an ADT, each instance (in imperative-style definitions) or each state (in functional-style definitions) is usually represented by a handle of some sort.[3] Modern object-oriented languages, such as C++ and Java, support a form of abstract data types. When a class is used as a type, it is an abstract type that refers to a hidden representation. In this model an ADT is typically implemented as a class, and each instance of the ADT is an object of that class. The module's interface typically declares the constructors as ordinary procedures, and most of the other ADT operations as methods of that class. However, such an approach does not easily encapsulate multiple representational variants found in an ADT. It also can undermine the extensibility of object-oriented programs. In a pure object-oriented program that uses interfaces as types, types refer to behaviors not representations.

Example: implementation of the stack ADTAs an example, here is an implementation of the stack ADT above in the C programming language. Imperative-style interface An imperative-style interface might be: typedef struct stack_Rep stack_Rep; representation (an opaque record). */ typedef stack_Rep *stack_T; instance (an opaque pointer). */ typedef void *stack_Item; stored in stack (arbitrary address). */ stack_T stack_create(void); instance, initially empty. */ void stack_push(stack_T s, stack_Item e); the stack. */ stack_Item stack_pop(stack_T s); the stack and return it . */ /* Type: instance /* Type: handle to a stack /* Type: value that can be

/* Create new stack /* Add an item at the top of /* Remove the top item from

Abstract data type int stack_empty(stack_T ts); empty. */ This implementation could be used in the following manner: #include stack_T t = stack_create(); int foo = 17; t = stack_push(t, &foo); stack. */ void *e = stack_pop(t); the stack. */ if (stack_empty(t)) { } /* Include the stack interface. */ /* Create a stack instance. */ /* An arbitrary datum. */ /* Push the address of 'foo' onto the /* Check whether stack is

7

/* Get the top item and delete it from /* Do something if stack is empty. */

This interface can be implemented in many ways. The implementation may be arbitrarily inefficient, since the formal definition of the ADT, above, does not specify how much space the stack may use, nor how long each operation should take. It also does not specify whether the stack state t continues to exist after a call s pop(t). In practice the formal definition should specify that the space is proportional to the number of items pushed and not yet popped; and that every one of the operations above must finish in a constant amount of time, independently of that number. To comply with these additional specifications, the implementation could use a linked list, or an array (with dynamic resizing) together with two integers (an item count and the array size) Functional-style interface Functional-style ADT definitions are more appropriate for functional programming languages, and vice-versa. However, one can provide a functional style interface even in an imperative language like C. For example: typedef struct stack_Rep stack_Rep; representation (an opaque record). */ typedef stack_Rep *stack_T; state (an opaque pointer). */ typedef void *stack_Item; address). */ /* Type: stack state /* Type: handle to a stack /* Type: item (arbitrary

stack_T stack_empty(void); /* Returns the empty stack state. */ stack_T stack_push(stack_T s, stack_Item x); /* Adds x at the top of s, returns the resulting state. */ stack_Item stack_top(stack_T s); /* Returns the item currently at the top of s. */ stack_T stack_pop(stack_T s); /* Remove the top item from s, returns the resulting state. */ The main problem is that C lacks garbage collection, and this makes this style of programming impractical; moreover, memory allocation routines in C are slower than allocation in a typical garbage collector, thus the performance impact of so many allocations is even greater.

Abstract data type

8

ADT librariesMany modern programming languages,such as C++ and Java, come with standard libraries that implement several common ADTs, such as those listed above.

Built-in abstract data typesThe specification of some programming languages is intentionally vague about the representation of certain built-in data types, defining only the operations that can be done on them. Therefore, those types can be viewed as "built-in ADTs". Examples are the arrays in many scripting languages, such as Awk, Lua, and Perl, which can be regarded as an implementation of the Map or Table ADT.

References[1] Barbara Liskov, Programming with Abstract Data Types, in Proceedings of the ACM SIGPLAN Symposium on Very High Level Languages, pp. 50--59, 1974, Santa Monica, California [2] Rudolf Lidl (2004). Abstract Algebra. Springer. ISBN81-8128-149-7., Chapter 7,section 40. [3] Robert Sedgewick (1998). Algorithms in C. Addison/Wesley. ISBN0-201-31452-5., definition 4.4.

Further Mitchell, John C.; Plotkin, Gordon (July 1988). "Abstract Types Have Existential Type" (http://theory.stanford. edu/~jcm/papers/mitch-plotkin-88.pdf). ACM Transactions on Programming Languages and Systems 10 (3).

External links Abstract data type (http://www.nist.gov/dads/HTML/abstractDataType.html) in NIST Dictionary of Algorithms and Data Structures

Data structure

9

Data structureIn computer science, a data structure is a particular way of storing and organizing data in a computer so that it can be used efficiently.[1][2] Different kinds of data structures are suited to different kinds of applications, and some are highly specialized to specific tasks. For example, B-trees are particularly well-suited for implementation of databases, while compiler implementations usually use hash tables to look up identifiers. Data structures are used in almost every program or software system. Data a hash table structures provide a means to manage huge amounts of data efficiently, such as large databases and internet indexing services. Usually, efficient data structures are a key to designing efficient algorithms. Some formal design methods and programming languages emphasize data structures, rather than algorithms, as the key organizing factor in software design.

Overview An array data structure stores a number of elements of the same type in a specific order. They are accessed using an integer to specify which element is required (although the elements may be of almost any type). Arrays may be fixed-length or expandable. Record (also called tuple or struct) Records are among the simplest data structures. A record is a value that contains other values, typically in fixed number and sequence and typically indexed by names. The elements of records are usually called fields or members. A hash or dictionary or map is a more flexible variation on a record, in which name-value pairs can be added and deleted freely. Union. A union type definition will specify which of a number of permitted primitive types may be stored in its instances, e.g. "float or long integer". Contrast with a record, which could be defined to contain a float and an integer; whereas, in a union, there is only one value at a time. A tagged union (also called a variant, variant record, discriminated union, or disjoint union) contains an additional field indicating its current type, for enhanced type safety. A set is an abstract data structure that can store specific values, without any particular order, and no repeated values. Values themselves are not retrieved from sets, rather one tests a value for membership to obtain a boolean "in" or "not in". An object contains a number of data fields, like a record, and also a number of program code fragments for accessing or modifying them. Data structures not containing code, like those above, are called plain old data structure. Many others are possible, but they tend to be further variations and compounds of the above.

Data structure

10

Basic principlesData structures are generally based on the ability of a computer to fetch and store data at any place in its memory, specified by an addressa bit string that can be itself stored in memory and manipulated by the program. Thus the record and array data structures are based on computing the addresses of data items with arithmetic operations; while the linked data structures are based on storing addresses of data items within the structure itself. Many data structures use both principles, sometimes combined in non-trivial ways (as in XOR linking) The implementation of a data structure usually requires writing a set of procedures that create and manipulate instances of that structure. The efficiency of a data structure cannot be analyzed separately from those operations. This observation motivates the theoretical concept of an abstract data type, a data structure that is defined indirectly by the operations that may be performed on it, and the mathematical properties of those operations (including their space and time cost).

Language supportMost assembly languages and some low-level languages, such as BCPL(Basic Combined Programming Language), lack support for data structures. Many high-level programming languages, and some higher-level assembly languages, such as MASM, on the other hand, have special syntax or other built-in support for certain data structures, such as vectors (one-dimensional arrays) in the C language or multi-dimensional arrays in Pascal. Most programming languages feature some sorts of library mechanism that allows data structure implementations to be reused by different programs. Modern languages usually come with standard libraries that implement the most common data structures. Examples are the C++ Standard Template Library, the Java Collections Framework, and Microsoft's .NET Framework. Modern languages also generally support modular programming, the separation between the interface of a library module and its implementation. Some provide opaque data types that allow clients to hide implementation details. Object-oriented programming languages, such as C++, Java and .NET Framework use classes for this purpose. Many known data structures have concurrent versions that allow multiple computing threads to access the data structure simultaneously.

References[1] Paul E. Black (ed.), entry for data structure in Dictionary of Algorithms and Data Structures. U.S. National Institute of Standards and Technology. 15 December 2004. Online version (http:/ / www. itl. nist. gov/ div897/ sqg/ dads/ HTML/ datastructur. html) Accessed May 21, 2009. [2] Entry data structure in the Encyclopdia Britannica (2009) Online entry (http:/ / www. britannica. com/ EBchecked/ topic/ 152190/ data-structure) accessed on May 21, 2009.

Further reading Peter Brass, Advanced Data Structures, Cambridge University Press, 2008. Donald Knuth, The Art of Computer Programming, vol. 1. Addison-Wesley, 3rd edition, 1997. Dinesh Mehta and Sartaj Sahni Handbook of Data Structures and Applications, Chapman and Hall/CRC Press, 2007. Niklaus Wirth, Algorithms and Data Structures, Prentice Hall, 1985.

Data structure

11

External links UC Berkeley video course on data structures (http://academicearth.org/courses/data-structures) Descriptions (http://nist.gov/dads/) from the Dictionary of Algorithms and Data Structures CSE.unr.edu (http://www.cse.unr.edu/~bebis/CS308/) Data structures course with animations (http://www.cs.auckland.ac.nz/software/AlgAnim/ds_ToC.html) Data structure tutorials with animations (http://courses.cs.vt.edu/~csonline/DataStructures/Lessons/index. html) An Examination of Data Structures from .NET perspective (http://msdn.microsoft.com/en-us/library/ aa289148(VS.71).aspx) Schaffer, C. Data Structures and Algorithm Analysis (http://people.cs.vt.edu/~shaffer/Book/C++ 3e20110915.pdf)

Analysis of algorithmsIn computer science, the analysis of algorithms is the determination of the amount of resources (such as time and storage) necessary to execute them. Most algorithms are designed to work with inputs of arbitrary length. Usually the efficiency or running time of an algorithm is stated as a function relating the input length to the number of steps (time complexity) or storage locations (space complexity). Algorithm analysis is an important part of a broader computational complexity theory, which provides theoretical estimates for the resources needed by any algorithm which solves a given computational problem. These estimates provide an insight into reasonable directions of search for efficient algorithms. In theoretical analysis of algorithms it is common to estimate their complexity in the asymptotic sense, i.e., to estimate the complexity function for arbitrarily large input. Big O notation, omega notation and theta notation are used to this end. For instance, binary search is said to run in a number of steps proportional to the logarithm of the length of the list being searched, or in O(log(n)), colloquially "in logarithmic time". Usually asymptotic estimates are used because different implementations of the same algorithm may differ in efficiency. However the efficiencies of any two "reasonable" implementations of a given algorithm are related by a constant multiplicative factor called a hidden constant. Exact (not asymptotic) measures of efficiency can sometimes be computed but they usually require certain assumptions concerning the particular implementation of the algorithm, called model of computation. A model of computation may be defined in terms of an abstract computer, e.g., Turing machine, and/or by postulating that certain operations are executed in unit time. For example, if the sorted list to which we apply binary search has n elements, and we can guarantee that each lookup of an element in the list can be done in unit time, then at most log2 n + 1 time units are needed to return an answer.

Cost modelsTime efficiency estimates depend on what we define to be a step. For the analysis to correspond usefully to the actual execution time, the time required to perform a step must be guaranteed to be bounded above by a constant. One must be careful here; for instance, some analyses count an addition of two numbers as one step. This assumption may not be warranted in certain contexts. For example, if the numbers involved in a computation may be arbitrarily large, the time required by a single addition can no longer be assumed to be constant. Two cost models are generally used:[1][2][3][4][5] the uniform cost model, also called uniform-cost measurement (and similar variations), assigns a constant cost to every machine operation, regardless of the size of the numbers involved

Analysis of algorithms the logarithmic cost model, also called logarithmic-cost measurement (and variations thereof), assigns a cost to every machine operation proportional to the number of bits involved The latter is more cumbersome to use, so it's only employed when necessary, for example in the analysis of arbitrary-precision arithmetic algorithms, like those used in cryptography. A key point which is often overlooked is that published lower bounds for problems are often given for a model of computation that is more restricted than the set of operations that you could use in practice and therefore there are algorithms that are faster than what would naively be thought possible.[6]

12

Run-time analysisRun-time analysis is a theoretical classification that estimates and anticipates the increase in running time (or run-time) of an algorithm as its input size (usually denoted as n) increases. Run-time efficiency is a topic of great interest in computer science: A program can take seconds, hours or even years to finish executing, depending on which algorithm it implements (see also performance analysis, which is the analysis of an algorithm's run-time in practice).

Shortcomings of empirical metricsSince algorithms are platform-independent (i.e. a given algorithm can be implemented in an arbitrary programming language on an arbitrary computer running an arbitrary operating system), there are significant drawbacks to using an empirical approach to gauge the comparative performance of a given set of algorithms. Take as an example a program that looks up a specific entry in a sorted list of size n. Suppose this program were implemented on Computer A, a state-of-the-art machine, using a linear search algorithm, and on Computer B, a much slower machine, using a binary search algorithm. Benchmark testing on the two computers running their respective programs might look something like the following:n (list size) Computer A run-time (in nanoseconds) 7 32 125 500 Computer B run-time (in nanoseconds) 100,000 150,000 200,000 250,000

15 65 250 1,000

Based on these metrics, it would be easy to jump to the conclusion that Computer A is running an algorithm that is far superior in efficiency to that of Computer B. However, if the size of the input-list is increased to a sufficient number, that conclusion is dramatically demonstrated to be in error:

Analysis of algorithms

13

n (list size)

Computer A run-time (in nanoseconds) 7 32 125 500 ... 500,000 2,000,000 8,000,000 ...

Computer B run-time (in nanoseconds) 100,000 150,000 200,000 250,000 ... 500,000 550,000 600,000 ... 1,375,000 ns, or 1.375 milliseconds

15 65 250 1,000 ... 1,000,000 4,000,000 16,000,000 ...

63,072 1012 31,536 1012 ns, or 1 year

Computer A, running the linear search program, exhibits a linear growth rate. The program's run-time is directly proportional to its input size. Doubling the input size doubles the run time, quadrupling the input size quadruples the run-time, and so forth. On the other hand, Computer B, running the binary search program, exhibits a logarithmic growth rate. Doubling the input size only increases the run time by a constant amount (in this example, 25,000 ns). Even though Computer A is ostensibly a faster machine, Computer B will inevitably surpass Computer A in run-time because it's running an algorithm with a much slower growth rate.

Orders of growthInformally, an algorithm can be said to exhibit a growth rate on the order of a mathematical function if beyond a certain input size n, the function f(n) times a positive constant provides an upper bound or limit for the run-time of that algorithm. In other words, for a given input size n greater than some n0 and a constant c, the running time of that algorithm will never be larger than c f(n). This concept is frequently expressed using Big O notation. For example, since the run-time of insertion sort grows quadratically as its input size increases, insertion sort can be said to be of order O(n). Big O notation is a convenient way to express the worst-case scenario for a given algorithm, although it can also be used to express the average-case for example, the worst-case scenario for quicksort is O(n), but the average-case run-time is O(n log n).[7]

Empirical orders of growthAssuming the order of growth follows power rule, measurements (of run time, say), so that , the coefficient a can be found by taking empirical , and calculating . If the order of growth indeed follows the power rule, at some problem-size points

the empirical value of a will stay constant at different ranges, and if not, it will change - but still could serve for comparison of any two given algorithms as to their empirical local orders of growth behaviour. Applied to the above table:

Analysis of algorithms

14

n (list size)

Computer A run-time (in nanoseconds) 7 32 125 500 ... 500,000 2,000,000

Local order of growth (n^_)

Computer B run-time (in nanoseconds) 100,000

Local order of growth (n^_)

15 65 250 1,000 ... 1,000,000 4,000,000

1.04 1.01 1.00 ...

150,000 200,000 250,000 ... 500,000

0.28 0.21 0.16 ...

1.00 1.00 ...

550,000 600,000 ...

0.07 0.06 ...

16,000,000 8,000,000 ... ...

It is clearly seen that the first algorithm exhibits a linear order of growth indeed following the power rule. The empirical values for the second one are diminishing rapidly, suggesting it follows another rule of growth and in any case has much lower local order of growth (and improving further still), empirically, than the first one.

Evaluating run-time complexityThe run-time complexity for the worst-case scenario of a given algorithm can sometimes be evaluated by examining the structure of the algorithm and making some simplifying assumptions. Consider the following pseudocode: 1 2 3 4 5 6 7 get a positive integer from input if n > 10 print "This might take a while..." for i = 1 to n for j = 1 to i print i * j print "Done!"

A given computer will take a discrete amount of time to execute each of the instructions involved with carrying out this algorithm. The specific amount of time to carry out a given instruction will vary depending on which instruction is being executed and which computer is executing it, but on a conventional computer, this amount will be deterministic.[8] Say that the actions carried out in step 1 are considered to consume time T1, step 2 uses time T2, and so forth. In the algorithm above, steps 1, 2 and 7 will only be run once. For a worst-case evaluation, it should be assumed that step 3 will be run as well. Thus the total amount of time to run steps 1-3 and step 7 is:

The loops in steps 4, 5 and 6 are trickier to evaluate. The outer loop test in step 4 will execute ( n + 1 ) times (note that an extra step is required to terminate the for loop, hence n + 1 and not n executions), which will consume T4( n + 1 ) time. The inner loop, on the other hand, is governed by the value of i, which iterates from 1 to n. On the first pass through the outer loop, j iterates from 1 to 1: The inner loop makes one pass, so running the inner loop body (step 6) consumes T6 time, and the inner loop test (step 5) consumes 2T5 time. During the next pass through the outer loop, j iterates from 1 to 2: the inner loop makes two passes, so running the inner loop body (step 6) consumes 2T6 time, and the inner loop test (step 5) consumes 3T5 time. Altogether, the total time required to run the inner loop body can be expressed as an arithmetic progression:

Analysis of algorithms

15

which can be factored[9] as

The total time required to run the inner loop test can be evaluated similarly:

which can be factored as

Therefore the total running time for this algorithm is:

which reduces to

As a rule-of-thumb, one can assume that the highest-order term in any given function dominates its rate of growth and thus defines its run-time order. In this example, n is the highest-order term, so one can conclude that f(n) = O(n). Formally this can be proven as follows: Prove that

(for n 0) Let k be a constant greater than or equal to [T1..T7] (for n 1) Therefore for A more elegant approach to analyzing this algorithm would be to declare that [T1..T7] are all equal to one unit of time greater than or equal to [T1..T7]. This would mean that the algorithm's running time breaks down as follows:[10] (for n 1)

Growth rate analysis of other resourcesThe methodology of run-time analysis can also be utilized for predicting other growth rates, such as consumption of memory space. As an example, consider the following pseudocode which manages and reallocates memory usage by a program based on the size of a file which that program manages: while (file still open) let n = size of file for every 100,000 kilobytes of increase in file size double the amount of memory reserved

Analysis of algorithms In this instance, as the file size n increases, memory will be consumed at an exponential growth rate, which is order O(2n).[11]

16

RelevanceAlgorithm analysis is important in practice because the accidental or unintentional use of an inefficient algorithm can significantly impact system performance. In time-sensitive applications, an algorithm taking too long to run can render its results outdated or useless. An inefficient algorithm can also end up requiring an uneconomical amount of computing power or storage in order to run, again rendering it practically useless.

Notes[1] Alfred V. Aho; John E. Hopcroft; Jeffrey D. Ullman (1974). The design and analysis of computer algorithms. Addison-Wesley Pub. Co.., section 1.3 [2] Juraj Hromkovi (2004). Theoretical computer science: introduction to Automata, computability, complexity, algorithmics, randomization, communication, and cryptography (http:/ / books. google. com/ books?id=KpNet-n262QC& pg=PA177). Springer. pp.177178. ISBN978-3-540-14015-3. . [3] Giorgio Ausiello (1999). Complexity and approximation: combinatorial optimization problems and their approximability properties (http:/ / books. google. com/ books?id=Yxxw90d9AuMC& pg=PA3). Springer. pp.38. ISBN978-3-540-65431-5. . [4] Wegener, Ingo (2005), Complexity theory: exploring the limits of efficient algorithms (http:/ / books. google. com/ books?id=u7DZSDSUYlQC& pg=PA20), Berlin, New York: Springer-Verlag, p.20, ISBN978-3-540-21045-0, [5] Robert Endre Tarjan (1983). Data structures and network algorithms (http:/ / books. google. com/ books?id=JiC7mIqg-X4C& pg=PA3). SIAM. pp.37. ISBN978-0-89871-187-5. . [6] Examples of the price of abstraction? (http:/ / cstheory. stackexchange. com/ questions/ 608/ examples-of-the-price-of-abstraction), cstheory.stackexchange.com [7] The term lg is often used as shorthand for log2 [8] However, this is not the case with a quantum computer [9] It can be proven by induction that [10] This approach, unlike the above approach, neglects the constant time consumed by the loop tests which terminate their respective loops, but it is trivial to prove that such omission does not affect the final result [11] Note that this is an extremely rapid and most likely unmanageable growth rate for consumption of memory resources

References Cormen, Thomas H.; Leiserson, Charles E.; Rivest, Ronald L. & Stein, Clifford (2001). Introduction to Algorithms. Chapter 1: Foundations (Second ed.). Cambridge, MA: MIT Press and McGraw-Hill. pp.3122. ISBN0-262-03293-7. Sedgewick, Robert (1998). Algorithms in C, Parts 1-4: Fundamentals, Data Structures, Sorting, Searching (3rd ed.). Reading, MA: Addison-Wesley Professional. ISBN978-0-201-31452-6. Knuth, Donald. The Art of Computer Programming. Addison-Wesley. Greene, Daniel A.; Knuth, Donald E. (1982). Mathematics for the Analysis of Algorithms (Second ed.). Birkhuser. ISBN3-7463-3102-X . Goldreich, Oded (2010). Computational Complexity: A Conceptual Perspective. Cambridge University Press. ISBN978-0-521-88473-0.

Amortized analysis

17

Amortized analysisIn computer science, amortized analysis is a method of analyzing algorithms that considers the entire sequence of operations of the program. It allows for the establishment of a worst-case bound for the performance of an algorithm irrespective of the inputs by looking at all of the operations. At the heart of the method is the idea that while certain operations may be extremely costly in resources, they cannot occur at a high-enough frequency to weigh down the entire program because the number of less costly operations will far outnumber the costly ones in the long run, "paying back" the program over a number of iterations.[1] It is particularly useful because it guarantees worst-case performance rather than making assumptions about the state of the program.

HistoryAmortized analysis initially emerged from a method called aggregate analysis, which is now subsumed by amortized analysis. However, the technique was first formally introduced by Robert Tarjan in his paper Amortized Computational Complexity, which addressed the need for a more useful form of analysis than the common probabilistic methods used. Amortization was initially used for very specific types of algorithms, particularly those involving binary trees and union operations. However, it is now ubiquitous and comes into play when analyzing many other algorithms as well.[1]

MethodThe method requires knowledge of which series of operations are possible. This is most commonly the case with data structures, which have state that persists between operations. The basic idea is that a worst case operation can alter the state in such a way that the worst case cannot occur again for a long time, thus "amortizing" its cost. There are generally three methods for performing amortized analysis: the aggregate method, the accounting method, and the potential method. All of these give the same answers, and their usage difference is primarily circumstantial and due to individual preference.[2] Aggregate analysis determines the upper bound T(n) on the total cost of a sequence of n operations, then calculates the average cost to be T(n) / n.[2] The accounting method determines the individual cost of each operation, combining its immediate execution time and its influence on the running time of future operations. Usually, many short-running operations accumulate a "debt" of unfavorable state in small increments, while rare long-running operations decrease it drastically.[2] The potential method is like the accounting method, but overcharges operations early to compensate for undercharges later.[2]

ExamplesAs a simple example, in a specific implementation of the dynamic array, we double the size of the array each time it fills up. Because of this, array reallocation may be required, and in the worst case an insertion may require O(n). However, a sequence of n insertions can always be done in O(n) time, because the rest of the insertions are done in constant time, so n insertions can be completed in O(n) time. The amortized time per operation is therefore O(n) / n = O(1). Another way to see this is to think of a sequence of n operations. There are 2 possible operations: a regular insertion which requires a constant c time to perform (assume c = 1), and an array doubling which requires O(j) time (where j> (w-M) This scheme does not satisfy the uniform difference property and is only . To understand the behavior of the hash function, notice that, if highest-order 'M' bits, then whether position or . Since and have the same appears on , it follows that . On the other hand, has either all 1's or all 0's as its highest order M bits (depending on is larger. Assume that the least significant set bit of is a random odd integer and odd integers have inverses in the ring -almost-universal; for any ,

will be uniformly distributed among

-bit integers with the least significant set bit on position

. The probability that these bits are all 0's or all 1's is therefore at most if , then higher-order M bits of . Finally, if then bit of

contain both 0's and 1's, so it is certain that is 1 and if and . . To obtain a truly 'universal'

only if bits is tight, as can be shown withalso example happens with probability are the 1, which This analysis and hash function, one can use the multiply-add-shift scheme

where

is a random odd positive integer with . With these choices of and

and ,

where

is chosen at random from for all

Universal hashing .[7]

143

Hashing vectorsThis section is concerned with hashing a fixed-length vector of machine words. Interpret the input as a vector of machine words (integers of bits each). If is a universal family with the uniform difference property, the following family dating back to Carter and Wegman[1] also has the uniform difference property (and hence is universal): , where each If is chosen independently at random.

is a power of two, one may replace summation by exclusive or.[8]

In practice, if double-precision arithmetic is available, this is instantiated with the multiply-shift hash family of.[9] Initialize the hash function with a vector of random odd integers on bits each. Then if the number of bins is for : . It is possible to halve the number of multiplications, which roughly translates to a two-fold speed-up in practice.[8] Initialize the hash function with a vector of random odd integers on bits each. The following hash family is universal[10]: . If double-precision operations are not available, one can interpret the input as a vector of half-words ( integers). The algorithm will then use multiplications, where -bit

was the number of half-words in the vector.

Thus, the algorithm runs at a "rate" of one multiplication per word of input. The same scheme can also be used for hashing integers, by interpreting their bits as vectors of bytes. In this variant, the vector technique is known as tabulation hashing and it provides a practical alternative to multiplication-based universal hashing schemes.[11]

Hashing stringsThis refers to hashing a variable-sized vector of machine words. If the length of the string can be bounded by a small number, it is best to use the vector solution from above (conceptually padding the vector with zeros up to the upper bound). The space required is the maximal length of the string, but the time to evaluate is just the length of (the zero-padding can be ignored when evaluating the hash function without affecting universality[8]). Now assume we want to hash family proposed by.[9]

, where a good bound on

is not known a priori. A universal , let

treats the string

as the coefficients of a polynomial modulo a large prime. If

be a prime and define: , where is uniformly random and is chosen

randomly from a universal family mapping integer domain . Consider two strings and let be length of the longer one; for the analysis, the shorter string is conceptually padded with zeros up to length coefficients . A collision before applying roots modulo implies that is a root of the polynomial with . . This polynomial has at most , so the collision probability is at most

The probability of collision through the random

brings the total collision probability to

. Thus, if the

Universal hashing prime

144

is sufficiently large compared to the length of strings hashed, the family is very close to universal (in statistical

distance). To mitigate the computational penalty of modular arithmetic, two tricks are used in practice [8]: 1. One chooses the prime modulo to be close to a power of two, such as a Mersenne prime. This allows arithmetic to be implemented without division (using faster operations like addition and shifts). For instance, on

modern architectures one can work with , while 's are 32-bit values. 2. One can apply vector hashing to blocks. For instance, one applies vector hashing to each 16-word block of the string, and applies string hashing to the results. Since the slower string hashing is applied on a substantially smaller vector, this will essentially be as fast as vector hashing.

References[1] Carter, Larry; Wegman, Mark N. (1979). "Universal Classes of Hash Functions". Journal of Computer and System Sciences 18 (2): 143154. doi:10.1016/0022-0000(79)90044-8. Conference version in STOC'77. [2] Miltersen, Peter Bro. "Universal Hashing" (http:/ / www. webcitation. org/ 5hmOaVISI) (PDF). Archived from the original (http:/ / www. daimi. au. dk/ ~bromille/ Notes/ un. pdf) on 24th June 2009. . [3] Motwani, Rajeev; Raghavan, Prabhakar (1995). Randomized Algorithms. Cambridge University Press. p.221. ISBN0-521-47465-5. [4] Baran, Ilya; Demaine, Erik D.; Ptracu, Mihai (2008). "Subquadratic Algorithms for 3SUM" (http:/ / people. csail. mit. edu/ mip/ papers/ 3sum/ 3sum. pdf). Algorithmica 50 (4): 584596. doi:10.1007/s00453-007-9036-3. . [5] Dietzfelbinger, Martin; Hagerup, Torben; Katajainen, Jyrki; Penttonen, Martti (1997). "A Reliable Randomized Algorithm for the Closest-Pair Problem" (http:/ / www. diku. dk/ ~jyrki/ Paper/ CP-11. 4. 1997. ps) (Postscript). Journal of Algorithms 25 (1): 1951. doi:10.1006/jagm.1997.0873. . Retrieved 10 February 2011. [6] Thorup, Mikkel. "Text-book algorithms at SODA" (http:/ / mybiasedcoin. blogspot. com/ 2009/ 12/ text-book-algorithms-at-soda-guest-post. html). . [7] Woelfel, Philipp (1999). "Efficient Strongly Universal and Optimally Universal Hashing" (http:/ / www. springerlink. com/ content/ a10p748w7pr48682/ ) (PDF). LNCS. 1672. Mathematical Foundations of Computer Science 1999. pp.262-272. doi:10.1007/3-540-48340-3_24. . Retrieved 17 May 2011. [8] Thorup, Mikkel (2009). "String hashing for linear probing" (http:/ / www. siam. org/ proceedings/ soda/ 2009/ SODA09_072_thorupm. pdf). Proc. 20th ACM-SIAM Symposium on Discrete Algorithms (SODA). pp.655664. ., section 5.3 [9] Dietzfelbinger, Martin; Gil, Joseph; Matias, Yossi; Pippenger, Nicholas (1992). "Polynomial Hash Functions Are Reliable (Extended Abstract)". Proc. 19th International Colloquium on Automata, Languages and Programming (ICALP). pp.235246. [10] Black, J.; Halevi, S.; Krawczyk, H.; Krovetz, T. (1999). "UMAC: Fast and Secure Message Authentication" (http:/ / www. cs. ucdavis. edu/ ~rogaway/ papers/ umac-full. pdf). Advances in Cryptology (CRYPTO '99). ., Equation 1 [11] Ptracu, Mihai; Thorup, Mikkel (2011). "The power of simple tabulation hashing". Proceedings of the 43rd annual ACM Symposium on Theory of Computing (STOC '11). pp.110. arXiv:1011.5200. doi:10.1145/1993636.1993638.

Further reading Knuth, Donald Ervin (1998). [The Art of Computer Programming], Vol. III: Sorting and Searching (2e ed.). Reading, Mass ; London: Addison-Wesley. ISBN0-201-89685-0. knuth.

External links Open Data Structures - Section 5.1.1 - Multiplicative Hashing (http://opendatastructures.org/versions/ edition-0.1d/ods-java/node31.html#SECTION00811000000000000000)

K-independent hashing

145

K-independent hashingA family of hash functions is said to be -independent or -universal[1] if selecting a hash function at random keys are independent random variables (see from the family guarantees that the hash codes of any designated precise mathematical definitions below). Such families allow good average case performance in randomized algorithms or data structures, even if the input data is chosen by an adversary. The trade-offs between the degree of independence and the efficiency of evaluating the hash function are well studied, and many -independent families have been proposed.

IntroductionThe goal of hashing is usually to map keys from some large domain (universe) bins (labelled into a smaller range, such as ). In the analysis of randomized algorithms and data structures, it is often

desirable for the hash codes of various keys to "behave randomly". For instance, if the hash code of each key were an independent random choice in , the number of keys per bin could be analyzed using the Chernoff bound. A deterministic hash function cannot offer any such guarantee in an adversarial setting, as the adversary may choose the keys to be the precisely the preimage of a bin. Furthermore, a deterministic hash function does not allow for rehashing: sometimes the input data turns out to be bad for the hash function (e.g. there are too many collisions), so one would like to change the hash function. The solution to these problems is to pick a function randomly from a large family of hash functions. The randomness in choosing the hash function can be used to guarantee some desired random behavior of the hash codes of any keys of interest. The first definition along these lines was universal hashing, which guarantees a low collision probability for any two designated keys. The concept of -independent hashing, introduced by Wegman and Carter in 1981,[2] strengthens the guarantees of random behavior to families of uniform distribution of hash codes. designated keys, and adds a guarantee on the

Mathematical DefinitionsThe strictest definition, introduced by Wegman and Carter[2] under the name "strongly universal hash family", is the following. A family of hash functions is -independent if for any distinct keys and any hash codes (not necessarily distinct) , we have:

This definition is equivalent to the following two conditions: 1. for any fixed , as is drawn randomly from , as , is uniformly distributed in , . are 2. for any fixed, distinct keys is drawn randomly from

independent random variables. Often it is inconvenient to achieve the perfect joint probability of may define a distinct -independent family to satisfy: and

due to rounding issues. Following,[3] one ,

Observe that, even if

is close to 1,

are no longer independent random variables, which is often a problem

in the analysis of randomized algorithms. Therefore, a more common alternative to dealing with rounding issues is to prove that the hash family is close in statistical distance to a -independent family, which allows black-box use of the independence properties.

K-independent hashing

146

References[1] Cormen, Thomas H.; Leiserson, Charles E.; Rivest, Ronald L.; Stein, Clifford (2009). Introduction to Algorithms (3rd ed.). MIT Press. ISBN0-262-03384-4. [2] Wegman, Mark N.; Carter, J. Lawrence (1981). "New hash functions and their use in authentication and set equality" (http:/ / www. fi. muni. cz/ ~xbouda1/ teaching/ 2009/ IV111/ Wegman_Carter_1981_New_hash_functions. pdf). Journal of Computer and System Sciences 22 (3): 265279. doi:10.1016/0022-0000(81)90033-7. Conference version in FOCS'79. . Retrieved 9 February 2011. [3] Siegel, Alan (2004). "On universal classes of extremely random constant-time hash functions and their time-space tradeoff" (http:/ / www. cs. nyu. edu/ faculty/ siegel/ FASTH. pdf). SIAM Journal on Computing 33 (3): 505543. Conference version in FOCS'89. .

Further reading Motwani, Rajeev; Raghavan, Prabhakar (1995). Randomized Algorithms. Cambridge University Press. p.221. ISBN0-521-47465-5.

Tabulation hashingIn computer science, tabulation hashing is a method for constructing universal families of hash functions by combining table lookup with exclusive or operations. It is simple and fast enough to be usable in practice, and has theoretical properties that (in contrast to some other universal hashing methods) make it usable with linear probing, cuckoo hashing, and the MinHash technique for estimating the size of set intersections. The first instance of tabulation hashing is Zobrist hashing (1969). It was later rediscovered by Carter & Wegman (1979) and studied in more detail by Ptracu & Thorup (2011).

MethodLet p denote the number of bits in a key to be hashed, and q denote the number of bits desired in an output hash function. Let r be a number smaller than p, and let t be the smallest integer that is at least as large as p/r. For instance, if r=8, then an r-bit number is a byte, and t is the number of bytes per key. The key idea of tabulation hashing is to view a key as a vector of t r-bit numbers, use a lookup table filled with random values to compute a hash value for each of the r-bit numbers representing a given key, and combine these values with the bitwise binary exclusive or operation. The choice of t and r should be made in such a way that this table is not too large; e.g., so that it fits into the computer's cache memory. The initialization phase of the algorithm creates a two-dimensional array T of dimensions 2r by t, and fills the array with random numbers. Once the array T is initialized, it can be used to compute the hash value h(x) of any given key x. To do so, partition x into r-bit values, where x0 consists of the low order r bits of x, x1 consists of the next r bits, etc. (E.g., again, with r=8, xi is just the ith byte of x). Then, use these values as indices into T and combine them with the exclusive or operation: h(x) = T[x0,0] T[x1,1] T[x2,2] ...

UniversalityCarter & Wegman (1979) define a randomized scheme for generating hash functions to be universal if, for any two keys, the probability that they collide (that is, they are mapped to the same value as each other) is 1/m, where m is the number of values that the keys can take on. They defined a stronger property in the subsequent paper Wegman & Carter (1981): a randomized scheme for generating hash functions is k-independent if, for every k-tuple of keys, and each possible k-tuple of values, the probability that those keys are mapped to those values is 1/mk. 2-independent hashing schemes are automatically universal, and any universal hashing scheme can be converted into a 2-independent scheme by storing a random number x in the initialization phase of the algorithm and adding x to each

Tabulation hashing hash value, so universality is essentially the same as 2-independence, but k-independence for larger values of k is a stronger property, held by fewer hashing algorithms. As Ptracu & Thorup (2011) observe, tabulation hashing is 3-independent but not 4-independent. For any single key x, T[x0,0] is equally likely to take on any hash value, and the exclusive or of T[x0,0] with the remaining table values does not change this property. For any two keys x and y, x is equally likely to be mapped to any hash value as before, and there is at least one position i where xixi; the table value T[yi,i] is used in the calculation of h(y) but not in the calculation of h(x), so even after the value of h(x) has been determined, h(y) is equally likely to be any valid hash value. Similarly, for any three keys x, y, and z, at least one of the three keys has a position i where its value zi differs from the other two, so that even after the values of h(x) and h(z) are determined, h(z) is equally likely to be any valid hash value. However, this reasoning breaks down for four keys because there are sets of keys w, x, y, and z where none of the four has a byte value that it does not share with at least one of the other keys. For instance, if the keys have two bytes each, and w, x, y, and z are the four keys that have either zero or one as their byte values, then each byte value in each position is shared by exactly two of the four keys. For these four keys, the hash values computed by tabulation hashing will always satisfy the equation h(w) h(x) h(y) h(z) = 0, whereas for a 4-independent hashing scheme the same equation would only be satisfied with probability 1/m. Therefore, tabulation hashing is not 4-independent. Siegel (2004) uses the same idea of using exclusive or operations to combine random values from a table, with a more complicated algorithm based on expander graphs for transforming the key bits into table indices, to define hashing schemes that are k-independent for any constant or even logarithmic value of k. However, the number of table lookups needed to compute each hash value using Siegel's variation of tabulation hashing, while constant, is still too large to be practical, and the use of expanders in Siegel's technique also makes it not fully constructive. One limitation of tabulation hashing is that it assumes that the input keys have a fixed number of bits. Lemire (2012) has studied variations of tabulation hashing that can be applied to variable-length strings, and shown that they can be universal (2-independent) but not 3-independent.

147

ApplicationBecause tabulation hashing is a universal hashing scheme, it can be used in any hashing-based algorithm in which universality is sufficient. For instance, in hash chaining, the expected time per operation is proportional to the sum of collision probabilities, which is the same for any universal scheme as it would be for truly random hash functions, and is constant whenever the load factor of the hash table is constant. Therefore, tabulation hashing can be used to compute hash functions for hash chaining with a theoretical guarantee of constant expected time per operation.[1] However, universal hashing is not strong enough to guarantee the performance of some other hashing algorithms. For instance, for linear probing, 5-independent hash functions are strong enough to guarantee constant time operation, but there are 4-independent hash functions that fail.[2] Nevertheless, despite only being 3-independent, tabulation hashing provides the same constant-time guarantee for linear probing.[3] Cuckoo hashing, another technique for implementing hash tables, guarantees constant time per lookup (regardless of the hash function). Insertions into a cuckoo hash table may fail, causing the entire table to be rebuilt, but such failures are sufficiently unlikely that the expected time per insertion (using either a truly random hash function or a hash function with logarithmic independence) is constant. With tabulation hashing, on the other hand, the best bound known on the failure probability is higher, high enough that insertions cannot be guaranteed to take constant expected time. Nevertheless, tabulation hashing is adequate to ensure the linear-expected-time construction of a cuckoo hash table for a static set of keys that does not change as the table is used.[3] Algorithms such as Karp-Rabin requires the efficient computation of hashing all consecutive sequences of characters. We typically use rolling hash functions for these problems. Tabulation hashing is used to construct families of strongly universal functions (for example, hashing by cyclic polynomials).

Tabulation hashing

148

Notes[1] Carter & Wegman (1979). [2] For the sufficiency of 5-independent hashing for linear probing, see Pagh, Pagh & Rui (2009). For examples of weaker hashing schemes that fail, see Ptracu & Thorup (2010). [3] Ptracu & Thorup (2011).

References Carter, J. Lawrence; Wegman, Mark N. (1979), "Universal classes of hash functions", Journal of Computer and System Sciences 18 (2): 143154, doi:10.1016/0022-0000(79)90044-8, MR532173. Lemire, Daniel (2012), "The universality of iterated hashing over variable-length strings", Discrete Applied Mathematics 160: 604617, arXiv:1008.1715, doi:10.1016/j.dam.2011.11.009. Pagh, Anna; Pagh, Rasmus; Rui, Milan (2009), "Linear probing with constant independence", SIAM Journal on Computing 39 (3): 11071120, doi:10.1137/070702278, MR2538852. Ptracu, Mihai; Thorup, Mikkel (2010), "On the k-independence required by linear probing and minwise independence" (http://people.csail.mit.edu/mip/papers/kwise-lb/kwise-lb.pdf), Automata, Languages and Programming, 37th International Colloquium, ICALP 2010, Bordeaux, France, July 6-10, 2010, Proceedings, Part I, Lecture Notes in Computer Science, 6198, Springer, pp.715726, doi:10.1007/978-3-642-14165-2_60. Ptracu, Mihai; Thorup, Mikkel (2011), "The power of simple tabulation hashing", Proceedings of the 43rd annual ACM Symposium on Theory of Computing (STOC '11), pp.110, arXiv:1011.5200, doi:10.1145/1993636.1993638. Siegel, Alan (2004), "On universal classes of extremely random constant-time hash functions", SIAM Journal on Computing 33 (3): 505543 (electronic), doi:10.1137/S0097539701386216, MR2066640. Wegman, Mark N.; Carter, J. Lawrence (1981), "New hash functions and their use in authentication and set equality", Journal of Computer and System Sciences 22 (3): 265279, doi:10.1016/0022-0000(81)90033-7, MR633535.

Cryptographic hash function

149

Cryptographic hash functionA cryptographic hash function is a hash function, that is, an algorithm that takes an arbitrary block of data and returns a fixed-size bit string, the (cryptographic) hash value, such that an (accidental or intentional) change to the data will (with very high probability) change the hash value. The data to be encoded is often called the "message," and the hash value is sometimes called the message digest or simply digest. The ideal cryptographic hash function has four main or significant properties: it is easy to compute the hash value for any given message it is infeasible to generate a message that has a given hashA cryptographic hash function (specifically, SHA-1) at work. Note that even small changes in the source input (here in the word "over") drastically change the resulting output, by the so-called avalanche effect.

it is infeasible to modify a message without changing the hash it is infeasible to find two different messages with the same hash Cryptographic hash functions have many information security applications, notably in digital signatures, message authentication codes (MACs), and other forms of authentication. They can also be used as ordinary hash functions, to index data in hash tables, for fingerprinting, to detect duplicate data or uniquely identify files, and as checksums to detect accidental data corruption. Indeed, in information security contexts, cryptographic hash values are sometimes called (digital) fingerprints, checksums, or just hash values, even though all these terms stand for functions with rather different properties and purposes.

PropertiesMost cryptographic hash functions are designed to take a string of any length as input and produce a fixed-length hash value. A cryptographic hash function must be able to withstand all known types of cryptanalytic attack. As a minimum, it must have the following properties: Preimage resistance Given a hash it should be difficult to find any message such that . This concept is

related to that of one-way function. Functions that lack this property are vulnerable to preimage attacks. Second-preimage resistance Given an input it should be difficult to find another input where such that

. This property is sometimes referred to as weak collision resistance, and functions that lack this property are vulnerable to second-preimage attacks. Collision resistance

Cryptographic hash function It should be difficult to find two different messages and such that . Such

150

a pair is called a cryptographic hash collision. This property is sometimes referred to as strong collision resistance. It requires a hash value at least twice as long as that required for preimage-resistance, otherwise collisions may be found by a birthday attack. These properties imply that a malicious adversary cannot replace or modify the input data without changing its digest. Thus, if two strings have the same digest, one can be very confident that they are identical. A function meeting these criteria may still have undesirable properties. Currently popular cryptographic hash functions are vulnerable to length-extension attacks: given and but not , by choosing a suitable an attacker can calculate where denotes concatenation. This property can be used to break naive authentication schemes based on hash functions. The HMAC construction works around these problems. Ideally, one may wish for even stronger conditions. It should be impossible for an adversary to find two messages with substantially similar digests; or to infer any useful information about the data, given only its digest. Therefore, a cryptographic hash function should behave as much as possible like a random function while still being deterministic and efficiently computable. Checksum algorithms, such as CRC32 and other cyclic redundancy checks, are designed to meet much weaker requirements, and are generally unsuitable as cryptographic hash functions. For example, a CRC was used for message integrity in the WEP encryption standard, but an attack was readily discovered which exploited the linearity of the checksum.

Degree of difficultyIn cryptographic practice, difficult generally means almost certainly beyond the reach of any adversary who must be prevented from breaking the system for as long as the security of the system is deemed important. The meaning of the term is therefore somewhat dependent on the application, since the effort that a malicious agent may put into the task is usually proportional to his expected gain. However, since the needed effort usually grows very quickly with the digest length, even a thousand-fold advantage in processing power can be neutralized by adding a few dozen bits to the latter. In some theoretical analyses difficult has a specific mathematical meaning, such as not solvable in asymptotic polynomial time. Such interpretations of difficulty are important in the study of provably secure cryptographic hash functions but do not usually have a strong connection to practical security. For example, an exponential time algorithm can sometimes still be fast enough to make a feasible attack. Conversely, a polynomial time algorithm (e.g., one that requires n20 steps for n-digit keys) may be too slow for any practical use.

IllustrationAn illustration of the potential use of a cryptographic hash is as follows: Alice poses a tough math problem to Bob, and claims she has solved it. Bob would like to try it himself, but would yet like to be sure that Alice is not bluffing. Therefore, Alice writes down her solution, appends a random nonce, computes its hash and tells Bob the hash value (whilst keeping the solution and nonce secret). This way, when Bob comes up with the solution himself a few days later, Alice can prove that she had the solution earlier by revealing the nonce to Bob. (This is an example of a simple commitment scheme; in actual practice, Alice and Bob will often be computer programs, and the secret would be something less easily spoofed than a claimed puzzle solution).

Cryptographic hash function

151

ApplicationsVerifying the integrity of files or messagesAn important application of secure hashes is verification of message integrity. Determining whether any changes have been made to a message (or a file), for example, can be accomplished by comparing message digests calculated before, and after, transmission (or any other event). For this reason, most digital signature algorithms only confirm the authenticity of a hashed digest of the message to be "signed." Verifying the authenticity of a hashed digest of the message is considered proof that the message itself is authentic. A related application is password verification. Passwords are usually not stored in cleartext, to improve security, but instead in digest form. To authenticate a user, the password presented by the user is hashed and compared with the stored hash. This also means that the original passwords cannot be retrieved if forgotten or lost, and they'll have to be replaced with new ones.

File or data identifierA message digest can also serve as a means of reliably identifying a file; several source code management systems, including Git, Mercurial and Monotone, use the sha1sum of various types of content (file content, directory trees, ancestry information, etc.) to uniquely identify them. Hashes are used to identify files on peer-to-peer filesharing networks. For example, in an ed2k link, an MD4-variant hash is combined with the file size, providing sufficient information for locating file sources, downloading the file and verifying its contents. Magnet links are another example. Such file hashes are often the top hash of a hash list or a hash tree which allows for additional benefits. One of the main applications of a hash function is to allow the fast look-up of a data in a hash table. Being hash functions of a particular kind, cryptographic hash functions lend themselves well to this application too. However, compared with standard hash functions, cryptographic hash functions tend to be much more expensive computationally. For this reason, they tend to be used in contexts where it is necessary for users to protect themselves against the possibility of forgery (the creation of data with the same digest as the expected data) by potentially malicious participants.

Pseudorandom generation and key derivationHash functions can also be used in the generation of pseudorandom bits, or to derive new keys or passwords from a single, secure key or password.

Hash functions based on block ciphersThere are several methods to use a block cipher to build a cryptographic hash function, specifically a one-way compression function. The methods resemble the block cipher modes of operation usually used for encryption. All well-known hash functions, including MD4, MD5, SHA-1 and SHA-2 are built from block-cipher-like components designed for the purpose, with feedback to ensure that the resulting function is not bijective. SHA-3 finalists include functions with block-cipher-like components (e.g., Skein, BLAKE) and functions based on other designs (e.g., JH, Keccak). A standard block cipher such as AES can be used in place of these custom block ciphers; that might be useful when an embedded system needs to implement both encryption and hashing with minimal code size or hardware area. However, that approach can have costs in efficiency and security. The ciphers in hash functions are built for hashing: they use large keys and blocks, can efficiently change keys every block, and have been designed and vetted for resistance to related-key attacks. General-purpose ciphers tend to have different design goals. In particular, AES has key and block sizes that make it nontrivial to use to generate long hash values; AES encryption becomes less

Cryptographic hash function efficient when the key changes each block; and related-key attacks make it potentially less secure for use in a hash function than for encryption.

152

MerkleDamgrd constructionA hash function must be able to process an arbitrary-length message into a fixed-length output. This can be achieved by breaking the input up into a series of equal-sized blocks, and operating on them in sequence using a one-way compression function. The compression function can either be specially designed for hashing or be built from a block cipher. A hash The MerkleDamgrd hash construction. function built with the MerkleDamgrd construction is as resistant to collisions as is its compression function; any collision for the full hash function can be traced back to a collision in the compression function. The last block processed should also be unambiguously length padded; this is crucial to the security of this construction. This construction is called the MerkleDamgrd construction. Most widely used hash functions, including SHA-1 and MD5, take this form. The construction has certain inherent flaws, including length-extension and generate-and-paste attacks, and cannot be parallelized. As a result, many entrants in the current NIST hash function competition are built on different, sometimes novel, constructions.

Use in building other cryptographic primitivesHash functions can be used to build other cryptographic primitives. For these other primitives to be cryptographically secure, care must be taken to build them correctly. Message authentication codes (MACs) (also called keyed hash functions) are often built from hash functions. HMAC is such a MAC. Just as block ciphers can be used to build hash functions, hash functions can be used to build block ciphers. Luby-Rackoff constructions using hash functions can be provably secure if the underlying hash function is secure. Also, many hash functions (including SHA-1 and SHA-2) are built by using a special-purpose block cipher in a Davies-Meyer or other construction. That cipher can also be used in a conventional mode of operation, without the same security guarantees. See SHACAL, BEAR and LION. Pseudorandom number generators (PRNGs) can be built using hash functions. This is done by combining a (secret) random seed with a counter and hashing it. Some hash functions, such as Skein, Keccak, and RadioGatn output an arbitrarily long stream and can be used as a stream cipher, and stream ciphers can also be built from fixed-length digest hash functions. Often this is done by first building a cryptographically secure pseudorandom number generator and then using its stream of random bytes as keystream. SEAL is a stream cipher that uses SHA-1 to generate internal tables, which are then used in a keystream generator more or less unrelated to the hash algorithm. SEAL is not guaranteed to be as strong (or weak) as SHA-1.

Cryptographic hash function

153

Concatenation of cryptographic hash functionsConcatenating outputs from multiple hash functions provides collision resistance as good as the strongest of the algorithms included in the concatenated result. For example, older versions of TLS/SSL use concatenated MD5 and SHA-1 sums; that ensures that a method to find collisions in one of the functions doesn't allow forging traffic protected with both functions. For Merkle-Damgrd hash functions, the concatenated function is as collision-resistant as its strongest component,[1] but not more collision-resistant.[2] Joux[3] noted that 2-collisions lead to n-collisions: if it is feasible to find two messages with the same MD5 hash, it is effectively no more difficult to find as many messages as the attacker desires with identical MD5 hashes. Among the n messages with the same MD5 hash, there is likely to be a collision in SHA-1. The additional work needed to find the SHA-1 collision (beyond the exponential birthday search) is polynomial. This argument is summarized by Finney [4]. A more current paper and full proof of the security of such a combined construction gives a clearer and more complete explanation of the above.[5]

Cryptographic hash algorithmsThere is a long list of cryptographic hash functions, although many have been found to be vulnerable and should not be used. Even if a hash function has never been broken, a successful attack against a weakened variant thereof may undermine the experts' confidence and lead to its abandonment. For instance, in August 2004 weaknesses were found in a number of hash functions that were popular at the time, including SHA-0, RIPEMD, and MD5. This has called into question the long-term security of later algorithms which are derived from these hash functions in particular, SHA-1 (a strengthened version of SHA-0), RIPEMD-128, and RIPEMD-160 (both strengthened versions of RIPEMD). Neither SHA-0 nor RIPEMD are widely used since they were replaced by their strengthened versions. As of 2009, the two most commonly used cryptographic hash functions are MD5 and SHA-1. However, MD5 has been broken; an attack against it was used to break SSL in 2008.[6] The SHA-0 and SHA-1 hash functions were developed by the NSA. In February 2005, a successful attack on SHA-1 was reported, finding collisions in about 269 hashing operations, rather than the 280 expected for a 160-bit hash function. In August 2005, another successful attack on SHA-1 was reported, finding collisions in 263 operations. Theoretical weaknesses of SHA-1 exist as well,[7][8] suggesting that it may be practical to break within years. New applications can avoid these problems by using more advanced members of the SHA family, such as SHA-2, or using techniques such as randomized hashing[9][10] that do not require collision resistance. However, to ensure the long-term robustness of applications that use hash functions, there is a competition to design a replacement for SHA-2, which will be given the name SHA-3 and become a FIPS standard around 2012.[11] Some of the following algorithms are used often in cryptography; consult the article for each specific algorithm for more information on the status of each algorithm. Note that this list does not include candidates in the current NIST hash function competition. For additional hash functions see the box at the bottom of the page.

Cryptographic hash function

154

Algorithm

Output size (bits)

Internal state size 256 256 384 128 128 8,736 58 words

Block size 256 1,024 128 512 512 256 3 words

Length size 256 64 64 64 -

Word size 32 32 32 32 32 32 164

Collision attacks (complexity) Yes ( 2105 Yes Yes ( 263.3 Yes ( 3 [13] ) ) ) [12] )

Preimage attacks (complexity) Yes ( 2192 [12] )

GOST HAVAL MD2 MD4 MD5 PANAMA RadioGatn

256 256/224/192/160/128 128 128 128 256 Up to 608/1,216 (19 words) 128 128/256 160/320 160 160 256/224 512/384 192/160/128 512

Yes ( 273

[14]

) ) )

[15]

Yes ( 278.4 Yes ( 2123.4

[16]

Yes ( 220.96 Yes

[17]

[18]

With flaws ( 2352 or [19] 2704 ) Yes ( 218 No No Yes ( 233.6 Yes ( 251 No No Yes ( 262:19 No [22] ) [20] ) No No No Yes ( 2184.3 No [16] ) [13] )

RIPEMD RIPEMD-128/256 RIPEMD-160/320 SHA-0 SHA-1 SHA-256/224 SHA-512/384 Tiger(2)-192/160/128 WHIRLPOOL

128 128/256 160/320 160 160 256 512 192 512

512 512 512 512 512 512 1,024 512 512

64 64 64 64 64 64 128 64 256

32 32 32 32 32 32 64 64 8

[21]

)

Note: The internal state here means the "internal hash sum" after each compression of a data block. Most hash algorithms also internally use some additional variables such as length of the data compressed so far since that is needed for the length padding in the end. See the Merkle-Damgrd construction for details.

References[1] Note that any two messages that collide the concatenated function also collide each component function, by the nature of concatenation. For example, if concat(sha1(message1), md5(message1)) == concat(sha1(message2), md5(message2)) then sha1(message1) == sha1(message2) and md5(message1)==md5(message2). The concatenated function could have other problems that the strongest hash lacks -- for example, it might leak information about the message when the strongest component does not, or it might be detectably nonrandom when the strongest component is not -- but it can't be less collision-resistant. [2] More generally, if an attack can produce a collision in one hash function's internal state, attacking the combined construction is only as difficult as a birthday attack against the other function(s). For the detailed argument, see the Joux and Finney references that follow. [3] Antoine Joux. Multicollisions in Iterated Hash Functions. Application to Cascaded Constructions. LNCS 3152/2004, pages 306-316 Full text (http:/ / www. springerlink. com/ index/ DWWVMQJU0N0A3UGJ. pdf). [4] http:/ / article. gmane. org/ gmane. comp. encryption. general/ 5154 [5] Jonathan J. Hoch and Adi Shamir (2008-02-20). On the Strength of the Concatenated Hash Combiner when all the Hash Functions are Weak (http:/ / eprint. iacr. org/ 2008/ 075. pdf). . [6] Alexander Sotirov, Marc Stevens, Jacob Appelbaum, Arjen Lenstra, David Molnar, Dag Arne Osvik, Benne de Weger, MD5 considered harmful today: Creating a rogue CA certificate (http:/ / www. win. tue. nl/ hashclash/ rogue-ca/ ), accessed March 29, 2009 [7] Xiaoyun Wang, Yiqun Lisa Yin, and Hongbo Yu, Finding Collisions in the Full SHA-1 (http:/ / people. csail. mit. edu/ yiqun/ SHA1AttackProceedingVersion. pdf) [8] Bruce Schneier, Cryptanalysis of SHA-1 (http:/ / www. schneier. com/ blog/ archives/ 2005/ 02/ cryptanalysis_o. html) (summarizes Wang et al. results and their implications) [9] Shai Ha