Turing, Halting Theorem, P, NP, NPC, Hamiltonian, TSP, SAT, Subset Sum….

Dr. Mustafa Sakalli Dr. Mustafa Sakalli Marmara Univ. Marmara Univ.

Very Preliminary Very Preliminary slights to be slights to be

modified, June modified, June of 2009, CS246 of 2009, CS246

CLRS chapter 34 CLRS chapter 34

A Turing MachineSource: Prof Michael Chan’s Discrete Math Notes.

• A Turing machine (only on mind) is a modified or extended version of finite state machine.

• Imagine a machine M, scanning along a tape which is infinitely long. Tape is divided into boxes, each of which is marked with an element of a finite alphabet A. M can be in any of states S. S has a finite number of states.

• Depending on the element M reads and he state it is in, following happens:– M either leaves or replaces the element it reads in the current block.– Then, it moves either direction, backward or forward to the adjacent blocks.– The M either keeps its state or changes to one of the other state states.

States vs elements of A.

A Touring Machine• Example: A={0, 1, 2, 3, 4}, S={.., 5, …}

– suppose machine is in state 5, – the element M reads in current tape position is 3, – and the table given below describes the behavior machine that will take, 1L2

says replace 3 by 1, shift one left and change the state to 2. • Convention: aDs Convention: aDs

– a a {A} any value to be inserted into the current position, in binary case {A} any value to be inserted into the current position, in binary case A={0, 1}. A={0, 1}.

– D D {L, R}: Direction shift by 1 either L or R. One step at a time. {L, R}: Direction shift by 1 either L or R. One step at a time. – ss {0, 1, …, n}: a new state to be taken (predetermined number of n {0, 1, …, n}: a new state to be taken (predetermined number of n

states). states). States vs elements of A.

• After subsequent actions machine takes machine may never halt.

• However it is possible to force the machine into halt by sending TM into a non-existent state.

And it halts here..And it halts here..Given below, starting in a different situation, then TM does not go into halt.Given below, starting in a different situation, then TM does not go into halt.

• States s{0..k-1}, without halting. State k, is allocated for the halting state.• Elements binary: A={0, 1}, and this represents natural numbers in the unary

system, it means that a tape consisting entirely of 0's will represent the integer 0, while a string of n 1's will represent a positive integer n. Numbers are also interspaced by single 0's. For example, the string …01111110111111111101001111110101110… will represent the sequence … 6; 10; 1; 0; 6; 1; 3… . here consecutive zeros equal to 0 (therefore two zeros in between denote 0..

• If there is any bit alive (1) in the sequence, then convention is that TM starts at the left most 1.

• Example. Design a Turing machine to compute the function f(n) = 1 for every n N {0}. Solution aimed in here is switching situation

01111100011100 into 00000010000010.. And starting from {s, a} = {0, 1}..

Designing a k-state TM

to the situation

• Another example f(n)=n+1,Another example f(n)=n+1, Walks n step backward after completing n+1Walks n step backward after completing n+1stst digit.. digit..

• Check the document for to add to numbers f(n) = n + m Check the document for to add to numbers f(n) = n + m for for every every n n N N { {0}0}. .

• Merging two TMs. Suppose two machines with states {0, …, k1-Merging two TMs. Suppose two machines with states {0, …, k1-1, k1 (halting state)}, and {0, …, k2-1, k2 (halting state)}, and the 1, k1 (halting state)}, and {0, …, k2-1, k2 (halting state)}, and the same alphabets. same alphabets.

• Merging two to M1M2, then new states: {0, …, k1, k1+1, …, Merging two to M1M2, then new states: {0, …, k1, k1+1, …, k1+k2}.. k1+k2}..

• The number of instruction in binary case, for n states, 2n, and the The number of instruction in binary case, for n states, 2n, and the number of choices for any particular instruction is (4n+1), then the number of choices for any particular instruction is (4n+1), then the number of different Turing machines is (4(n+1))number of different Turing machines is (4(n+1))2n2n, so a finite , so a finite collection of TMs…collection of TMs…

• A sub collection Hn of Tn will halt if starts from a blank tape. So, A sub collection Hn of Tn will halt if starts from a blank tape. So, Hn is a finite collection, therefore Hn is a finite collection, therefore theoretically possible to count theoretically possible to count the steps each Turing Machine takes before it goes into haltthe steps each Turing Machine takes before it goes into halt. .

• B(M) = The number of steps M takes, before it halts, then the function B(M) = The number of steps M takes, before it halts, then the function BBM(n)=BBM(n)=(n) = max(n) = maxMMHHnn BB((MM), ),

is known as the busy beaver problem, which is to determine a computing is known as the busy beaver problem, which is to determine a computing program which will give value program which will give value (n) for every n (n) for every n N. N.

• They are 5-Tuple Turing MachinesThey are 5-Tuple Turing Machines

• Deterministic Turing machines Deterministic Turing machines

• All machines are started on initially blank tapesAll machines are started on initially blank tapes

• A Turing Machine that writes the maximum number of 1’s for its number of state A Turing Machine that writes the maximum number of 1’s for its number of state ∑(n), Rado’s function, which is the maximum number of 1’s left on the tape. ∑(n), Rado’s function, which is the maximum number of 1’s left on the tape. S(n) maximum number of moves that can be made by n-state halting Turing S(n) maximum number of moves that can be made by n-state halting Turing Machine.Machine.

• The search state is extremely large

• Not possible to determine whether a particular Turing machine will halt.

• Neither ∑(n) or S(n) are computable functions.• A problem is Computable if it can be computed by a Turing Machine.

• If the halting problem were solvable then the busy beaver problem would be solvable.

• Logically impossible to have a computing program for every n for which the Logically impossible to have a computing program for every n for which the function function (n) is computable. Therefore it is non-computable. (n) is computable. Therefore it is non-computable. Note that for a Note that for a particular n particular n N, possible to compute N, possible to compute (n) but what is not possible is that there (n) but what is not possible is that there cannot exist one computing program that applicable for every case of cannot exist one computing program that applicable for every case of (n).(n).

Three State Machine

∆ ∆ ∆ ∆ ∆ ∆ ∆ ∆∆ ∆ ∆ ∆… …1 111 1 1

Step State Function

1 A ∆/1, R

2 B ∆/1, L

3 A 1/1, L

4 C ∆/1, L

5 B ∆/1, L

6 A ∆/1, R

7 B 1/1, R

8 B 1/1, R

9 B 1/1, R

10 B 1/1, R

11 B ∆/1, L

12 A 1/1, L

13 C 1/1, R

H

Busy Beaver ProblemBusy Beaver Problem

0  State 0 10  State 1 11  State 0011  State 0111  State 1101  State 2

Two State Machine

• ∑∑(n): Is non computable(n): Is non computable– Proof:Proof: ∑(n+1) > ∑(n) for all n. ∑(n+1) > ∑(n) for all n. Observe that the function Observe that the function (n): N(n): NN is strictly N is strictly

increasing. increasing. (n+1)>(n+1)>(n), Suppose (n), Suppose MMHHnn, satisfies B(n) = , satisfies B(n) = (n), for which n is halting (n), for which n is halting state. Now consider using state. Now consider using MM to construct another machine to construct another machine M’M’HHn+1.n+1. Adding one more Adding one more state of instructions, in the table, then the halting state will be n+1 to make new state of instructions, in the table, then the halting state will be n+1 to make new MM’ to ’ to halt. halt. We replace halting state adding another intermediate stateWe replace halting state adding another intermediate state

– So B(So B(MM’)=’)=(n)+1 (n)+1 B( B(MM’)’)(n+1), which follows (n+1), which follows (n)<(n)<(n+1).(n+1).

– Second step, any computing program on a computer can be describes in terms of a Second step, any computing program on a computer can be describes in terms of a TM. But No a TM can exist for every nTM. But No a TM can exist for every nN states N states (n). (n).

– Thirs step is compare a collection of Turing Machines Thirs step is compare a collection of Turing Machines

1. Suppose a BB, M1. Suppose a BB, M11 of f(n)=n+1, M of f(n)=n+1, M11 has 2 states.. halts in 2 steps!! has 2 states.. halts in 2 steps!!

2. Another BB, M2. Another BB, M22 of f(n)=2n, M of f(n)=2n, M22 has 9 states and one more BB M has 9 states and one more BB M33 has k states. has k states.

3. Congregated Turing Machine M3. Congregated Turing Machine Mallall = M = M11(M(M22 i)M i)M33, , (2+9i+k), it halts after (2+9i+k), it halts after ∑(n) = ∑(n) = 2+9i+k 1’s.. 2+9i+k 1’s..

4. With i copies of TM, M4. With i copies of TM, M2 2 will halt after will halt after (2(2ii)!!!.)!!!.

From From MMallall (2(2ii))(2+9i+k).. (2+9i+k)..

However However expression 2+9i+k is linear in i, such that for sufficiently large i, 2expression 2+9i+k is linear in i, such that for sufficiently large i, 2 ii>>(2+9i+k) (2+9i+k) which contradicts to observation made in 3. which contradicts to observation made in 3.

• S(n): Is non computable. Proof: There exists a TM M with n states that writes S(n): Is non computable. Proof: There exists a TM M with n states that writes ∑(n) 1’s on its tape before halting. Such a TM must make at least ∑(n) moves. ∑(n) 1’s on its tape before halting. Such a TM must make at least ∑(n) moves. Hence S(n) ≥ ∑(n). Hence S(n) ≥ ∑(n).

Busy Beaver ProblemBusy Beaver Problem

P=NP?P=NP?• First four pages is the summary of the article “NP completeness of First four pages is the summary of the article “NP completeness of

Minesweeper” by Ian Stewart. Minesweeper” by Ian Stewart.

• An algorithm - a procedure solving some problem - in the An algorithm - a procedure solving some problem - in the computational term: how efficient - the running time - the number computational term: how efficient - the running time - the number of clock time required to get the answer ~ the initial data? of clock time required to get the answer ~ the initial data?

• Theoretically the main distinction between problems that are of Theoretically the main distinction between problems that are of type P - polynomial time- and those that are not. type P - polynomial time- and those that are not.

• A problem is of type PA problem is of type P if it can be solved using an algorithm if it can be solved using an algorithm whose running time grows no faster than some fixed power of the whose running time grows no faster than some fixed power of the number of symbols required to specify the initial data. Problems of number of symbols required to specify the initial data. Problems of this type are easy to solve.this type are easy to solve.

• Otherwise the problem is non-P. Non-P problems are hardOtherwise the problem is non-P. Non-P problems are hard. Of . Of course it's not quite as simple as that, but it's a good rule of thumb. course it's not quite as simple as that, but it's a good rule of thumb.

• Intuitively, problems in P can be solved efficiently, whereas non-P Intuitively, problems in P can be solved efficiently, whereas non-P problems cannot be solved algorithmically in any practical manner problems cannot be solved algorithmically in any practical manner because any algorithm will take a ridiculously long time to get an because any algorithm will take a ridiculously long time to get an answer. answer.

• If a problem is of type P, possible to exhibit it in polynomial time. If a problem is of type P, possible to exhibit it in polynomial time. For example, sorting a list of numbers or and searching a string For example, sorting a list of numbers or and searching a string for some sequence of symbols, “which is why commercial for some sequence of symbols, “which is why commercial databases can sort data; or word processors can carry out search-databases can sort data; or word processors can carry out search-and-replace operations”. and-replace operations”.

• In contrast, In contrast, Traveling Salesman Problem finds the shortest route Traveling Salesman Problem finds the shortest route whereby a salesman can visit every city on some itinerary is whereby a salesman can visit every city on some itinerary is believed to be non-Pbelieved to be non-P, , which has never been provedwhich has never been proved. .

• Hard to prove that a problem is non-PHard to prove that a problem is non-P? ? Because requires to Because requires to contemplate all possible algorithms, and show none of which can contemplate all possible algorithms, and show none of which can solve the problem in polynomial timesolve the problem in polynomial time. A mind-boggling task. . A mind-boggling task.

• The best that has been done to date is to prove that a broad class The best that has been done to date is to prove that a broad class of candidate non-P problems are all on the same footing, if any of candidate non-P problems are all on the same footing, if any one of them can be solved in polynomial time, then all can be. one of them can be solved in polynomial time, then all can be. The problems involved here are said to have 'nondeterministic The problems involved here are said to have 'nondeterministic polynomial' running time: type NPpolynomial' running time: type NP. .

• NP is not the same as non-P. NP is not the same as non-P. A problem is NP, for which if it is A problem is NP, for which if it is possible to check a solution in a polynomial timepossible to check a solution in a polynomial time. .

P=P=??NPNP• a jigsaw puzzle, if solved, usually quick to check whether the a jigsaw puzzle, if solved, usually quick to check whether the

solution is right, since the checking runs in polynomial time. But solution is right, since the checking runs in polynomial time. But solving the puzzle, trying every potential solution in turn and solving the puzzle, trying every potential solution in turn and checking each, checking each, the number of potential solutions grows much the number of potential solutions grows much faster than any fixed power of the number of piecesfaster than any fixed power of the number of pieces. .

• Many of NP problems have 'equivalent' running timesMany of NP problems have 'equivalent' running times. . • Specifically, an NP problem is said to be NP-complete if the Specifically, an NP problem is said to be NP-complete if the

existence of a polynomial time solution for that problem implies existence of a polynomial time solution for that problem implies that all (of its sub) NP problems have a polynomial time solution. that all (of its sub) NP problems have a polynomial time solution. Solving one in polynomial time, means all the other ones are in Solving one in polynomial time, means all the other ones are in polynomial timepolynomial time. .

• A vast range of problems are known to be A vast range of problems are known to be NP-completeNP-complete. The . The P=NP? problem asks whether types P and NP are (despite all P=NP? problem asks whether types P and NP are (despite all appearances to the contrary) the same. The expected answer is 'no'. appearances to the contrary) the same. The expected answer is 'no'. However, if any NP-complete problem turns out to be of type P — However, if any NP-complete problem turns out to be of type P — - having a polynomial time solution — - than NP must equal P. We - having a polynomial time solution — - than NP must equal P. We therefore expect all NP-complete problems to be non-P, not proven therefore expect all NP-complete problems to be non-P, not proven yet. yet.

• One simplest known NP-complete problem is SAT, the logical One simplest known NP-complete problem is SAT, the logical satisfiability of a Boolean condition. satisfiability of a Boolean condition.

• Boolean circuits are logic gates AND, OR and NOT. Each gate Boolean circuits are logic gates AND, OR and NOT. Each gate may accept a number of inputs, and outputs the logical value of may accept a number of inputs, and outputs the logical value of that combination. that combination.

• The SAT problem asks, for a given Boolean circuit, whether there The SAT problem asks, for a given Boolean circuit, whether there exist choices of inputs that produce the output T. May sound easy! exist choices of inputs that produce the output T. May sound easy! if a circuit contains a large/or a huge number of gates and inputs. if a circuit contains a large/or a huge number of gates and inputs.

• The link to the computer game comes The link to the computer game comes with the consistency with the consistency problem, for example consistency of the logical assertionproblem, for example consistency of the logical assertion. .

• Kaye proves that Minesweeper is equivalent to SAT, in the sense Kaye proves that Minesweeper is equivalent to SAT, in the sense that for a given Boolean circuit a SAT can be 'encoded' as a that for a given Boolean circuit a SAT can be 'encoded' as a Minesweeper Consistency Problem for some position in the game, Minesweeper Consistency Problem for some position in the game, using a code procedure that is running in using a code procedure that is running in polynomial timepolynomial time. .

• Therefore, Therefore, solving the Minesweeper Consistency Problem in solving the Minesweeper Consistency Problem in polynomial time, would mean solving the SAT problem for that polynomial time, would mean solving the SAT problem for that circuit in polynomial timecircuit in polynomial time. In other words, Minesweeper is NP-. In other words, Minesweeper is NP-complete.complete.

Side informatin on Algorithmic complexitySide informatin on Algorithmic complexity is some measure of is some measure of algorithmic length, “how complex a system is in terms of the algorithmic length, “how complex a system is in terms of the length of the shortest computer program, or set of algorithms”, to length of the shortest computer program, or set of algorithms”, to perform the computation. Rather information theoretical approach perform the computation. Rather information theoretical approach is taken, is taken, – Kolmogorov complexity. The complexity of a string x as the length of its Kolmogorov complexity. The complexity of a string x as the length of its

shortest description p on a universal Turing machine U shortest description p on a universal Turing machine U (K(x)=min{l(p):U(p)=x}). A string is simple if it can be described by a (K(x)=min{l(p):U(p)=x}). A string is simple if it can be described by a short program, like "the string of one million ones", and is complex if there short program, like "the string of one million ones", and is complex if there is no such short description, like for a random string whose shortest is no such short description, like for a random string whose shortest description is specifying it bit-by-bit. Independent of the choice of U, description is specifying it bit-by-bit. Independent of the choice of U, computer. Having similar properties with Shannon's entropy (information) computer. Having similar properties with Shannon's entropy (information) S, but K is superior!! to S in many respects!! it says. K is an excellent S, but K is superior!! to S in many respects!! it says. K is an excellent universal complexity measure, suitable e.g. for quantifying Occam's razor. universal complexity measure, suitable e.g. for quantifying Occam's razor. Entities should not be multiplied unnecessarilyEntities should not be multiplied unnecessarily William of Occam. William of Occam. Drawback is incomputable, approximated by MML/MDL or Lempel-Ziv Drawback is incomputable, approximated by MML/MDL or Lempel-Ziv compression or others. compression or others.

– Algorithmic Probability. . "Solomonoff“ takes a universal computer and Algorithmic Probability. . "Solomonoff“ takes a universal computer and randomly generate an input program. The program will compute some randomly generate an input program. The program will compute some possibly infinite output. The algorithmic probability of any given finite possibly infinite output. The algorithmic probability of any given finite output prefix q is the sum of the probabilities of the programs that compute output prefix q is the sum of the probabilities of the programs that compute something starting with q. Certain long objects with short programs have something starting with q. Certain long objects with short programs have high probability. Incomputable. high probability. Incomputable.

– Descriptional complexity. Descriptional complexity.

• Computational complexity rather deals with the running Computational complexity rather deals with the running time and space efficiencies. time and space efficiencies. CClassifies the problems by how lassifies the problems by how hard they are. hard they are. e.g. e.g. the problem of searching an ordered list the problem of searching an ordered list at at most has lgnmost has lgn time complexity. time complexity.– Universal Search complexity "Levin".Universal Search complexity "Levin". Time-bounded, Time-bounded, computablecomputable, ,

``Levin'' complexity penalizing a slow program by adding the log of its ``Levin'' complexity penalizing a slow program by adding the log of its running time to its lengthrunning time to its length. This leads to computable variants of AC and . This leads to computable variants of AC and AP. It solves all inversion problems in optimal (apart from some AP. It solves all inversion problems in optimal (apart from some multiplicative constant) time. multiplicative constant) time.

• Solving problems quickly, i.e., in close to linear time, or at least in some fraction of polynomial time as a function of the input size

• Evidence that many important problems can not be solved quickly.• NP-complete problems are hard problems.

– Use a heuristic: come up with a method for solving a reasonable fraction of the common cases.

– Solve approximately: come up with a solution that you can prove that is close to right.

– Use an exponential time solution: if you really have to solve the problem exactly and stop worrying about finding a better solution.

• Decision problems– Given an input and a question of a problem, determine if the answer is yes or no

• Optimization problems, np-hardness– Find a solution with the “best” value

• Optimization problems can be cast as decision problems that are easier to study– E.g.: Shortest path: for a G = unweighted directed

• Find a path between u and v that uses the fewest edges

• Does a path exist from u to v consisting of at most k edges?

• Class PClass P are called are called tractable. tractable. Consists of (decision) Consists of (decision) problems (solvable in polynomial time), performance is problems (solvable in polynomial time), performance is given i.e worst-case running time as given i.e worst-case running time as O(nO(nkk)), for some , for some constant k, O(nconstant k, O(n22), O(n), O(n33), O(1), O(n lg n) ), O(1), O(n lg n)

• Class non-PClass non-P Problems not in P are called Problems not in P are called intractableintractable or unsolvable, non-p examples O(2or unsolvable, non-p examples O(2nn), O(), O(nnnn), O(), O(nn!)!)– But can be solved in reasonable time only for small inputsBut can be solved in reasonable time only for small inputs– Or, can not be solved at all Or, can not be solved at all

• Do non-polynomial algorithms always perform worse Do non-polynomial algorithms always perform worse than polynomial algorithms?than polynomial algorithms?– nn1,000,0001,000,000 is is technicallytechnically tractable, but in reality impossible tractable, but in reality impossible – nnlog log log log log log nn is is technicallytechnically intractable, but in reality it is intractable, but in reality it is

possiblepossible

Tractable - IntractableTractable - Intractable

• Decision problems as sets of encodings of instances; the encodings that are in the set are "yes" instances.

DefinitionDefinition: P is the class of decision problems : P is the class of decision problems DD and there exists an algorithm and there exists an algorithm AA returns "yes" if and only if input returns "yes" if and only if input II is in is in DD. (i.e., . (i.e., AA correctly decides correctly decides DD), and ), and the running time of the running time of AA = = OO((nnkk) where ) where nn is the size of the I to be decided and is the size of the I to be decided and kk is is a constant. a constant. Informally, P is the class of decision problems that can be decided in Informally, P is the class of decision problems that can be decided in polynomial time for the given instances. Such an algorithm polynomial time for the given instances. Such an algorithm AA is said to run in is said to run in polynomial timepolynomial time. .

Examples of class P problems:

• Problem: SINGLE-SOURCE-SHORTEST-PATHInstance: A weighted, directed graph G = (V, E), a pair of vertices v and w, both in V, and a number k in R.Question: Is there a path in G from v to w of length at most k? This is in P: Using Djkstra's Algorithm, it runs in time polynomial in the size of the instance. Proving the result of the decision problem, use the sequence of vertices along a path of length at most k as a certificate; verify the length of this path without having to trust Djkstra's Algorithm.

• Problem: LESS-THANInstance: Two integers, n and m.Question: Is n less than m? The size of the problem is O(log n + log m), using binary representation of the numbers. This problem can be decided in O(log n + log m) time, which is linear in the sizes of the numbers (i.e., k=1 in the definition of P).

• P roughly corresponds to algorithmic notion of efficient computation; if a problem is in P, then it can be solved efficiently.

• Some problems are not in P. For example, consider this problem: • Problem: HALTING

Instance: A C program A.Question: Will A ever call exit()? However, there is no general algorithm for deciding this problem in polynomial time. (It turns out that there is no algorithm for doing this at all, in any kind of time, but you don't have to believe that until later.)

• There exist many problems not known whether or not they are in P. For There exist many problems not known whether or not they are in P. For example: Problemexample: Problem: GRAPH-ISOMORPHISM: GRAPH-ISOMORPHISMInstance: Two graphs G and H.Question: Are G and H isomorphic? (i.e., are they the same shape? Can we relabel the vertices of one to make it identical to the other?) It isn't known whether GRAPH-ISOMORPHISM is in P.

P

NP NP-complete

Non-P problems (intractable) Non-P problems (intractable) • Turing 1930, Problems Turing 1930, Problems unsolvableunsolvable by by anyany algorithm. The most algorithm. The most

well known is the well known is the halting problem: halting problem: Given an arbitrary Given an arbitrary algorithm and its input, will that algorithm eventually halt, or algorithm and its input, will that algorithm eventually halt, or will it continue forever in an “will it continue forever in an “infinite loopinfinite loop?”?”

• Hamiltonian Paths: Given a G(V, E), find a path visiting every Hamiltonian Paths: Given a G(V, E), find a path visiting every vertex just once. vertex just once. Searching such a path is an optimization Searching such a path is an optimization problem. Decision problem: But deciding if there is such a path problem. Decision problem: But deciding if there is such a path is a Decision Problem of yes or nois a Decision Problem of yes or no. .

• TSP: Finding a minimum weight Hamiltonian cycle. TSP whose TSP: Finding a minimum weight Hamiltonian cycle. TSP whose all the edge weights is set to one, is a special case of Hamiltonian all the edge weights is set to one, is a special case of Hamiltonian cycle. cycle. Decision problem: For a given G and for a total weight of w. Is Decision problem: For a given G and for a total weight of w. Is there a TSP with a total weight at most wthere a TSP with a total weight at most w. .

• Can be classified in various categories based on their degree of Can be classified in various categories based on their degree of difficulty, e.g., NP, NPC, NP-harddifficulty, e.g., NP, NPC, NP-hard

Nondeterministic algorithm circuit Nondeterministic algorithm circuit Definition: Nondeterministic algorithm circuit Definition: Nondeterministic algorithm circuit has two stages:has two stages:1)1) NondeterministicNondeterministic (“ (“guessingguessing”) stage: ”) stage: generate randomly an generate randomly an

arbitrary string that can be thought of a candidate solution arbitrary string that can be thought of a candidate solution (“certificate”)(“certificate”)

2)2) DeterministicDeterministic (“ (“verificationverification”) stage: ”) stage: take the certificate and take the certificate and the instance to the problem and returns YES if the certificate the instance to the problem and returns YES if the certificate represents a solution in polynomial timerepresents a solution in polynomial time. Given a “certificate” . Given a “certificate” of a solution, verify that the certificate is correct in time of a solution, verify that the certificate is correct in time polynomial to the size of the input. polynomial to the size of the input.

Poly of poly = poly. O(p(p(|s|))+p(|s|))) = O(p(|s|)).Poly of poly = poly. O(p(p(|s|))+p(|s|))) = O(p(|s|)).• Any algorithm passes these two stages of exam is said to be a Any algorithm passes these two stages of exam is said to be a

Non-Deterministic Problem. Non-Deterministic Problem. • Examples: Examples:

– Hamiltonian-cycle, given a certificate of a sequence (Hamiltonian-cycle, given a certificate of a sequence (vv11,,vv22,…, ,…, vvnn), easily verified in poly time.), easily verified in poly time.

– 3-CNF, given a certificate of an assignment 0s, 1s, easily 3-CNF, given a certificate of an assignment 0s, 1s, easily verified in poly time. verified in poly time.

(Poly) Reduction Algorithm for B

Decision algorithm for A

YES YES

NO NO

Nondeterministic algorithm circuitNondeterministic algorithm circuit

• A problem A problem pp NP for an instance, and any other instances of the NP for an instance, and any other instances of the problem problem pp NP can be translated as NP can be translated as pp in poly time. in poly time.

• All current known NP hard problems proved to be NPC.All current known NP hard problems proved to be NPC.• P P NP, NPC NP, NPC NP NP• P = NP (or P P = NP (or P NP, or P NP, or P NP) ??? NP) ???• NPC = NP (or NPC NPC = NP (or NPC NP, or NPC NP, or NPC NP) ??? NP) ???• P P NP: one of the most perplexing open research problems in NP: one of the most perplexing open research problems in

(theoretical) computer science since 1971.(theoretical) computer science since 1971.• No poly algorithm found for any NPC problem (even though No poly algorithm found for any NPC problem (even though

there are so many NPC problems)there are so many NPC problems)• There is no proof of statement that a poly algorithm cannot exist There is no proof of statement that a poly algorithm cannot exist

for any of NPC problems.for any of NPC problems.• Theoretical computer scientists believe that NPC is intractable Theoretical computer scientists believe that NPC is intractable

(i.e., hard, and P (i.e., hard, and P NP). NP).

NPCNPC

PP

NPNP

P NP, NPC NP, P NPC =

Why searching for NPC• If proved NPC, a good evidence for its intractability (hardness).• No need to waste time on trying to find out an efficient algorithm, rather focus

on an approximate algorithm or solution for a special cases• Some problems might seem to be easy, but in fact, np hard (NPC).

Decision VS. Optimization Problems • Decision problem: solution is binary “YES” or “NO”• Optimization problem: seeking solution of an optimal solution.• Easier to define reduction from and between decision problems than

optimization. NPC is confined to decision problem. (but also applicable to optimization problem.)

• Examples: – SHORTEST-PATH (optimization)

Given G, and vertices u, v, find a path from u to v with minimum number of edges (and with minimum weight).

– PATH (decision)Given G, vertices u, v, and k, whether exists a path from u to v consisting of at most k edges.

• If there is solution for an optimization problem, the algorithm can be used to solve the corresponding decision problem, and if optimization is easy, its corresponding decision is also easy. Or if provides evidence that decision problem is hard, then the corresponding optimization problem is also hard.

• Reduction is a way of saying that one problem is Reduction is a way of saying that one problem is “easier”“easier” than another. than another.

• We say that problem A is easier than problem B, (i.e., we We say that problem A is easier than problem B, (i.e., we write write “A “A B” B”) if we can solve A using the algorithm ) if we can solve A using the algorithm that solves B.that solves B.

• Idea:Idea: transform the inputs of A to inputs of Btransform the inputs of A to inputs of B

• Given two problems A, B, we say that A is polynomially Given two problems A, B, we say that A is polynomially reducible reducible to B to B (A (A pp B) B) if: if: – There exists a function There exists a function f f that converts every input of A to that converts every input of A to

inputs of B in polynomial time inputs of B in polynomial time – And every yes instance of A, corresponds a distict yes instance And every yes instance of A, corresponds a distict yes instance

of B and similarly every no instance to no. A(i) = YES of B and similarly every no instance to no. A(i) = YES B(f(i)) = YES, A(i) = NO B(f(i)) = YES, A(i) = NO B(f(i)) = NO B(f(i)) = NO

Reducing a problem to anotherReducing a problem to another

• These both conditions are tested in polytime. These both conditions are tested in polytime. • A problem B is said to be A problem B is said to be NP-completeNP-complete (notice B is (notice B is

underquestion) if: underquestion) if: i.i. BB NP, nondeterministic circuit test.NP, nondeterministic circuit test.ii.ii. A A pp B B for all problems in NP, A for all problems in NP, A NP, reducibility test. NP, reducibility test.

Reducing all problems to the B that is under question.Reducing all problems to the B that is under question.

• If B satisfies only reduction property (2) we say that B is If B satisfies only reduction property (2) we say that B is NP-hardNP-hard

• No polynomial time algorithm has been discovered for an No polynomial time algorithm has been discovered for an NP-CompleteNP-Complete problem problem

• No one has ever proven that no polynomial time No one has ever proven that no polynomial time algorithm can exist for any algorithm can exist for any NP-CompleteNP-Complete problem problem

Non-Deterministic Poly Complete, NPCNon-Deterministic Poly Complete, NPC

Implications of Reduction

- If A p B and B P, then A P - if A p B and A P, then B P

f Problem B yes

no

yes

no

Problem A

Proving Polynomial Time1. Use a polynomial time reduction algorithm to transform A into B2. Run a known polynomial time algorithm for B3. Use the answer for B as the answer for A

Polynomial time algorithm to decide A

fPolynomial time

algorithm to decide B yes

no

yes

no

P & NP-Complete Problems• Shortest simple path

– Given a graph G = (V, E) find a shortest path from a source to all other vertices

– Polynomial solution: O(VE)• Longest simple path

– Given a graph G = (V, E) find a longest path from a source to all other vertices

– NP-complete• Euler tour

– G = (V, E) a connected, directed graph find a cycle that traverses each edge of G exactly once (may visit a vertex multiple times)

– Polynomial solution O(E)• Hamiltonian cycle

– G = (V, E) a connected, directed graph find a cycle that visits each vertex of G exactly once

– NP-complete

Class P ProblemsClass P Problems• Let Let nn= the length of binary encoding of a problem (i.e., input size), = the length of binary encoding of a problem (i.e., input size), TT((nn) is the ) is the

time to solve it.time to solve it.• A problem is A problem is poly-time solvablepoly-time solvable if if TT((nn) =) =OO((nnkk) for some constant ) for some constant kk..• Complexity class Complexity class PP=set of problems that are =set of problems that are poly-time solvablepoly-time solvable..

Poly Time VerificationPoly Time Verification• PATH problem: Given <G,PATH problem: Given <G,uu,,vv,,kk>, whether exists a path from >, whether exists a path from uu to to vv with at most with at most

k k edges?edges?• Given a path Given a path pp from from uu to to vv, verifying whether the length of , verifying whether the length of pp is at most is at most kk??• Poly Time Verification, encoding, and languagePoly Time Verification, encoding, and language• Hamiltonian cyclesHamiltonian cycles

– A simple path containing every vertex.A simple path containing every vertex.– HAM-CYCLE={<G>: G is a Hamiltonian graph, i.e. containing Hamiltonian cycle}.HAM-CYCLE={<G>: G is a Hamiltonian graph, i.e. containing Hamiltonian cycle}.– Suppose Suppose nn is the length of is the length of encodingencoding of G. of G.– HAM-CYCLE can be considered as a HAM-CYCLE can be considered as a LanguageLanguage after encoding, i.e. a subset of after encoding, i.e. a subset of * *

where where ={0,1}*.={0,1}*.• The naïve algorithm for determining HAM-CYCLE runs in The naïve algorithm for determining HAM-CYCLE runs in ((mm!)=!)=(2(2mm) time, ) time,

where where mm is the number of vertices, is the number of vertices, mm nn1/21/2..

More Discussion on Poly time problemsMore Discussion on Poly time problems ((nn100100) vs. ) vs. (2(2nn))• Reasonable to regard a problem of Reasonable to regard a problem of ((nn100100) as intractable, however, very few ) as intractable, however, very few

practical problem with practical problem with ((nn100100).).• Poly time keeps same in many different computation models, e.g., poly class of Poly time keeps same in many different computation models, e.g., poly class of

serial random-access machine serial random-access machine poly class of abstract Turing machine poly class of abstract Turing machine poly poly class of parallel computer (#processors grows polynomially with input size)class of parallel computer (#processors grows polynomially with input size)

• Poly time problems have nice closure properties under addition, multiplication Poly time problems have nice closure properties under addition, multiplication and composition.and composition.

Encoding impact on complexity Encoding impact on complexity • The problem instance must be represented in a way the program (or machine) The problem instance must be represented in a way the program (or machine)

can understand.can understand.• General encoding is “binary representation”.General encoding is “binary representation”.• Different encoding will result in different complexities.Different encoding will result in different complexities.• Example: an algorithm, only input is integer Example: an algorithm, only input is integer kk, running time is , running time is ((kk). ).

– If If kk is represented in is represented in unaryunary: a string of : a string of kk 1s, the running time is 1s, the running time is ((kk) = ) = ((nn) on ) on length-length-nn input, input, poly on poly on nn..

– If If kk is represented in is represented in binary: binary: the input length the input length nn = = log log kk +1, the running time is +1, the running time is ((kk) = ) = (2(2nn), ), exponential on exponential on nn. .

• Ruling out Ruling out unaryunary, other encoding methods are same. , other encoding methods are same. Examples of encoding and complexityExamples of encoding and complexity• Given integer Given integer nn, check whether , check whether nn is a composite. is a composite.• Dynamic programming for subset-sum.Dynamic programming for subset-sum.

SAT satisfiability problemSAT satisfiability problem

Boolean combinational circuitBoolean combinational circuit– Boolean combinational elements, wired together, inputs and outputs Boolean combinational elements, wired together, inputs and outputs

(binary), the number of outputs to 1. (binary), the number of outputs to 1. logic gateslogic gates: NOT gate, AND gate, : NOT gate, AND gate, OR gate. OR gate. true tabletrue table:: giving the outputs for each setting of inputs, giving the outputs for each setting of inputs,

– true assignmenttrue assignment is given for a set of boolean inputs, is given for a set of boolean inputs, – satisfying assignmentsatisfying assignment: : a true assignment causing the output to be 1. A a true assignment causing the output to be 1. A

circuit is circuit is satisfiablesatisfiable if it has a satisfying assignment. if it has a satisfying assignment.

Circuit satisfying problem: given a boolean combinational circuit composed of Circuit satisfying problem: given a boolean combinational circuit composed of AND, OR, and NOT, is it satisfiable?AND, OR, and NOT, is it satisfiable?

• CIRCUIT-SAT={<C>: C is a satisfiable boolean circuit}CIRCUIT-SAT={<C>: C is a satisfiable boolean circuit}• Implication: if a subcircuit always produces 0, then the subcircuit can be Implication: if a subcircuit always produces 0, then the subcircuit can be

replaced by a simpler subcircuit that omits all gates and just output a 0.replaced by a simpler subcircuit that omits all gates and just output a 0.

Solving circuit-satisfiability problem• Intuitive solution:

– for each possible assignment, check whether it generates 1.– suppose the number of inputs is k, then the total possible assignments are 2k. So the running time

is (2k). When the size of the problem is (k), then the running time is not poly.

Circuit-satisfiability problem is NP-complete• Lemma 34.5:(page 990)

– CIRCUIT-SAT belongs to NP.• Proof: CIRCUIT-SAT is poly-time verifiable.

– Given (an encoding of) a CIRCUIT-SAT problem C and a certificate, which is an assignment of boolean values to (all) wires in C.

– The algorithm is constructed as follows: just checks each gates and then the output wire of C: • If for every gate, the computed output value matches the value of the output wire given in the certificate

and the output of the whole circuit is 1, then the algorithm outputs 1, otherwise 0.• The algorithm is executed in poly time (even linear time).

• An alternative certificate: a true assignment to the inputs.• Lemma 34.6: (page 991)

– CIRCUIT-SAT is NP-hard.• Proof: Suppose X is any problem in NP

– construct a poly-time algorithm F maps every problem instance x in X to a circuit C=f(x) such that the answer to x is YES if and only if CCIRCUIT-SAT (is satisfiable).

• Since XNP, there is a poly-time algorithm A which verifies X.• Suppose the input length is n and Let T(n) denote the worst-case running time. Let k be

the constant such that T(n)=O(nk) and the length of the certificate is O(nk).

Circuit-satisfiability problem is NP-hard• Idea is to represent the computation of A as a sequence of

configurations, c0, c1,…,ci,ci+1,…,cT(n), each ci can be broken into – (program for A, program counter PC, auxiliary machine state, input x, certificate

y, working storage) and – ci is mapped to ci+1 by the combinational circuit M implementing the

computer hardware.– The output of A: 0 or 1– is written to some designated location in working

storage. If the algorithm runs for at most T(n) steps, the output appears as one bit in cT(n).

– Note: A(x,y)=1 or 0.

• The reduction algorithm F constructs a single combinational circuit C as follows:– Paste together all T(n) copies of the circuit M.– The output of the ith circuit, which produces ci, is directly fed into the

input of the (i+1)st circuit.– All items in the initial configuration, except the bits corresponding to

certificate y, are wired directly to their known values.– The bits corresponding to y are the inputs to C.– All the outputs to the circuit are ignored, except the one bit of cT(n)

corresponding to the output of A.

• Two properties remain to be proven:– F correctly constructs the reduction, i.e., C is satisfiable if and only if there

exists a certificate y, such that A(x,y)=1.Suppose there is a certificate y, such that A(x,y)=1. Then if we apply the bits of y

to the inputs of C, the output of C is the bit of A(x,y), that is C(y)= A(x,y) =1, so C is satisfiable.

Suppose C is satisfiable, then there is a y such that C(y)=1. So, A(x,y)=1.– F runs in poly time.

• Poly space:– Size of x is n.– Size of A is constant, independent of x.– Size of y is O(nk).– Amount of working storage is poly in n since A runs at most O(nk). – M has size poly in length of configuration, which is poly in O(nk), and hence is poly in

n.– C consists of at most O(nk) copies of M, and hence is poly in n.– Thus, the C has poly space.

• The construction of C takes at most O(nk) steps and each step takes poly time, so F takes poly time to construct C from x.

• In summary if SAT is NPC.. – 1- CIRCUIT-SAT belongs to NP, verifiable in poly time.– 2- CIRCUIT-SAT is NP-hard, every NP problem can be reduced to

CIRCUIT-SAT in poly time, notice that Curcitsat is under question.– 3- Thus CIRCUIT-SAT is NP-complete.

36

NP-completeness proof basis• Lemma 34.8 (page 995)

– If X is a problem (class) such that P'p X for some P' NPC, then X is NP-hard. Moreover, if X NP, then X NPC.

• Steps to prove X is NP-complete– Prove X NP.

• Given a certificate, the certificate can be verified in poly time. – Prove X is NP-hard.

• Select a known NP-complete P'.• Describe a transformation function f that maps every instance x of P' into an instance f(x)

of X.• Prove f satisfies that the answer to xP' is YES if and only if the answer to f(x)X is

YES for all instance x P'.• Prove that the algorithm computing f runs in poly-time.

NPC proof –Formula Satisfiability (SAT)• SAT definition

– n boolean variables: x1, x2,…, xn.– M boolean connectives: any boolean function with one or two inputs and one

output, such as ,,,,,… and – Parentheses.

• A SAT is satisfiable if there exists an true assignment which causes to evaluate to 1.

• SAT={< >: is a satifiable boolean formula}.• The historical honor of the first NP-complete problem ever shown.

37

SAT is NP-complete• Theorem 34.9: (page 997)

– SAT is NP-complete.• Proof:

i. SAT belongs to NP. To prove this statement, two stages: Given (suggesting) a satisfying assignment, and verifying this statement with the program, algorithm that replaces each variable with its value and evaluates the formula, in polytime. 2 stages of non-deterministic algorithm circuit, a- nondeterministically suggesting a solution string and b-deterministically verifying that solution string if it belongs to the SAT problem under question, with yes or no answers to prove statement SAT is NP-hard

ii. Use reduction circuit now to show CIRCUIT-SAT can be reduced to SAT. Remember circuitsat is npc. CIRCUIT-SATp SAT, i.e., any instance of circuit satisfiability can be reduced in polytime to an instance of formula satisfiability. Remember we couldnot have used an npc like circuit-sat, then we would be obliged to prove that every np problem could be reduced to SAT, this would be a tedious work but we only reduce an npc to an np problem under question to complete our proof. SAT is underquestion. Redcution must be poly reduction, so entire circuit is poly but solutions are not, I mean solution of SAT or Cirsat, neither can be prroved poly solvable, only reductions, rights side npc, if poly reduction is possible to an np, then left side is npc.

38

Example of reduction of CIRCUIT-SAT to SAT= x10(x10(x7 x8 x9)) (x9(x6 x7)) (x8(x5 x6)) (x7(x1 x2 x4)) (x6 x4)) (x5(x1 x2)) (x4x3)

INCORRECT REDUCTION: = x10= x7 x8 x9=(x1 x2 x4) (x5 x6) (x6 x7)=(x1 x2 x4) ((x1 x2) x4) (x4 (x1 x2 x4))=….

39

NPC Proof –3-CNF Satisfiability• 3-CNF definition3-CNF definition

– A A literalliteral in a boolean formula is an occurrence of a in a boolean formula is an occurrence of a variable or its negation. variable or its negation. A clauseA clause is a logical is a logical variable or a number of logical variablesvariable or a number of logical variables. .

– CNF (Conjunctive Normal Form) is a boolean CNF (Conjunctive Normal Form) is a boolean formula expressed as AND of clauses, each of formula expressed as AND of clauses, each of which is the OR of one or more literals. which is the OR of one or more literals.

– 3-CNF is a CNF in which each clause has exactly 3-CNF is a CNF in which each clause has exactly 3 distinct literals (a literal and its negation are 3 distinct literals (a literal and its negation are distinct)distinct)

• 3-CNF-SAT: whether a given 3-CNF is 3-CNF-SAT: whether a given 3-CNF is satiafiable?satiafiable?

40

3-CNF-SAT is NP-complete• Proof: 3-CNF-SAT NP. Easy.

– 3-CNF-SAT is NP-hard. (show SAT p3-CNF-SAT)• Suppose is any boolean formula, Construct a binary ‘parse’ tree, with

literals as leaves and connectives as internal nodes.• Introduce a variable yi for the output of each internal nodes.• Rewrite the formula to ' as the AND of the root variable and a

conjunction of clauses describing the operation of each node. • The result is that in ', each clause has at most three literals.• Change each clause into conjunctive normal form as follows:

– Construct a true table, (small, at most 8 by 4)– Write the disjunctive normal form for all true-table items evaluating to 0– Using DeMorgan law to change to CNF.

• The resulting '' is in CNF but each clause has 3 or less literals.• Change 1 or 2-literal clause into 3-literal clause as follows:

– If a clause has one literal l, change it to (lpq)(lpq) (lpq) (lpq).– If a clause has two literals (l1 l2), change it to (l1 l2 p) (l1 l2 p).

41

Binary parse tree for =((x1 x2) ((x1 x3) x4))x2

'= y1(y1 (y2x2)) (y2 (y3 y4)) (y4 y5) (y3 (x1 x2)) (y5 (y6 x4)) (y6 (x1 x3))

42

Example of Converting a 3-literal clause to CNF format

Disjunctive Normal Form:i'=(y1y2x2)(y1y2x2) (y1y2x2) (y1y2x2)

Conjunctive Normal Form:i''=(y1y2x2)(y1y2x2) (y1y2x2)(y1y2x2)

43

3-CNF is NP-complete and reduced 3-CNF are equivalent:

– From to ' , keep equivalence. – From ' to '' , keep equivalence.– From '' to final 3-CNF, keep equivalence.

• Reduction is in poly time,– From to ' , introduce at most 1 variable and 1

clause per connective in . – From ' to '' , introduce at most 8 clauses for each

clause in '.– From '' to final 3-CNF, introduce at most 4 clauses for

each clause in ''.

44

NP-completeness proof structure

45

NPC proof -- CLIQUE• Definition: a clique in an undirected graph G=(V,E) is a subset

V'V of vertices, each pair of which is connected by an edge in E, i.e., a clique is a complete subgraph of G.

• Size of a clique is the number of vertices in the clique.• Optimization problem: find the maximum clique.• Decision problem: whether a clique of given size k exists in the

graph? • CLIQUE={<G,k>: G is a graph with a clique of size k.}• Intuitive solution: ??? CLIQUE is NP-complete• Theorem 34.11: (page 1003)

– CLIQUE problem is NP-complete.• Proof:

– CLIUEQE NP: given G=(V,E) and a set V'V as a certificate for G. The verifying algorithm checks for each pair of u,vV', whether <u,v> E. time: O(|V'|2|E|).

– CLIQUE is NP-hard: • show 3-CNF-SAT pCLUQUE.• The result is surprising, since from boolean formula to graph.

46

=(x1x2x3)(x1x2x3)(x1x2x3) and its reduced graph G

C1=x1x2 x3

47

CLIQUE is NP-complete

• Reduction from 3-CNF-SAT to CLUQUE.– Suppose =C1 C2… Ck be a boolean formula in 3-CNF with k clauses. – We construct a graph G=(V,E) as follows:

• For each clause Cr =(l1r l2

r l3r), place a triple of v1

r, v2r, v3

r into V• Put the edge between two vertices vi

r and vjs when:

– rs, that is vir and vj

s are in different triples, and– Their corresponding literals are consistent, i.e, li

r is not negation of ljs .

– Then is satisfiable if and only if G has a clique of size k.

• Prove the above reduction is correct:– If is satisfiable, then there exists a satisfying assignment, which makes at least one

literal in each clause to evaluate to 1. Pick one this kind of literal in each clause. Then consider the subgraph V' consisting of the corresponding vertex of each such literal. For each pair vi

r,vjs V', where rs. Since li

r,ljs are both evaluated to 1, so li

r is not negation of lj

s, thus there is an edge between vir and vj

s. So V' is a clique of size k. – If G has a clique V' of size k, then V' contains exact one vertex from each triple.

Assign all the literals corresponding to the vertices in V' to 1, and other literals to 1 or 0, then each clause will be evaluated to 1. So is satisfiable.

• It is easy to see the reduction is in poly time.• The reduction of an instance of one problem to a specific instance of the other

problem.

48

Traveling-salesman problem is NPC • TSP={<G,c,k>: G=(V,E) is a complete graph, c is a function from VVZ, kZ, and G has a traveling salesman tour with cost at most k.}• Theorem 34.14: (page 1012)

– TSP is NP-complete.TSP is NP-complete• TSP belongs to NP:

– Given a certificate of a sequence of vertices in the tour, the verifying algorithm checks whether each vertex appears once, sums up the cost and checks whether at most k. in poly time.

• TSP is NP-hard (show HAM-CYCLEpTSP)– Given an instance G=(V,E) of HAM-CYCLE, construct a TSP instance <G',c,0) as

follows (in poly time):• G'=(V,E'), where E'={<i,j>: i,j V and ij} and • Cost function c is defined as c(i,j)=0 if (i,j) E, 1, otherwise.

– If G has a hamiltonian cycle h, then h is also a tour in G' with cost at most 0.– If G' has a tour h' of cost at most 0, then each edge in h' is 0, so each edge belong

to E, so h' is also a hilmitonian cycle in G.

49

Subset Sum is NPC• Subset sum problem: is there a non-empty subset of positive

integers s1, s2, s3, …, sn having a sum exactly equal to zero? Or an equivalent to an integer t (knapsack special case)?

• {1, 6, -3, -2, -5, 7, 9}, t =7 kind.. A = [7], [1, 6], [-2, -5, 7, 1, 6], [-3, -2, -5, 1, 6, 5].. Many subsets are possible..

• NP-Complete, a decision problem, np-hard if it is an optimization prb.

• Precision of ± 1% amounts solving the problem just for the first 7 digits and + 1 for the sign.. However if there are 2100, bits, this will be 7% of the constraints solved, think that every bit represents an answer of yes or no decision for one question, 27 out of 2100. remaining 293 unsolved problems. The only way subset solution can be a solution to other NP problems is to solve all the problems exactly.

50

Subset Sum is NPC• Very much applicable in cryptology against crytoattacks. • Studied for approximation problems as well.

• SSS can be viewed as depending on two variables. N the number of decision variables, P is the precision of the problem. Problem is most difficult when both variables are in the same order, it is exponential. Becomes easier when either of variables gets smaller.

• The methods practical – Exhaustive search: when the number of variables N is small– Dynamic algorithms applicable when the precision is a small fixed number.

• Solution becomes nonexponential and practical if entire solution space is countable.. Two ways to count, SSS problem in example, 2N possible combinations of variables that can be counted for small N, or 2P numerical sums all of which can be counted with the dynamic algorithm.. When N=P, and both large no solution space can be counted.

• Checking if subset sums to right number, 2N subsets, N elements O(2N N). Exponential time. This can be reduced by sorting, by splitting N elements into two sets, and calculating 2N/2 sums, in two arrays, sort, and checking the sum, one array in ascending and the other in descending order, and starting.

• Subset problem of size n.Subset problem of size n. Nonsystematic search of the space takes Nonsystematic search of the space takes O(p2O(p2nn)) time, where p is the time time, where p is the time

needed to evaluate each member of the solution space.needed to evaluate each member of the solution space.• Permutation problem of size n.Permutation problem of size n.

Nonsystematic search takes Nonsystematic search takes O(pn!)O(pn!) time. P is the same as above. time. P is the same as above. • A systematic approach is backtracking and branch and bound A systematic approach is backtracking and branch and bound

perform; often taking less time than the time taken by a perform; often taking less time than the time taken by a exhoustive search. exhoustive search.

• Solution space:Solution space: A tree structure such that the leaves represent A tree structure such that the leaves represent members of the solution space. For a size n, this tree members of the solution space. For a size n, this tree – subset problem structure has subset problem structure has 22n n leaves. leaves.– permutation problem structure has permutation problem structure has n! leaves.n! leaves.

• Limitations: Memory and time: too big trees require to much Limitations: Memory and time: too big trees require to much memory and waste to long time.memory and waste to long time.

• Portions of the tree are created by the backtracking and branch Portions of the tree are created by the backtracking and branch and bound algorithms as needed.and bound algorithms as needed.

In addressing subset and permutation prblms In addressing subset and permutation prblms Backtracking Branch and BoundBacktracking Branch and Bound

Tree for Subset Problem (2n leaves)• At level i the members of the solution space are

partitioned by their xi values.• Members with xi = 1 are in the left subtree.• Members with xi = 0 are in the right subtree.

x1=1 x1= 0

x2=1 x2= 0 x2=1 x2= 0

x3=1 x3= 0

x4=1 x4=0

1110 1011 0111 0001

Subset Tree For n = 4Subset Tree For n = 4

Permutation Tree (n! Leaves) for n = 3 • At level i the members of the solution space are partitioned At level i the members of the solution space are partitioned

by their xby their xii values. values.• Members (if any) with xMembers (if any) with xi i = 1 are in the first subtree.= 1 are in the first subtree.• Members (if any) with xMembers (if any) with xi i = 2 are in the next subtree.= 2 are in the next subtree.

x1=1 x1=2x1= 3

x2= 2 x2= 3 x2= 1 x2= 3 x2= 1 x2= 2

x3=3 x3=2 x3=3 x3=1 x3=2 x3=1

123 132 213 231 312 321

Backtracking and bounding, DFS AlgoBacktracking and bounding, DFS Algo• Recursive implementation is applicable. Need a stack to retain Recursive implementation is applicable. Need a stack to retain

the path from the root to the branch traversed. Search from dark the path from the root to the branch traversed. Search from dark red to light goes on.. red to light goes on..

• The solution space tree exists The solution space tree exists only on mindonly on mind. . Bounding functionsBounding functions: : • While moving deeper if xWhile moving deeper if xii==1, s++ (the sum of the subset), ==1, s++ (the sum of the subset),

moving backward if xmoving backward if xii== 1, s--. If x== 1, s--. If xii == 0, remains indifferent. == 0, remains indifferent. • Keeping the track of the sum in a variable of s: Keeping the track of the sum in a variable of s: s+=s+( forward s+=s+( forward

xxii==1 ) - ( backward x==1 ) - ( backward xii==1 ). ==1 ). • T is the target value, visiting a node, T is the target value, visiting a node,

– s==t, terminate, s==t, terminate, – s>t backtrack to parent to parent node, else move forward into the s>t backtrack to parent to parent node, else move forward into the

subtrees. subtrees.

• A second variable r keeping the sum of the nodes not visited A second variable r keeping the sum of the nodes not visited yet. Before moving to a right child, check if the current subset yet. Before moving to a right child, check if the current subset sum (s + r) >= t. If not, backtrack.sum (s + r) >= t. If not, backtrack.

xx11=1=1 xx11= 0= 0

xx22=1=1 xx22= 0= 0 xx22=1=1 xx22= 0= 0

BacktrackingBacktracking

• DFS Each move takes O(1) time DFS Each move takes O(1) time back of forward. Space required back of forward. Space required is O(tree height).is O(tree height).

• With effective bounding With effective bounding functions, large instances can functions, large instances can often be solved.often be solved.

• For some problems (e.g., 0/1 For some problems (e.g., 0/1 knapsack), the answer (or a very knapsack), the answer (or a very good solution) may be found good solution) may be found quickly but a lot of additional quickly but a lot of additional time is needed to complete the time is needed to complete the search of the tree.search of the tree.

• Time limitations: Run Time limitations: Run backtracking if time is feasible.backtracking if time is feasible.

• May never find a solution ifMay never find a solution if tree depth is infinite.

Branch and BoundBranch and Bound

• Search the tree using a breadth-first search (FIFO branch and bound).

• Search the tree as in a bfs, but replace the FIFO queue with a stack (LIFO branch and bound).

• Replace the FIFO queue with a priority queue (least-cost (or max priority) branch and bound).

• The priority of a node p in the queue is based on an estimate of the likelihood that the answer node is in the subtree whose root is p.

• Space required is O(number of leaves).

• Finds solution closest to root. Least-cost branch and bound directs the search to parts of the space most likely to contain the answer. So it could perform better than backtracking.

56

Subset Sum is NPC• SUBSET-SUM={<S,t>: S is a set of integers

and there exists a S'S such that t=sS's.}

• Theorem 34.15: (page 1014)– SUBSET-SUM is NP-complete.

• SUBSET-SUM belongs to NP.– Given a certificate S', check whether t is sum of S'

can be finished in poly time.

• SUBSET-SUM is NP-hard (show 3-CNF-SATpSUBSET-SUM).

57

SUBSET-SUM is NPC

• Given a 3-CNF formula =C1 C2… Ck with literals x1, x2,…, xn. Construct a SUBSET-SUM instance as follows:– Two assumptions: no clause contains both a literal

and its negation, and either a literal or

its negation appears in at least one clause.– The numbers in S are based on 10 and have n+k

digits, each digit corresponds to (or is labeled by) a literal or a clause.

– Target t=1…1||4…4 (n 1’s and k 4’s)

58

SUBSET-SUM is NPC– Target t=1…1||4…4 (n 1’s and k 4’s)

– For each literal xi, create two integers:

• vi=0…01(i)0…0||0…01(l)0…01(w)0…0, where xi appears in Cl,…,Cw.

• vi'=0…01(i)0…0||0…1(m)0……01(p)0…0, where xi appears in Cm,…,Cp. .

• Clearly, vi and vi' can not be equal in right k digits, moreover all vi and vi' in S are distinct.

– For each clause Cj, create two integers:

• sj=0…0||0…01(j)0…0,

• sj'=0…0||0…02(j)0…0.

• all sj and sj' are called “slack number”. Clearly, all sj and sj' in S are distinct.

– Note: the sum of digits in any one digit position is 2 or 6, so when there is no carries when adding any subset of the above integers.

59

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.

60

SUBSET-SUM is NPC• The above reduction is done in poly time.• The 3-CNF formula is satisfiable if and only if there

is a subset S' whose sum is t. – suppose has a satisfying assignment.

• Then for i=1,…,n, if xi=1 in the assignment, then vi is put in S', otherwise, then vi' is put in S'.

• The digits labeled by literals will sum to 1.• Moreover, for each digit labeled by a clause Cj and in its three

literals, there may be 1, 2, or 3 assignments to be 1. correspondingly, both sj and sj' or sj', or sj is added to S' to make the sum of the digit to 4.

• So S' will sum to 1…14…4. – Suppose there is a S' which sums to 1…14…4. then S'

contains exact one of vi and vi' for i=1,…,n. if vi S', then set xi=1, otherwise, vi' S', then set xi=0. It can be seen that this assignment makes each clause of to evaluate to 1. so is satisfiable.