CS423 Finite Automata & Theory ofComputation

MW 8:00 - 9:20 in Blow 333

Prof. Weizhen Mao, wm@cs.wm.edu

1

General InformationI Office Hours: On Zoom. MW 3:00 - 4:30 or by email.I Grader: TBDI Textbook: Intro to the theory of computation (any edition),

Michael Sipser.I Prerequisites/background: Linear algebra, Data structures

and algorithms, and Discrete math

2

Use of BlackboardI AnnouncementsI HomeworkI Solution setsI Lecture notesI Recorded lecturesI Grades (HWs, Midterm, Final)I Check at least weekly

3

Lecture Notes (in various forms and places)I Lecture slides: http://www.cs.wm.edu/∼ wm/I Lecture notes posted on BB (A high-level summary)I Recorded lecture posted on BBI Your own class notes

Course OrganizationI Automata theory: 50%I Computability theory: 25%I Complexity theory: 25%

4

GradingI 10 homework assignments: 35%I Midterm: 25%I Final: 40%

Grading PolicyI [90,100]: A or A-I [80,90): B+, B, or B-I [70,80): C+, C, or C-I [60,70): D+, D, or D-I [0,60): F

5

Homework Submission Policy and AccomodationI Your homework needs to be typeset in LaTex with all

figures drawn nicely and inserted. The pdf file is expectedto be submitted to BB before or on the due date

I However, since we are in a pandemic semester, I will allowone late day for each homework to all students who needthe extra time.

I In calculating the final grades, homework with the lowestgrade will be dropped

I Additional extensions may be permitted for illness andfamily emergency. Requests must be made prior to thedue date

6

Homework Completion PolicyI Homework must be typeset in LaTex. Nothing should be

handwritten, including diagrams and tablesI Empty-hand policy when you discuss homework problems

with your classmatesI List your collaborators for every homework problemI Cite all references you used to solve a problemI In no case you should copy verbatim from other people’s

work without proper attribution, as this is consideredplagiarism, thus a violation of the Honor Code

7

ExamsI Midterm: Timed, on BB, and likely in mid-MarchI Final Exam: 9:00-12:00, May11

Other Important DatesI Jan. 26: First day of classI Jan. 26 - Feb. 4: Add/DropI Mar. 9 - 11: Midterm on BBI Mar. 12 - 20: Spring BreakI Mar. 28: Last day to withdrawI May 6: Last day of classI May 11: Final examI May 20 -22: Commencement

8

Writing Assignments for CSci423WI A paper of 2 to 3 pages about a someone pioneering the

field of ToC and one of his/her major researchcontributions, written in the IEEE standard format for CSconferences. The title of the paper should be in the formatof “name and result”, e.g., “Alan Turing and the Enigma”, or“Stephen Cook and the Satisfiability Problem”, or “Rivest,Shamir, and Adleman and the RSA Cryptosystem”. Moredetails after Spring break.

I The class will be divided into ten groups, each beingresponsible for the production of one homework solutionset. More details later.

9

Honor CodeI ”As a member of the William and Mary community, I pledge

on my honor not to lie, or steal, either in my academic orpersonal life. I understand that such acts violates theHonor Code and undermine the community of trust, ofwhich we are all stewards.”

I Academic honesty, the cornerstone of teaching andlearning, lays the foundation for lifelong integrity. TheHonor Code is, as always, in effect in this course. Pleasego to the ”Honor System” page in the wm.edu site to readthe complete honor code document.

10

Student Accessibility ServicesWilliam & Mary accommodates students with disabilities inaccordance with federal laws and university policy. Any studentwho feels they may need an accommodation based on theimpact of a learning, psychiatric, physical, or chronic healthdiagnosis should contact Student Accessibility Services staff at757-221-2512 or at sas@wm.edu to determine ifaccommodations are warranted and to obtain an official letter ofaccommodation. For more information, please go towww.wm.edu/sasPlease inform me privately of any accommodations you havearranged with the Office of SAS.

My goal: I will try my best to be understanding, flexible, fair,and helpful, while creating a high quality learning experience forall of you.

11

Covid SemesterI For this course in spring 2022, here is the way that we will

address student absences: a student’s absence itself willnot be part of the student’s grade; I will also recordcorresponding lectures and upload them to Blackboard sothat the student can access the missing ones; the studentshould also communicate with me on how to handlepossible delay of homework etc.

I For this course in spring 2022, here is the way we willaddress instructor absence: If only a couple of lectures aremissed, I will make up some of the missing ones throughrecorded videos; if too many lectures are missed, thedepartment will get another instructor involved.

12

1.1 Three areas of ToC (Sipser 0.1)Theory of Computation is to study the fundamental capabilitiesand limitations of computers. It contains three areas.I Automata theory: Models of computation. Seeking a

precise but concise definition of a computer.I Computability theory: What can and cannot a computer

do? Computationally unsolvable versus computationallysolvable problems. Determining whether a problem isunsolvable by computer algorithms.

I Complexity theory: What can a computer do efficiently?Computationally hard versus computationally easyproblems. For example, factorization versus sorting.Cryptography needs hard problems such as factorization toensure security.

13

1.2 Sets and Languages (Sipser 0.2)Languages are central concepts in automata theory.I Alphabet Σ: Finite and nonempty, e.g., Σ = 0,1 and

Σ = a,b, . . . ,z.I String (or word), e.g., w = 01110; empty string ε; length of

a string, |w |; concatenation of two strings w1w2; reverse ofa string wR, and substring of a string.

I Language: A language is a set of strings, e.g., ε, /0, Σ,A = w | w has an equal number of 0s and 1s=ε,01,10,0011,0110,1010, · · ·, andB = 0n1n|n ≥ 1= 01,0011,000111, · · ·. Note B ⊂ A.

14

I Regular operators: Let A and B be two languages.I Union: A∪B = x | x ∈ A or x ∈ B.I Concatenation: A ·B = xy | x ∈ A and y ∈ B.

Example: If A = 01,10 and B = 0,1, thenAB = 010,011,100,101.

I Star:A∗ = x1x2 · · ·xk | all k ≥ 0 and each xi ∈ A for i = 1,2, . . . ,k.Note ε ∈ A∗

Example: If A = 0,1, thenA∗ = ε,0,1,00,01,10,11,000,001,010,011,111, . . .Example: If A = 01,11, thenA∗ = ε,01,11,01 01,11 01, . . . ,11 11 01 11, . . .

15

I More operatorsI Intersection: A∩B = x | x ∈ A and x ∈ BI Complement: A is the set of all elements under

consideration that are not in A. Usually, A = Σ∗−A.I Difference: A−B = A∩B.I Power of language:

Ak = x1x2 · · ·xk | xi ∈ A for i = 1,2, · · · ,k= AAk−1.I Observation:

I A0 = εI A∗ = A0∪A1∪A2∪·· · (zero or more)I A+ = A1∪A2∪·· · (one or more)I A∗ = A+∪ε

Quiz 1: If A = 1,01, then A2 =?Quiz 2: If A = 01,11,0, then |A3|=?Claim: If |A|= n, then |Ak |= |A|k = nk . True or false?Quiz 3: A+ = A∗−ε?

16

I Why languages, not problems:I Meta Claim: Computational problem→ decision problem.

Example: Traveling Saleman problem (TSP)Input: G = (V ,E ,w)Goal: Find a tour (cycle) with minimun total weight

I Example: Decision problem for TSPInput: G = (V ,E ,w) and B ≥ 0Question: Is there a tour in G such that the total weight ofthe tour is no more than B?

I Decision problem: Given an input x , does x satisfy propertyP, or is x ∈ y |y satisfies P?Input data of any form, such as matrix, graph, list, etc., canbe coded into strings

I Membership in a language: Given a language A and astring w ∈ Σ∗, is w a member of A, or is w ∈ A?

17

1.3 Proof techniques (Sipser 0.4)I By definition: Convert terms in the hypothesis to their

definitions.I By construction: To prove the existence of X , construct it.I By contradiction: H→ C is equivalent to C→ H or

C∧H→ T , where T is an axiom, a proven truth/fact.Example: The number of primes is infinite.Example:

√2 is irrational.

I By induction: Used to prove a statement S(n), ∀n ≥ c.The logic behind the method:S(n) for n ≥ c iff S(c)∧∀k(S(k)→ S(k + 1)).The proof includes (1) basis step, (2) inductive hypothesis,and (3) inductive step.Example: For n ≥ 1, ∑

ni=1 i2 = 1

6n(n + 1)(2n + 1).

18

A Quick Review of LanguagesI Alphabet Σ, string or word, substring, languageI Common operations borrowed from set theory: union,

intersection, differenceI New operations: concatenation, complement, star, power

19

2.1 What is a Finite Automaton?(Sipser 1.1, 31-34)I Finite automata are the simplest computational models for

computers with an extremely limited amount of memory.I Use of automata theory in software applications includes:

study of the behavior of digital circuits, lexical analyzer incompilers, text pattern matching, and verification offinite-state systems.

I They are designed to accept some strings, therefore torecognize a language, which is the set of accepted strings.

I Given an input string, a finite automaton reads the string,one symbol at a time, from left to right. Using a transitionfunction, the FA changes from one state to another basedon what symbol is being read. Eventually, it reads allsymbols in the string. If at this time the FA enters a finalstate, then the FA halts and the string is accepted.

20

Example 1 of a DFAI Alphabet: Σ = 0,1.I Four states: q0,q1,q2,q3, in which q0 is the start state and

q3 is a final state.

I Transition function δ:

δ 0 1→ q0 q0 q1

q1 q0 q2q2 q0 q3∗q3 q3 q3

I What does δ tell us?δ(q0,0) = q0, δ(q0,1) = q1δ(q1,0) = q0, δ(q1,1) = q2δ(q2,0) = q0, δ(q2,1) = q3δ(q3,0) = q3, δ(q3,1) = q3

21

Example 1 (continued)

0

0

0

0

1 2 31 1 1

0,1

Figure 1: A DFA that accepts strings with substring 111

Try w1 = 100111010 and w2 = 0011010001The language recognized/accepted isL1 = w ∈ 0,1∗ | w contains substring 111

22

2.2 DFA (Sipser 1.1, 35-44)I DFA M = (Q,Σ,δ,q0,F ), where

I Q is a finite set of statesI Σ is an alphabetI q0 ∈Q is the start stateI F ⊆Q is a set of accept/final statesI δ : Q×Σ→Q is a transition function, where δ(q,a) = p is

the next state of M if the current state is q and the currentsymbol is a

I Components of a DFAI A tape of squares, with the same length of the input stringI A control unit that keeps track of the current state and

follows the δ functionI The head that reads and moves to the right, one square at

a timeI The DFA accepts the input string if the head reaches the

end of the tape and the control unit sees the final state

23

I Extending δ to δ : Q×Σ∗→Q: For any q ∈Q and anyw = xa ∈ Σ∗, define δ(q,x) recursively as below:I δ(q,ε) = q andI δ(q,w) = δ(δ(q,x),a)

I Language recognized by DFA M: L(M) = w |δ(q0,w) ∈ F.I A language is called a regular language if some DFA

recognizes it.I How does a DFA accept a string? A path in the diagram

that starts from q0 and ends at qf ∈ F such that theconcatenation of the symbols on the path matches theinput string.

24

Example 2Let L2 = w ∈ 0,1∗ | w has even numbers of 0s and 1sConstruct DFA M such that L(M) = L2.

ee

eo oo

0

0

1 1 1 1

0

0

oe

Figure 2: A DFA that accepts strings of even number of 0s and 1s

25

Example 3 (Sipser p. 36):A DFA M is given below:

δ 0 1→ q0 q0 q1∗q1 q2 q1q2 q1 q1

0 1 2

0

0

1

1

0, 1

Figure 3: Describe the language of the DFA

26

Example 3 (continued)

The DFA in this example accepts strings that have at least one1 and an even number of 0s after the last 1.

Reminder: More on DFA design in Sipser pp. 37-44.

27

Example 4 Design a DFA that acceptsL4 = w ∈ 0,1∗|w contains substring 001

0,1

0 0 1

1

1

0

0 1 2 3

Figure 4: A DFA that accepts strings with substring 001

28

Example 5 Give a DFA that acceptsL5 = w ∈ 0,1∗| numerical value of w is a multiple of 3, e.g.,for string 0110, its numerical value is 6, then it is in thelanguage, but for 101 with a numerical value of 5, it is not amultiple of 3, thus is not in the language.

0 1 2

0

0

1

1

0 1

Figure 5: A DFA that accepts strings with numerical value that is amultiple of 3

29

Example 5 (continued)Assume x is portion of the input string being read so far.Let [x ] be the numerical value of x , e.g., if x = 0010, then[x ] = 2.Entering state qi means that the string read so far has anumerical value that has a remainder of i when divided by 3.I Case 0: The current state is q0, then [x ] = 3k . If the next

symbol is 0, then [x0] = 2 ·3k = 6k , a multiple of 3. So thenext state should be q0. But if the next symbol is 1, then[x1] = 2 ·3k + 1 = 6k + 1, with a remiander of 1 whendivided by 3. So the next state should be q1.δ(q0,0) = q0, δ(q0,1) = q1

I Case 1: Similar to Case 0. δ(q1,0) = q2, δ(q1,1) = q0

I Case 2: Similar to above. δ(q2,0) = q1, δ(q2,1) = q2

30

A Quick Review of DFAsI Definitions of δ : Q×Σ→Q and δ : Q×Σ∗→QI The language of DFA M:

L(M) = w ∈ 0,1∗ | δ(q0,w) ∈ FI A language accepted by a DFA is a regular languageI In designing a DFA with n nodes and alphabet Σ, the

following two properties must be satisfied:I (1) each node must have |Σ| out-going arcsI (2) the total number of arcs must be n · |Σ|I A dead-end state may be added to a DFA to satisfy the

above propertiesI An example of including a dead-end state in a DFA that

accepts strings that start with a 01:

0,1

1

0

0

1

DE

0 12

0,1

31

2.3 NFA (Sipser 1.2, pp. 47-54)I NFA N = (Q,Σ,δ,q0,F ), where

I Q is a finite set of states,I Σ is an alphabet,I q0 ∈Q is the start state,I F ⊆Q is a set of accept/final states, andI δ : Q× (Σ∪ε)→ 2Q is a transition function, where

δ(q,a) = P is the set of states that N may enter if thecurrent state is q and the current symbol is a.In the case of δ(q,ε) = P, N ignores the current symbol andgoes from q to any state in P without reading any symbol.

32

I ε-closure: For any P ⊆Q, E(P) is the set of all statesreachable from any state in P via ≥ 0 ε-transitions.

I Extending δ to δ : Q×Σ∗→ 2Q: For any q ∈Q and anyw = xa ∈ Σ∗, defineI δ(q,ε) = E(q) andI δ(q,w) = E(∪k

i=1δ(pi ,a)) if w = xa andδ(q,x) = p1, . . . ,pk.

I Language of NFA N: L(N) = w |δ(q0,w)∩F 6= /0.I How does an NFA accept a string? Among all paths from

q0 to qf ∈ F , there is some path such that theconcatenation of symbols on the path matches the string.

33

Example 1: Language of strings that contain substring 111

0, 10, 1

1 1 10 1 2 3

Figure 6: An NFA that accepts strings with substring 111

34

Example 2: A = w ∈ 0,1∗|w has 101 or 11 as substrings

0, 10, 1

1 10 1 2 3

0, ε

Figure 7: An NFA that accepts strings with substrings 101 or 11

Clarification:I Consider string 001011. There are at least two paths from

q0 to q3

I Examples of ε-closure (or E-closure): E(q0) = q0;E(q0,q1) = q0,q1,q2

35

Example 3:B = w ∈ 0,1∗|w has a 1 in the 3rd position from the right end

0, 1

1

0 1 2 3

0, 10, 1

Figure 8: An NFA that accepts strings with a 1 in the third positionfrom the right end

36

Example 4: An NFA that accepts decimal numbers (a numberthat may have + or − preceding it, but must have a decimalpoint, e.g., .123, 23., +1.2, -1.0).

0−9

0−9

0−9

ε

+, −

0−9

.

.

0 1

2

3

4

Figure 9: An NFA that accepts strings that are decimal numbers withor without signs

37

2.4 DFAs⇔ NFAs (Sipser 1.2, pp.54-58)Subset construction method: Given NFA N = (Q,Σ,δ,q0,F ),construct a DFA M = (Q′,Σ,δ′,q′0,F

′) such that L(M) = L(N).I Q′ = 2Q (power set), i.e., Q′ contains all subsets of Q.

Note that if |Q|= n then |Q′|= 2n. This is just the worstcase. Since many states in M are inaccessible ordead-end states and thus may be thrown away, so inpractice, |Q′| may be much less than 2n.

I q′0 = E(q0).I F ′ = R ∈Q′|R∩F 6= /0.I For each R ∈Q′ and each a ∈ Σ, δ′(R,a) = E(∪p∈Rδ(p,a)).

Definition: Any language that can be accepted by DFA or NFAis called a regular language (RL).Theorem: The equivalence of DFAs, NFAs, and RLs.

38

Converting an NFA to a DFA with subset constructionExample 1:

0, 10, 1

1 10 1 2 3

0, ε

0

0,2

0,1,2 0,1,2,3

0,3

NFA

DFA

0,2,3

00

00

0

0

1 1

11

1

1

Figure 10: Converting an NFA to DFA with subset construction

39

Example 2:

0

0

0,1

1ε

0

1 2

0,2 1,2

2

1

0

0,1

0

0 01

01

1

1

φ

1

DFANFA

0,1,2

Figure 11: Converting an NFA to a DFA

40

Example 3:

0

01

1

03

1

0

013

002

00 1

1

10123012

1

1

023

0

1

0

0

0

DFA1

0, 1

1

0 1 2 3

0, 10, 1

NFA

Figure 12: Converting an NFA to a DFA with subset construction

41

Example 3: (continued)Without using the subset construction method, can a DFA bedesigned to accept all strings that has a 1 in the third positionfrom the right end?

1

000

001

100 010 110

101 011 111

0

0

0

01

1

1

1

1 1

1

00

0

0

DFA2

Figure 13: Design the same DFA from scratch

42

Example 4: A bad case for the subset construction:|QN |= n + 1 and |QD|= 2n.

0 1 2 n−1 n1 0,1 0,10,10,1

0,1

Figure 14: A case that converting an NFA to a DFA causes anexponential increasing of states in the DFA

43

2.5 Closure Properties of RL’s (Sisper 1.1, pp. 44-47 and 1.2pp. 58-63)I Union: If A and B are regular, so is A∪B.I Concatenation: If A and B are regular, so is AB.I Star: If A is regular, so is A∗. (Need a new start state.)

44

I Complementation: If A is regular, so is A (which is Σ∗−A).I Intersection: If A and B are regular, so is A∩B (which is

A∪B).I Difference: If A and B are regular, so is A−B (which is

A∩B.I Reverse: If A is regular, so is AR.I Homomorphism: If A is regular, so is h(A) (which ish(w)|w ∈ A for a homomorphism h : Σ→ (Σ′)∗).(Discuss later)

I Inverse homomorphism: If A is regular, so is h−1(A) (whereh−1(A) = w |h(w) ∈ A).

45

Example: Prove that A = w ∈ a,b∗|w is of odd length andcontains an even number of a’s is regular.I Let A1 = w |w is of odd length Let A2 = w |w has an

even number of a’sI Since DFAs exist to accept A1 and A2, both are RLsI A = A1∩A2. By the CP of RLs under intersection, A1∩A2

is RL. So A is RL

DFA for A2DFA for A1

a,b

a,b

a

ab b

Figure 15: DFAs for A1 and A2

46

3.1 Definition of REs (Sipser 1.3 (pp. 63-66))Regular expressions (REs) are to represent regular languages.Let L(R) be the language that regular expression R represents.A recursive definition is given below:I Basis: ε and /0 are REs, and L(ε) = ε and L( /0) = /0.

For any a ∈ Σ, a is an RE and L(a) = a.I Induction: If R1 and R2 are REs, then

I R1∪R2 is an RE, with L(R1∪R2) = L(R1)∪L(R2),I R1R2 is an RE, with L(R1R2) = L(R1)L(R2),I R∗1 is an RE, with L(R∗1) = (L(R1))∗, andI (R1) is an RE, with L((R1)) = L(R1).

47

Remark:I Precedence order for regular-expression operators: Star,

concatenation, and finally union. () may override this order.I Use of R+ and Rk .I Algebraic laws:

I R1∪R2 = R2∪R1, (R1∪R2)∪R3 = R1∪ (R2∪R3), and(R1R2)R3 = R1(R2R3).

I /0∪R = R∪ /0 = R, εR = Rε = R, /0R = R /0 = /0, andR∪R = R.

I R1(R2∪R3) = R1R2∪R1R3 and(R1∪R2)R3 = R1R3∪R2R3.

I (R∗)∗ = R∗, /0∗ = ε, R+ = RR∗ = R∗R, and R∗ = R+∪ ε.

48

3.2 Understanding REsI RE is a pattern for all strings in a RL. The goal is to make

the RE as simple and readable as possible. Consider thefollowing examples to simplify REs

I 1∪10∗⇒ 10∗

I (0∗1∗)∗⇒ (0∪1)∗

I ((0∪1)(0∪1)∗)∗⇒ (0∪1)∗

49

Given a language, design its RE

Example 1: w has no substring 10: 0∗1∗

Example 2: w has even number of 1’s: (0∗10∗10∗)∗∪0∗

Example 3: w has odd length: ((0∪1)(0∪1))∗(0∪1)Example 4: w has a 1 in 3rd or 2nd position from right end:(0∪1)∗1(0∪1)(0∪1)∪ (0∪1)∗1(0∪1)⇒(0∪1)∗1(0∪1)((0∪1)∪ ε)⇒(0∪1)∗1(0∪1)(0∪1∪ ε)Example 5: A language of strings that consist of alternating 0sand 1s: (01)∗∪ (10)∗∪0(10)∗∪1(01)∗.

50

Example 7: D = w has odd number of alternating blocks of 0sand 1’s (Note: ε 6∈ D)I 0|1|0 ∈ D, 0|1|0|1 6∈ D, 11|00|111 ∈ D, 0|11|000|11|0|1 6∈ DI Observation 1: Odd number of blocks implies strings in D

must start and end with the same symbol.I RE for D based on ob.1: 0(0∪1)∗0∪1(0∪1)∗1∪0∪1I Observation 2: Draw boundaries between blocks. Near

each boundary, we see substring 01 or 10. A string in Dmust have equal number of substrings of 01 and 10.

I RE for D based on ob.2: 0+(1+0+)∗∪1+(0+1+)∗

51

Example 8: E = w | In w , each 1 is immediately preceeded bya 0 and followed by a 0I Look for substring 010I 0010001000010010 ∈ E . Rewrite the string as

(001)(0001)(00001)(001)(0)⇒(0+1)(0+1)(0+1)(0+1)(0+)

I RE: (0+1)∗0+∪ ε or equally correct, 0+(10+)∗∪ ε

52

Proof of Closure Properties of RLsUnion, concatenation and star:

N*

N1

N2

Union

Concat

Star

ε

ε

ε

ε

ε

ε

ε

An example:

0,1

1

1

0 0 0,1

1

1

0

0

ε

100 accepted100 not accepted

N1 N2

N

53

RE⇒ NFA (Sipser 1.3 pp. 67-69)Since REs are defined recursively, it is suitable to construct theequivalent NFAs recursively.I Basis: NFAs for simple RE: ε, /0, and a for a ∈ Σ.I Induction: Given the NFAs for REs R1 and R2, what are

the NFAs for R1∪R2, R1R2, and R∗1?Example 1: Converting RE (0∗10∗10∗)∗ to an NFA (which NFAis correct?)

ε

0 0 0

1 1

ε

0 0 0

1 1

ε

Figure 16: Converting a RE to an NFA

54

Example 2: Converting RE (00∗1)∗(ε∪10∗)

1

0

0

0

1

ε

Figure 17: Converting a RE to an NFA

Theorem: DFA, NFA, and RE are equivalent ways to accept orrepresent RLs. In particular, RE⇒NFA, NFA⇒DFA, DFA⇒RE(didn’t discuss in our class)

55

4.1 Regular versus nonregular languagesI A = 0∗1∗I B = 0n1n|n ≥ 0I C = w ∈ 0,1∗|w has an equal number of 0s and 1sI D = w ∈ 0,1∗|w has an equal # of substrings 01 and 10

56

Proving nonregularity by pumping lemma (Sisper 1.4 pp.77-82)Theorem (The pumping lemma for regular languages):For any regular language A, there exists a constant p (whosevalue depends on A) such that ∀s ∈ A with |s| ≥ p, s can bepartitioned into three substrings s = xyz such that

1. |y |> 0;2. |xy | ≤ p; and3. ∀i ≥ 0, string xy iz ∈ A.

(Note: xy0z = xz, xy1z = xyz, xy2z = xyyz)

57

How to use the pumping lemma to prove that a language A isnot regular:I (1)Assume that A is regular by contradiction. (2)Then the

pumping lemma applies to A. (3)Let p be the constant inthe pumping lemma. (three steps to start the proof)

I Select s ∈ A with |s|= f (p)≥ p.I By the pumping lemma, ∃x ,y ,z such that s = xyz with|y |> 0, |xy | ≤ p and xy iz ∈ A for any i ≥ 0.

I For any x ,y ,z such that s = xyz, |y |> 0, and |xy | ≤ p, findi ≥ 0 such that xy iz 6∈ A. A contradiction!

58

Example 1 (Sipser p. 80): Prove that B = 0n1n|n ≥ 0 is notregular.I Assume B is RL. Then PL applies to B. Let p be the

constant in PL.I Select s = 0p1p ∈ B with |s|= 2p > pI By PL, s = 0 . . .0︸ ︷︷ ︸

p

1 . . .1︸ ︷︷ ︸p

= xyz, |y |> 0 and |xy | ≤ p.

I Since y 6= ε and |xy | ≤ p, then y = 0+

I Choose i = 0. Then xy0z = xz = 0p′1p for p′ < pI So xy0z 6∈ B. A contradiction to PL. So B is non-R.

59

Example 2 (Sipser p. 81): Prove that F = ww |w ∈ 0,1∗ isnot regular.I Assume F is RL. Then PL applies to F . Let p be the

constant in PL.I Select s = 0p10p1 ∈ F with |s|= 2p + 2 > pI By PL, s = 0 . . .0︸ ︷︷ ︸

p

10 . . .0︸ ︷︷ ︸p

1 = xyz, |y |> 0, |xy | ≤ p

I Since y 6= ε and |xy | ≤ p, then y = 0+

I Choose i = 2. Then xy2z = xyyz = 0p′10p1 for p′ > pI So xy2z 6∈ F . A contradiction to PL. So F is non-R.

60

Example 3: Prove that A = 1r |r is a prime is not regular.I Some strings in A: 11, 111, 11111, 1111111, etc..I Assume A is RL. Then PL applies. Let p be the constant.I Select s = 1q, where q is a prime and q ≥ pI By PL, s = 1 . . .1︸ ︷︷ ︸

q

= xyz, x = 1h1 (h1 ≥ 0), y = 1h2 (h2 > 0),

z = 1q−h1−h2

I By PL, ∀i ≥ 0, xy iz = 1h1+i ·h2+(q−h1−h2) = 1(i−1)h2+q ∈ AI Choose i = q + 1. Then xy iz = xyq+1z = 1q(h2+1) 6∈ AI A contradiction to PL. So A is non-R.

61

Example 4: (Sipser p. 82) Prove that D = 1n2 |n ≥ 1 is non-R.I Some strings in D: 1, 1111, 111111111, etc.I Assume D is RL. Then PL applies. Let p be the constantI Select s = 1p2 ∈ D, |s|= p2 ≥ pI By PL, s = 1 . . .1︸ ︷︷ ︸

p2

= xyz, x = 1h1 (h1 ≥ 0), y = 1h2 (h2 > 0),

z = 1p2−h1−h2 , and h2 ≤ |xy | ≤ pI By PL, ∀i , xy iz ∈ DI Choose i = 2. Consider |xy2z|

= h1 + 2h2 + (p2−h1−h2) = h2 + p2. A perfect square?I Some algebra:

p2 = 0 + p2 < (h2 + p2)≤ p + p2 < 1 + 2p + p2 = (p + 1)2

p2 < |xy2z|< (p + 1)2. So |xy2z| is not a perfect square.Then xy2z 6∈ D. A contradiction to PL. So D is non-R.

62

Example 5: Prove that A = 10n1n | n ≥ 0 is not regular.I Assume A is RL. The PL applies. Let p be the constant.I Select s = 10p1p ∈ A. |s|= 2p + 1 > pI By PL, s = 10 . . .0︸ ︷︷ ︸

p

1 . . .1︸ ︷︷ ︸p

= xyz, |y |> 0, |xy | ≤ p

I By PL, ∀i , xy iz ∈ A. Consider two cases for y .I Case 1. y contains the first 1: x = ε, y = 10∗.

Choose i = 0. xy0z = xz = 0p′1p 6∈ A, for p′ ≤ pI Case 2. y does not contain the first 1: x = 10∗, y = 0+.

Choose i = 0. xy0z = xz = 10p′1p 6∈ A, for p′ < pI For both cases, we have found contradiction to PL. So A is

non-R.

63

Example 6: Prove that A = (01)a0b|a > b ≥ 0 is not regular.I Assume A is RL. The PL applies. Let p be the constant.I Select s = (01)p0p−1 ∈ A. |s|= 3p−1.I By PL, s = 01 . . .01︸ ︷︷ ︸

2p

0 . . .0︸ ︷︷ ︸p−1

= xyz, |y |> 0, |xy | ≤ p

I Consider y which is entirely in (01)p.I Case 1: Even length, i.e., y = 01 . . .01 or y = 10 . . .10.

Choose i = 0 to remove at least a substring 01 or 10,violating a > b

I Case 2: Odd length, i.e., y = 01 . . .10 or y = 10 . . .01 ory = 0 or y = 1. Choose i = 2 to create substrings of 00 or11, which is not allowed .

I In each case listed, we can find an i such that xy iz 6∈ A. Acontradiction to the PL. So A is non-R.

64

4.2 Prove nonregularity by closure propertiesTo prove that A is non-regular, assume it is regular. Find aregular language B and a language operator that preservesregularity, and then apply the operator on A and B to get aregular language C. If C is known to be non-regular, acontradiction is found.

Example 1: Prove thatC = w ∈ 0,1∗|w has an equal # of 0s and 1s is not regular.I Assume C is regularI Let B = 0∗1∗. B is known to be RLI Let D = C∩B = 0n1n. D is known to be non-RI But D is regular by CP under intersectionI A contradiction! So C is non-R

65

Example 2: Prove that A = 0m1n | m 6= n is not regular.I Assume A is RLI 0∗1∗= 0n1n∪A (two disjoint sets)I 0n1n= 0∗1∗−AI Since 0∗1∗ and A are both RL, the difference of the two

RLs is still RL by CP.I So 0n1n is RL. A contradiction.I A is non-R

66

Example 3: Prove that A = ambncm+n | m,n ≥ 0 is non-RAbout homomorphism:h : Σ→ (Σ1)∗, e.g., h(a) = 01, h(b) = 0, h(c) = ε

h : Σ∗→ (Σ1)∗, e.g., h(ab) = 010h : h(A) = h(w) | ∀w ∈ AI Assume A is RL.I Define homomorphism h such that h(a) = 0, h(b) = 0,

h(c) = 1I h(A) = 0m0n1m+n= 0m+n1m+n= 0n1nI By CP under homomorphism, since A is RL, so is 0n1nI A contradiction. So A is non-R

67

5.1 Context-free grammars (Sipser 2.1 pp. 100-105)I CFG G = (V ,Σ,R,S), where V is the set of variables, Σ is

the set of terminals (alphabet), R is the set of rules in theform of V → (V ∪Σ)∗ (head→body), and S ∈ V is the startvariable.

I The CFG that generates all palindromes (strings that readthe same forward and backward) over 0,1 isG = (S,0,1,R,S), where R containsS→ 0S0|1S1|0|1|ε.

I Any language that can be generated by a CFG is calledcontext-free.

68

I Let u,v ,w be strings in (V ∪Σ)∗. If A→ w is a rule, thenuAv yields uwv , written uAv =⇒ uwv . We say u derives v ,written u ∗⇒ v , if ∃u1, . . . ,uk ∈ (V ∪Σ)∗ such thatu =⇒ u1 =⇒ ···=⇒ uk =⇒ v . Here, =⇒ means one stepand ∗⇒ means zero or more steps.

I Leftmost and rightmost derivations: ⇒lm ,∗⇒

lm , ⇒rm ,∗⇒

rm .

I The language of a CFG G, L(G) = w ∈ Σ∗|S ∗⇒ w.L(G) is said to be a CFL.

69

Some Simple CFGs and their CFLs

Example 1: L = 0n1n|n ≥ 0 a context-free language. It canbe generated by the following context-free grammar.

S→ 0S1|ε.

Example 2: Given a CFG G, describe L(G).

S→ AA, A→ AAA|bA|Ab|aLeftmost derivation: S⇒ AA⇒ bAA⇒ baA⇒ baaRightmost derivation: S⇒ AA⇒ Aa⇒ bAa⇒ baa

L(G) = w ∈ a,b∗|w has an even (nonzero) number of a′s.

Example 3: A CFG for simple expressions in programminglanguages:

S→ S + S|S ∗S|(S)|II→ Ia|Ib|I0|I1|a|b

70

5.2 Parse trees and ambiguity

ε

S

S

S

0

0

(a)

S

S S+

I S S

a I I

a

*

b

S

S S

S S+ I

I I a

a b

*

(c)(b)

1

1

Figure 18: Parse trees and ambiguity

(a) Parse tree and derivation S⇒ 0S1⇒ 00S11⇒ 0011(b) and (c) Two parse trees (or derivations) for string a + b ∗a.

71

Parse treesI A parse tree is a tree representation for a derivation, in

which each interior node is a variable, each leaf node iseither a terminal, or ε, and if an interior node is a variable Aand its children are X1, . . . ,Xk , then there must be a ruleA→ X1 · · ·Xk .

I Yield of a parse tree: Concatenation of the leaf nodes in aparse tree rooted at the start variable.

I Four equivalent notions:1. S ∗⇒ w ;2. S

∗⇒lm w ;

3. S∗⇒

rm w ; and4. A parse tree with root S and yield w .

72

Ambiguity in grammars and languages (Sipser 2.1 pp.105-106)I A CFG G = (V ,Σ,R,S) is ambiguous if there is w ∈ Σ∗ for

which there are at least two parse trees (or leftmostderivations).

I Grammar G: S→ S + S|S ∗S|(S)|I and I→ Ia|Ib|I0|I1|a|bis ambiguous since a + b ∗a has two parse trees.

I Some ambiguous grammars have an equivalentunambiguous grammar. For example, an unambiguousgrammar for the simple expressions is G′: S→ S + T |T ,T → T ∗F |F , F → (S)|I, and I→ Ia|Ib|I0|I1|a|b.

73

I A context-free language is said to be inherently ambiguousif all its grammars are ambiguous.

I There is no algorithm to determine whether a given CFG isambiguous. There is no algorithm to remove ambiguityfrom an ambiguous CFG. There is no algorithm todetermine whether a given CFL is inherently ambiguous.

74

Chomsky normal form (Sipser 2.1 pp. 106-109)

The Chomsky Normal Form (CNF): Any nonempty CFL withoutε has a CFG G in which all rules are in one of the following twoforms: A→ BC and A→ a, where A,B,C are variables, and a isa terminal. Note that one of the uses of CNF is to turn parsetrees into binary trees.

75

5.3 More CFGs design

Example 1: ambncm+n|m,n ≥ 0I Rewrite the pattern as ambncncm

I S→ aSc|T , T → bTc|εExample 2: ambm cndn|m,n ≥ 0∪ambncndm|m,n ≥ 0I S→ S1|S2

I S1→ AB, A→ aAb|ε, B→ cBd |εI S2→ aS2d |C, C→ bCc|ε

76

Example 3: 0m1n|m 6= nI Rewrite the language as 0m1n|m < n∪0m1n|m > n,

which is 0m1n−m1m∪0n0m−n1nI S→ S1|S2

I S1→ 0S11|A, A→ 1A|1I S2→ 0S21|B, B→ 0B|0

77

Example 5: L = aibjck ∈ a,b,c∗ | i + j 6= kThe grammar will pair a and c until running out one of the two.Then the grammar will consider the following cases.L = L1∪L2∪L3

I Case 1: i = k . Then j 6= 0, i.e., j ≥ 1. So L1 = aib+c iI Case 2: i > k . Then j ≥ 0. So L2 = aka+b∗ckI Case 3: i < k . And j 6= k − i . So L3 = aibjck−ic i

S→ aSc | S1 | S2 | S3S1→ bS1 | b (Case1: To generate b+)S2→ aS2 | S2b | a (Case 2: To generate a+b∗)S3→ bS3c | S1 | C, C→ cC | c (Case 3: To generate bj andck−i s.t. j 6= k − i)

78

Example 6: L = aibj | i 6= j and 2i 6= j.Draw an x-axis and mark two points, one for i and one for 2i .These two points divides the x-axis into three intervals: j < i ,i < j < 2i , and j > 2i .

L3

i 2i

j<i j>2ii<j<2i

L2L1

Figure 19: Intervals that j falls in

L = L1∪L2∪L3: S→ S1 | S2 | S3L1 = aibj | j < i: S1→ aS1b | A, A→ aA | aL2 = aibj | i < j < 2i: S2→ aS2b | aTb, T → aTbb | abbL3 = aibj | j > 2i: S3→ aS3bb | B, B→ Bb | b

79

Example 7: L = x#y | x ,y ∈ 0,1∗, |x | 6= |y |Consider two cases: |x |< |y | and |x |> |y |.I Case 1 |x |< |y |: x = 01 and y = 100. String

x#y = 01#100. After pairing 01 and 00, what’s left is #1.I Case 2 |x |> |y |: x = 110 and y = 00. String

x#y = 110#00. After pairing 11 and 00, what’s left is 0#.S→ 0S0 | 0S1 | 1S0 | 1S1 | T # | #TT → 0T | 1T | 0 | 1

80

6.1 PDAs (Sipser 2.2 pp. 102-114)I PDA = NFA + Stack (still with limited memory but more

than that in FAs)I PDA M = (Q,Σ,Γ,δ,q0,F ), where

I Q: A finite set of statesI Σ: A finite set of input symbols (input alphabet)I Γ: A finite set of stack symbols (stack alphabet)I δ: The transition function from Q× (Σ∪ε)× (Γ∪ε) to

2Q×(Γ∪ε)

I q0: The start stateI F : The set of final states

81

a

q pX

H H

Y

aab b b b b a b a

a

b b

Figure 20: How a transition step occurs within a PDA ifδ(q,a,X ) = (p,Y )

82

I What does δ(q,a,X ) = (p,Y ) mean? If the current stateis q, the current input symbol is a, and the stack symbol atthe top of the stack is X , then the automaton changes tostate p and replace X by Y .

I What if ε replaces a, or X , or Y? For example,δ(q,ε,X ) = (p,Y ): No cursor move. X replaced by Y(pop + push)δ(q,a,ε) = (p,Y ): Push Yδ(q,a,X ) = (p,ε): Pop Xδ(q,ε,ε) = (p,Y ): No cursor move. Push Yδ(q,a,ε) = (p,ε): No stack changeδ(q,ε,X ) = p,ε): No cursor move. Pop Xδ(q,ε,ε) = (p,ε): No change except state

I The state diagram of PDAs: For transitionδ(q,a,X ) = (p,Y ), draw an arc from state q to state plabeled with a,X → Y .

83

I Instantaneous description (ID) of a PDA: (q,w ,γ)represents the configuration of a PDA in the state of q withthe remaining input of w yet to be read and the stackcontent of γ. (The convention is that the leftmost symbol inγ is at the top of the stack.)

I Binary relation ` on ID’s: (q,aw ,Xβ) ` (p,w ,Y β) ifδ(q,a,X ) contains (p,Y ). ` represents one move of the

PDA, and∗` represents zero or more moves of the PDA.

I Language of a PDA M (or language recognized by M) is

L(M) = w |(q0,w ,ε)∗` (f ,ε,γ) for f ∈ F.

I How does a PDA check the stack is empty? At thebeginning of any computation, it pushes a special symbol $to the initially empty stack by having transitionδ(q0,ε,ε) = (q,$).

84

Example 1 (Sipser p. 112): A PDA that recognizes0n1n|n ≥ 0.

0, ε−>0 1, 0−>ε

1, 0−>εε, ε−> $ ε, −>ε$

Figure 21: An example of a PDA

85

Example 2 (Sipser p. 114): A PDA that recognizesaibjck |i , j ,k ≥ 0, i = j or i = k= anbnc∗∪anb∗cn.

ε, ε−>ε

ε, ε−>ε

ε, ε−> $ ε, −>ε$

ε, −>ε$ε−>

ε, ε−>ε

a,

b, a −>ε

ε−>εb,

ε −>εc,

−>εc, a

ai=k

i=j

Figure 22: Another PDA

Example: How the PDA in the above example accepts inputaabbcc.Theorem: The equivalence of PDA, CFG, and CFL

86

7.1 Proving non-CFLs by pumping lemma (Sipser 2.3 (pp.125-129))

Theorem 2.34 (The pumping lemma for CFLs)Let A be a CFL. Then there exists a constant p such that ∀s ∈ Awith |s| ≥ p, we can write s = uvxyz such that

1. |vy |> 0; (not allow v = y = ε)2. |vxy | ≤ p; and3. ∀i ≥ 0, string uv ixy iz ∈ A.

Recall in the PL for RLs, s is partitioned into x ,y ,z satisfying1. |y |> 0;2. |xy | ≤ p; and3. ∀i ≥ 0, string xy iz ∈ A

87

How to use the pumping lemma to prove that a language isnot Context-free?I Assume that A is context-free by contradiction. Then the

pumping lemma applies to A. Let p be the constant in thepumping lemma. (Always begin the proof with these threesteps.)

I Select s ∈ A with |s|= f (p)≥ p. (This may be tricky. Butstart with an intuitive simple string that uses p in thelength.)

I By the pumping lemma, s = uvxyz with (1)|vy |> 0,(2)|vxy | ≤ p, and (3)uv ixy iz ∈ A, ∀i ≥ 0.

I Prove for ANY u,v ,x ,y ,z such that s = uvxyz, |vy |> 0,and |vxy | ≤ p, find i ≥ 0 such that uv ixy iz 6∈ A.A contradiction to (3) in PL!

88

Example 2.36 (Sipser p.128): Prove B = anbncn|n ≥ 0 isnon-CF.Pf. Assume B is CF. Then PL applies. Let p be the constant.I Select s = apbpcp = a · · ·b · · ·c · · · ∈ B, with |s|= 3p > p.I By PL, s = uvxyz with |vy |> 0 and |vxy | ≤ p.I Consider what vxy can be. Imagine vxy is a sliding window

that moves left to right within s.I Case 1: vxy contains one symbol type (a+, b+, or c+)I Case 2: vxy contains two symbol types ( a+b+ or b+c+)I Choose i = 0 for both cases. Then uv0xy0v 6∈ B for both.I A contradiction to the PL!

89

Example 2.38 (Sipser p. 129): Prove D = ww |w ∈ 0,1∗ isnon-CF.Pf. Three sentences to start the proof. Then,I Select s = 0p10p1 = 0 · · ·010 · · ·01 ∈ D, |s|= 2p + 2 > p.I By PL, s = uvxyz with |vy |> 0 and |vxy | ≤ pI We consider the following partition:

s = 0p−1 ·0 ·1 ·0 ·0p−11, where u = 0p−1,v = 0, x = 1,y = 0, and z = 0p−11.

I For any i ≥ 0,uv ixy iz = 0p−1 ·0i ·1 ·0i ·0p−11 = 0p−1+i ·1 ·0i+p−1 ·1 ∈ D

I No contradiction. Need to choose a different s.I Try s = 0p1p0p1p and i = 0

(Exercise or read p. 129 bottom)

90

Example 3: Prove that A = 0j1j2 is non-CF.I Select s = 0p1p2

= uvxyz ∈ A, where |vy |> 0 and |vxy | ≤ p.I vxy = 0+,1+,or 0+1+ (three cases)I For the first two cases, choose i = 0 to shrink the 0 and 1

blocks, respectively, thus making the string uxz 6∈ AI Case 3: vxy = 0+1+.I Case 3.1: v or y contains both 0 and 1, i.e., v or y = 0+1+.

Let i = 2. Then uv2xy2z contains substring 0+1+0+1+,thus not in A.

I Case 3.2: v and y do not contain both 0 and 1, whichmeans that v and y each contain at most one symbol type,i.e. v = 0|v | and y = 1|y |. (Note: v or y may be ε but notboth)

91

Continue with Case 3.2I Let i = 2. Then uv2xy2z = 0p+|v | ·1p2+|y |

I Is (p + |v |)2 = (p2 + |y |)?I Is p2 + 2p|v |+ |v |2 = p2 + |y |?I Is 2p|v |+ |v |2 = |y |? Or Left = Right?I If |y |= 0 and |v | 6= 0: Left > RightI If |y | 6= 0 and |v |= 0: Left < RightI If |y | 6= 0 and |v | 6= 0:

Left = 2p|v |+ |v |2 > p ≥ |vxy | ≥ |y |= RightSo Left > Right

I So for all combinations of |v | and |y |, Left 6= Right. Souv2xy2z 6∈ A

I A contradiction!

92

Example 4: Prove that L = aibjc id j | i , j ≥ 0 is non-CF.I Assume L is CF. Then the PL applies to L. Let p be the

constant.I Select s = apbpcpdp. s ∈ L and |s|= 4p > p.I By PL, s = uvxyz with |vy |> 0 and |vxy | ≤ p.I Since |vxy | ≤ p, v and y cannot contain both a’s and c’s,

nor can it contain both b’s and d ’s. Further |vy |> 0. Wehave uv0xy0z = uxz 6∈ L, because it either contains fewera’s than c’s, or fewer c’s then a’s, or fewer b’s than d ’s, orfewer d ’s than b’s.

I A contradiction to the PL.I So L is non-CF.

93

7.2 Proving non-CFLs by closure propertiesI Closed under union: If A and B are CF, so is A∪B.

Proof: CFG GA with SA→ ··· and CFG GB with SB → ··· .Define a CFG that generates A∪B as S→ SA | SB plus thegrammars GA and GB.

I Closed under concatenation: If A and B are context-free,so is AB.Proof: CFG GA with SA→ ··· and CFG GB with SB → ··· .Define a CFG that generates AB as S→ SASB plus thegrammars GA and GB.

I Closed under star: If A is context-free, so is A∗.Proof: Consider CFG G with S1→ ··· . Define a CFG thatgenerates A∗ as S→ SS1 | ε plus the grammar G.

94

I Closed under reverse: If A is context-free, so is AR.I Not closed under intersection: Consider A = anbncm and

B = ambncn.I Not closed under complementation: Note that

A∩B = A∪B.I Not closed under difference: Note that A = Σ∗−A.

95

I Intersect with a regular language: If A is context-free and Bis regular, then A∩B is context-free.

I Difference from a regular language: If A is context-free andB is regular, then A−B is context-free. Note thatA−B = A∩B.

96

How can closure properties of CFLs be used to prove thata given language is non-CF?

Example 1: A = w ∈ a,b,c∗|na(w) = nb(w) = nc(w) isnon-CF. (Note: na(w) is defined to be the number of a′s in w .)I Assume A is CF. Let B = a∗b∗c∗, a RL.I A∩B = anbncn must be CF by the closure property of

the intersection of a CFL and a RL.I But we proved before that anbncn is non-CF.I A contradiction.I So A must be non-CF.

97

How can closure properties of CFLs be used to prove thata given language is CF?Example 2: B = aibjck |j > i + k is CF.I Since j > i + k , let j = i + k + h for some h > 0.I aibjck = ai ·bi+k+h ·ck = ai ·bi+h ·bk ·ck = (aibi)(b+)(bkck )

I This is the concatenation of three CFLs.I By the closure property of CFLs under cancatenation, B is

CF.

98

About the midterm:True/False. Cheat sheet only. 60min + 30min. 25%. AvailableFriday afternoon to Monday evening. Sample questions below.I DFA/NFA/RL:

I A DFA can have more than one final state.I The union of any infinite number of regular languages is

regular.I Any superset (subset) of a regular language is regular.

I RE:I (0∗1∗)∗ = (0∪1)∗

I (R1∪R2)∗R2 = (R∗1R2)∗

I L((0101∪01∪ ε)∗)⊂ L((01)∗∪ ε)

I CFG/CFL:I Any regular language is context-free.I The language generated by CFG G, where

S→ aaSb | aSb | a | ε, is ambiguous.I Any superset (subset) of a CFL is CF (non-CF)

99

Computability Theory: An introductionI A study of capability and limitation of computers, or

equivalently, what they can do and what they cannot.I Given a problem, can it be solved at all?I The set of all problems can be divided into two subsets.

One subset contains all solvable problems, such assorting, finding shortest path in a graph. The other subsetcontains those unsolvable problems.

I Computability Theory is to study techniques to prove if agiven problem is solvable or unsolvable.

Solvable unsolvableAll

Figure 23: Solvable vs. unsolvable

100

8.1 Unsolvable problemsI A problem is said to be unsolvable/undecidable if it cannot

be solved/decided by any algorithm.I Most interesting problems are optimization problems (OPT)I Decision problems (DEC) ask a yes-no question.I Example: The Traveling Salesman Problem (TSPOPT)

Visit every city and go back home.Input: An edge-weighted graph G = (V ,E ,w)Output: A tour (simple cycle of all vertices) with min totalweight

I Corresponding decision problem (TSPDEC)Input: G = (V ,E ,w) and B ≥ 0Question: Is there a tour in G with total weight ≤ B?

101

I Meta Claim: DEC is no harder than its corresponding OPTI So, to study hardness of an OPT, we focus on its DEC.I Any DEC is actually a language since the yes-no question

in DEC can be interpreted as asking membership of astring in a language.

I Example: Prime (DEC)Input: An integer x ≥ 2Question: Is x a prime? (This is a yes/no question.)

I Lprime = < x > |x is prime (This is a language)Lprime is actually the language of all prime numbersencoded in binary representation.

102

I Encoding anything to a binary string:I Integer x to binary string < x >I Graph G to < G >I Matrix M to < M >I List L to < L >

I Revisit TSPDEC and its corresponding language :I Input (or Instance): G and B ≥ 0

Question: Does G contain a tour with the total weight ≤ B?I LTSPDEC = < G,B > |There is a tour with total weight ≤ B

Is string < G,B > a member of language LTSPDEC?

103

I The number of languages over a non-unary alphabet isuncountably infinite. So is the number of DECs (or decisionproblems).

I However, the number of programs that a computer can useto solve problems is countably infinite. Therefore, there aremore problems than there are programs. Thus, there mustbe some unsolvable problems.

104

I An unsolvable (or undecidable) problem :The famous Halting Problem (by Turing):I Input: Any Turing Machine M and any string sI Question: Does M halt on s?

I The modern version:I Input: Any program P and any input II Output: “Yes” if P terminates on I and “No” otherwise.

Or Question: Does P terminate on I?

AlgorithmP

I

Yes

No

Figure 24: Does P terminate/halt on I?

105

8.2 Turing machine (Sipser 3.1, pp. 165-175)I A Turing machine includes a control unit, a read-write

head, and a one-way infinite tape.

δ(

R/W head

0 1 1 1 110 0 0 1 0 B B

q q, 1)=(p, X, L)

Tape and tape squares Infinitely long

Figure 25: Picture of a Turing Machine

106

I TM M = (Q,Σ,Γ,δ,q0,qaccept ,qreject ), whereI Q: The finite set of states for the control unit.I Σ: An alphabet of input symbols, not containing the “blank

symbol”, B.I Γ: The complete set of tape symbols. Σ∪B ⊂ Γ.I δ: The transition function from Q×Γ to Q×Γ×D, where

D = L,R.I For example, δ(q,0) = (p,X ,L) and δ(p,Y ) = (q,B,R).I q0: The start state.I qaccept : The accept state.I qreject : The reject state.

107

I Configuration: Use a string to describe the look of a TM ata certain time, instead of drawing a picture of the TM.For example, string X1 · · ·Xi−1qXi · · ·Xn gives a description(snapshot) of the TM at a time, when the current state is q,the tape content is X1 · · ·Xn, and the head is scanning(pointing to) Xi . Such a string is called the configuration ofthe TM at a certain time.

108

I How a TM changes its configurations:I If δ(q,Xi ) = (p,Y ,L), then

X1 · · ·Xi−1qXi · · ·Xn ` X1 · · ·Xi−2pXi−1YXi+1 · · ·Xn.I If δ(q,Xi ) = (p,Y ,R), then

X1 · · ·Xi−1qXi · · ·Xn ` X1 · · ·Xi−1YpXi+1 · · ·Xn.

p

Xi−1X1... Xi

... Xn

q

B X X1

i−2 Xi−1 X ......i+1 Xn

p

Y

δ( q, Xi)=(p, Y, L)

B

X X X1 i−1...

i... Xn B

q

X X Y X... ... X1 i−1 i+1 n B

δ( q, Xi)=(p, Y, R )

Figure 26: Transitions applied on configurations

109

I Three important configurations:(1) Starting configuration q0w ,(2) accepting configuration uqacceptv ,(3) rejecting configuration uqrejectv ,where (2) and (3) are called the halting configurations.

I Language of a Turing machine M (or languagerecognized/accepted by M) is

L(M) = w ∈ Σ∗|q0w∗` αqacceptβ for any α,β ∈ Γ∗.

I Note: To produce `, type ”backslash vdash” in the mathmode.

I For any given input, a TM has three possible outcomes:accept, reject, and loop. Accept and reject mean that theTM halts on the given input, but loop means that the TMdoes not halt on the input.

110

I TRL: A language A is Turing-recognizable if there is a TMM such that A = L(M). In other words,I ∀w ∈ A, M accepts w by entering qaccept .I ∀w 6∈ A, M does not accept (i.e., it may reject or loop).

I TDL: A language A is Turing-decidable if there is a TM Msuch that A = L(M) and M halts on all inputs. In otherwords,I ∀w ∈ A, M accepts w .I ∀w 6∈ A, M rejects w .

Such TMs are a good model for algorithms.

A TM M to decide

acceptM

A TM to accept/recognize/verify

Mww in A

accept

reject

Figure 27: TRLs vs. TDLs

111

How to design a TM that recognizes/accepts a language?

Example 1: Give a implementation-level description of a TM Mthat accepts 0n1n|n ≥ 0, i.e., L(M) = 0n1n|n ≥ 0

Idea: w = 000111⇒ X00Y11⇒ XX0YY1⇒ XXXYYY

M =”On input string w = 0n1n

1. If w = ε, accept2. Mark the first 0 with X , move right to mark the first 1 with Y3. Move left to find the leftmost 0. If no 0, accept, else go to

stage 2”

112

Example 1: (more) Define a TM that accepts 0n1n | n ≥ 0δ 0 1 X Y Bq0 (q1,X ,R)1 - - (q3,Y ,R)8 (qa,B,R)0

q1 (q1,0,R)2 (q2,Y ,L)4 - (q1,Y ,R)3 -q2 (q2,0,L)6 - (q0,X ,R)7 (q2,Y ,L)5 -q3 - - - (q3,Y ,R)9 (qa,B,R)10

qa - - - - -

Y−>Y, R

q0 q1

q2q3 qa

0−>X, R

B −> B, R

B −> B, R X −> X, R1 −> Y, L

Y −> Y, R0 −> 0, R

Y −> Y, L0 −> 0, L

Y−>Y, R

Figure 28: Transition diagram for TM

113

Example 2: Give an implementation-level description of a TMM that decides 0n1n|n ≥ 0.

M = ”On any input string w ∈ 0,1∗

1. If w 6= 0∗1∗, reject2. Sweep left to right. If no 0 and 1 are found, accept. If only

0 is found or 1 is found, but not both, reject. If both 0 and 1are found, go to stage 3

3. Mark the leftmost 0 with X. Move head to right to find andmark the first 1 with Y

4. Move head to left end, and then go to stage 2”

114

Example 3.7 (Sipser p. 171): Give a TM M that decidesA = 02n |n ≥ 0= 0,00,0000,00000000, · · ·.

Consider the following strings to figure out an algorithm (TM):(1) odd length, e.g., w1 = 00000⇒ 0X0X0;(2) even length, e.g.,w2 = 00000000⇒ 0X0X0X0X ⇒ 0XXX0XXX ⇒ 0XXXXXXX ;w3 = 000000⇒ 0X0X0X

TM M = ”On input string w ∈ 0∗:1. Sweep left to right, crossing off every other 02. If in stage 1 the tape contained a single 0, accept (e.g., w2)3. If in stage 1 the tape contained more than a single 0 and

the number of 0s was odd, reject (e.g., w1 and w3)4. Move head to the left end of the tape5. Go to stage 1”

115

8.4 Variations of TMs (Sipser 3.2 (pp. 148-159))I TM with multi-tapes (and multi-heads)

(δ : Q×Γk →Q×Γk ×L,Rk ).I TM with multi-strings (and multi-heads).I TM with multi-heads.I TM with multi-tracks.I TM with two-way infinite tape.I TM with multi-dimensional tape.I Nondeterministic TM’s (δ : Q×Γ→ 2Q×Γ×D).

Consider a move in NTM, δ(q3,X ) = (q5,Y ,R),(q3,X ,L).How does the NTM know which step it should take? Oneway to look at this is: the NTM is the ”luckiest possibleguesser” and it always picks a transition that eventuallyleads to an accepting state if there is such a transition.

116

Theorem: The equivalent computing power of the aboveTM’s:

For any language L, if L = L(M1) for some TM M1 withmulti-tapes, multi-strings, multi-heads, multi-tracks, two-wayinfinite tape, multi-dimensional tape, or nondeterminism, thenL = L(M2) for some basic TM M2.

117

Theorem: The equivalent computing speed of the above TM’sexcept for nondeterministic TM’s:

For any language L, if L = L(M1) for some TM M1 withmulti-tapes, multi-strings, multi-heads, multi-tracks, two-wayinfinite tape, or multi-dimensional tape in a polynomial numberof steps, then L = L(M2) for some basic TM M2 in a polynomialnumber of steps (with a higher degree).

Or in other words, all reasonable models of computation cansimulate each other with only a polynomial loss of efficiency.

Note: The speed-up of a nondeterministic TM vs. a basic TM isexponential.

118

The Church-Turing Thesis:

Any reasonable attempt to model mathematically algorithmsand their time performance is bound to end up with a model ofcomputation and associated time cost that is equivalent toTuring machines within a polynomial. (The power of TM.)

119

Nondeterministic TMsI δ : Q×Γ→ 2Q×Γ×D.I Consider a move in NTM, δ(q3,X ) = (q5,Y ,R),(q3,X ,L).

How does the NTM know which step it should take?I One way to look at this is that the NTM is the ”luckiest

possible guesser” and it always picks a transition thateventually leads to an accepting state if there is such atransition.

I The other way is to imagine that the NTM branches intomany copies, each of which follows one of the possibletransition.

I DTM (path) versus NTM (tree): See the wiki page for”Nondeterministic Turing Machine”.

120

8.3 Properties of TDLs and TRLs

Theorem: A TDL is also a TRL, but not vice versa.Theorem: About A and A:

1. If A is Turing-decidable, so is A.2. If A and A are both Turing-recognizable, then A is

Turing-decidable. (See Theorem 4.22, p.210)3. For any A and A, we have one of the following possibilities:

(1) Both are Turing-decidable;(2) Neither is Turing-recognizable;(3) One is Turing-recognizable but not decidable, the otheris not Turing-recognizable.

TDL

ALL

TRL

Figure 29: A language and its complement

121

Some closure properties:TRLs and TDLs are both closed underI UnionI IntersectionI ConcatenationI Star

In addition, TDLs are closed under complement, and TRLs areclosed under homomorphism.

122

Examples to prove closure properties:

Example 1: If L1 and L2 are TD, so is L1∪L2.

Pf: Let TM M1 and TM M2 decide L1 and L2, respectively. Thenwe have the following TM M to decide L1∪L2.

TM M = ”On input w :1. Run M1 on w2. If M1 accepts, accept3. else run M2 on w4. If M2 accepts, accept5. else reject”

123

Example 2: If L1 and L2 are TR, so is L1∪L2.

Pf: Let TM M1 and TM M2 recognize L1 and L2, respectively.Then we have the following TM M to recognize L1∪L2.

TM M = ”On input w ∈ L1∪L2

1. Run M1 and M2 alternately on w , one step at a time2. If either accepts, accept”

124

Example 3: If L is TD, so is L∗.

Pf: Let TM M decide L. Then we have the following TM M∗ todecide L∗.Note: w ∈ L∗ if w = w1w2 · · ·wk for some k ∈ [1, |w |], wherewi ∈ L for i = 1, · · · ,k .

TM M∗ = ”On input w1. If w = ε, accept2. ∀k = 1,2, · · · , |w |3. ∀ partitions of w into k substrings, i.e., w1,w2, · · · ,wk

4. Run M on w1,w2, · · · ,wk

5. If M accepts wi , ∀i = 1, · · ·k , accept6. reject”

125

Example 4: If L is TR, so is L∗

Pf: Let TM M recognize L. Then we have the following NTM Nto recognize L∗

NTM N = ”On input w ∈ L∗

1. Nondeterministically generate (guess) a partition of w intow1,w2, · · · ,wk

2. Run M on w1,w2, · · · ,wk

3. If M accepts wi , ∀i = 1,2, · · · ,k , accept”

126

9.1 A binary encoding scheme for TMsI TM⇔ binary number.

Q = q1,q2, . . . ,q|Q| with q1 to be the start state, q2 to bethe accept state, and q3 to be the reject state.Γ = X1,X2, . . . ,X|Γ|.D = D1,D2 with D1 to be L and D2 to be R.A transition δ(qi ,Xj) = (qk ,Xl ,Dm) is coded as0i10j10k10l10m.A TM is coded as C111C211 · · ·11Cn, where each C is thecode for a transition.

I An example: δ(q2,X3) = (q1,X4,D1) can be coded as001000101000010

I An example: 000010010100100 is the encoding ofδ(q4,X2) = (q1,X2,D2)

127

I TM M with input w is represented by < M,w > andencoded as < M > 111w .

I Using similar schemes, we can encode DFA, NFA, PDA,RE, and CFG into binary strings.

128

9.2 Decidable languages (Sipser 4.1, pp. 194-201)I ADFA = < B,w > |B is a DFA that accepts string w.I ANFA = < B,w > |B is an NFA that accepts string w.I AREX = < R,w > |R is a RE that generates string w.I EDFA = < B > |B is a DFA and L(B) = /0.I EQDFA = < B1,B2 > |B1 and B2 are DFAs and L(B1) =

L(B2).I ACFG = < G,w > |G is a CFG that generates string w.I ECFG = < G > |G is a CFG and L(G) = /0.I Every CFL is decidable.

Note: The proofs of these TDLs can be found in Sipser’s book.

129

Diagonalization: A proof methodI The size of an infinite set: Countably infinite (or countable)

and uncountably infinite (or uncountable).I A set A is countable if there is a 1-1 correspondence with

N = 1,2,3, . . . (the set of natural numbers).I The following sets are countable.

1. The set of even (or odd) numbers2. The set of rationale numbers3. The set of binary strings4. The set of TMs

I But, the set of languages is uncountable. (How to provethis?)

I So, there are more languages than there are TMs. Sothere must be languages that are non-TRL.

130

Example (Sipser p. 205): R, the set of real numbers, isuncountable.Pf: Assume R is countable. Then R = r1, r2, . . ..

n f (n)

r1 3. 14159 . . .r2 55. 55555 . . .r3 0. 12345 . . .r4 0. 50000 . . .· · · · · · · · ·

1. Consider the fractions after the decimal point of each realnumbers. Focus on the diagonal, 1,5,3,0, · · · .2. Define a new real number r = 0.d1d2d3d4 · · · , where di canbe anything but the i th number in the diagonal, e.g.,r = 0.2401 · · · .3. So r cannot be r1, r2, r3, r4, · · · , thus r can not be in the table.4. Contradiction to that all real numbers are in the table.

131

9.4 A non-TRLI Consider the binary alphabet.I Order and label strings: ε,0,1,00,01,10,11, · · · .

Let wi be i th string in the above lexicographic ordering.I Order and label TMs: M1,M2,M3, · · · .

Let Mi be the TM whose code is wi , i.e. < Mi >= wi .In case wi is not a valid TM code, let Mi be the TM thatimmediately rejects any input, i.e., L(Mi) = /0.

ε 0 1 00 01 10 11 000 · · ·w1 w2 w3 w4 w5 w6 w7 w8 · · ·M1 M2 M3 M4 M5 M6 M7 M8 · · ·

I For any string wi , there is a TM Mi

I For any TM Mi , there is a string wi , where < Mi >= wi .

132

I Define diagonalization language AD = wi |wi 6∈ L(Mi).I The corresponding decision problem:

Input: Any binary string wiQuestion: Is wi not accepted by Mi?

I Prove that AD is non-TR (not a TRL).Proof:

1. Suppose, by contradiction, AD is TR, i.e., there is a TM Msuch that AD = L(M).

2. Then M = Mi with code wi for some i .3. wi ∈ AD iff wi 6∈ L(Mi ) by definition of AD.4. wi ∈ AD iff wi ∈ L(Mi ) by AD = L(Mi ).5. A contradiction within the two iff statements

.

133

9.5 A TRL but non-TDL (Sipser 4.2 (pp. 173-174 and179-182))I A universal TM:

I Each TM (among those discussed) can only solve a singleproblem, however, a computer can run arbitrary algorithms.Can we design a general-purposed TM that can solve awide variety of problems just as a computer?

I Theorem: There is a universal TM U which simulates anarbitrary TM M with input w and produces the same output.TM U = ”On input < M,w >

Run M on w”I TM U is an abstract model for computers just as TM M is a

formal notion for algorithms.

M<M, w>

w

Figure 30: The universal Turing machine

134

A review of some new concepts from the last lectureI Encoding of TM, DFA, NFA, PDA, RE, Graph, Matrix, list,

etc. to binary strings: e.g., < M >, < M,w >, < M1,M2 >

I Infinite sets: Countable vs. uncountable. Compare the setof TMs (countable) vs. the set of languages (uncountable).There are languages without a TM to accept/recognize.

I Diagonalization: a method of self-referencing, used todemonstrate certain properties are inherently contradictory.For example, Russell’s paradox and Barber’s paradox

I Correspondence between binary strings and Turingmachines, i.e., for any wi , there is a Mi and for any M,there is i s.t. < M >= wi . Thus M can be renamed as Mi

I The diagonalization language AD = wi | wi 6∈ L(Mi)non-TR.

I Universal TM U: An abstract model for computers

135

I Let ATM = < M,w > |M accepts string wOr equivalently ATM = < M,w > |w ∈ L(M)Or equivalently as a decision problemInput: A TM M and a string wQuestion: Is w accepted by M?

ATM is called the universal language.

ATM is TR since it can be recognized by TM U.

TM U = ”On input < M,w >∈ ATMRun M on wIf M accepts w , accept”

136

I ATM = < M,w > |w ∈ L(M) is non-TD. (By C&C)1. Assume that ATM is decided by TM T .

T<M,w>a

r

Input to T is < M,w >Output from T is accept if w ∈ L(M) and reject if w 6∈ L(M)

2. On input < M,w >, T accepts < M,w > iff M accepts w .(We can also say, T rejects < M,w > iff M rejects w .)

3. Define TM D as follows:

<M,<M>><M>

a

r a

rT

4. Observe that D accepts < M > iff T rejects < M,< M >>.5. Feed < D > to D.

<D,<D>>

a

r a

rT<D>

6. From steps 4 and 2, D accepts < D > iff T rejects< D,< D >> iff D rejects < D >.

7. A contradiction in step 6.

137

I ATM = < M,w > |w ∈ L(M) is non-TD.Proof by contradiction:

1. Assume that ATM is decided by TM T .Input to T is < M,w >Output from T is accept if w ∈ L(M) and reject if w 6∈ L(M)

2. On input < M,w >, T accepts < M,w > iff M accepts w .(We can also say, T rejects < M,w > iff M rejects w .)

3. Define TM D = ”On input < M >Run T on < M,< M >>If T accepts, rejectIf T rejects, accept

4. Observe that D accepts < M > iff T rejects < M,< M >>.5. Feed < D > to D.6. From steps 4 and 2, D accepts < D > iff T rejects

< D,< D >> iff D rejects < D >.7. A contradiction in step 6.

138

10.1 A summary of terminology in Computability TheoryI Language, Decision Problem, ProblemI TM, Algorithm, SolutionI Decide, Solve, (Decidable, Solvable)I Undecidable, UnsolvableI Accept, Recognize, (Acceptable, Recognizable)

139

10.2 A review of some languages and correspondingdecision problemsI AD = wi |wi 6∈ L(Mi) (non-TR)

Input: Any binary string wiQuestion: Is wi not accepted by Mi?

I ATM = < M,w > |w ∈ L(M) (TR but non-TD)Input: TM M and string wQuestion: Does M accept w?It is undecidable whether TM M accepts string w for anygiven M and w .

I HALTTM = < M,w > |M halts on w (TR but non-TD)Input: TM M and string wQuestion: Does M halt on w?It is undecidable whether TM M halts string w for any givenM and w .

140

10.3 Reducibility or Reduction(Sipser 5 (pp. 216-220))

We say that problem A reduces (or is reducible) to problem B,written as A≤ B, if we can use a solution (TM) to B to solve A(i.e., if B is decidable/solvable, so is A.).We may use reducibility to prove undecidability as follows:

1. Let A be non-TD, such as AD or ATM . Wish to prove B isnon-TD.

2. Assume B is TD. Then there exists a TM MB to decide B.3. If we can use MB as a sub-routine to construct a TM MA

that decides A, then A is TD. We have a contradiction.4. The construction of TM MA using TM MB establishes that A

reduces to B, i.e., A≤ B. (A is no harder than B)5. Corollary 5.23 (Sipser p. 236):

If A≤ B and A is non-TD, then B is non-TD.

141

10.4 A proof of the non-TD ATM by reductionI Proof sketch:

1. Assume ATM is TD, by contradiction.

2. Let TM S decide ATM , by the definition of a TDL.

3. Try to construct a TM D that decides AD. The constructionwill include TM S. This shows AD is TD.

4. A contradiction since we know AD is non-TR.

Note: This proof uses the TM S for ATM to build a TM D forAD, i.e., AD ≤ ATM .

I Recall two languages:1. ATM = < M,w > |w ∈ L(M).

2. AD = wi |wi 6∈ L(Mi ). (AD is non-TR)

142

Prove that ATM is non-TD by reduction1. Assume ATM is TD, by contradiction.

2. Let TM S decide ATM , i.e.,

S(< M,w >) =

accept w ∈ L(M)

reject w 6∈ L(M)

3. Construct a TM D that decides AD, a non-TRL.TM D =”On input wi

Run S on < Mi ,wi >If S accepts, reject else accept”

4. Why does D decide AD?S accepts < Mi ,wi > iffwi ∈ L(Mi) iff wi 6∈ AD iff D rejectswi . So S accepts iff D rejects.

5. So AD is TD. A contradiction.

143

10.5 The halting problem (Theorem 5.1 (pp. 216-217))I HALTTM = < M,w > |M halts on string w.I ATM = < M,w > |M accepts wI ATM ⊆ HALTTM

I HALTTM is TR since it can be recognized by TM U.I Theorem 5.1 HALTTM is non-TD.

(Will show ATM ≤ HALTTM )Theorem HALTTM is non-TD. Prove by reduction from ATM , i.e.,ATM ≤ HALTTM

144

1. Assume TM R decides HALTTM . Then R accepts < M,w >iff M halts on w . Construct TM S to decide ATM .

TM S = ”On input < M,w >Run R on < M,w >if R rejects, rejectif R accepts, run M on w until it halts

if M accepts, accept; else reject”2. Why does S accept ATM?

R rejects < M,w >⇒ M doesn’t halt on w ⇒ M doesn’taccept w ⇒ < M,w >6∈ ATM ⇒ S rejectsR accepts < M,w >⇒ M halts on w (accepts or rejects?Need to run M on w to find out)M accepts w ⇒ < M,w >∈ ATM

3. Since we constructed a TM S that decides ATM using TMR, so ATM is TD. A contradiction to that ATM is proved to benon-TD.

145

10.6 Other non-TD problems (Sipser 5.1 (pp. 216-220))The following problems about Turing machines are non-TD:I Whether L(M) = /0 for any TM M.

ETM = < M > |L(M) = /0NETM = < M > |L(M) 6= /0 (complement of ETM )

I Whether L(M1) = L(M2) for any two TMs M1 and M2.EQTM = < M1,M2 > |L(M1) = L(M2)

I Whether L(M) is finite for any TM MFINITETM = < M > |L(M) is finite

I Whether ε ∈ L(M) for any TM M.ESTRINGTM = < M > |ε ∈ L(M)

I Whether L(M) = Σ∗ for any TM M.ALLTM = < M > |L(M) = Σ∗

Rice’s Theorem: Every nontrivial property of the TRLs (orTMs) is undecidable.

146

Theorem 5.2 Prove that ETM = < M > |L(M) = /0 is non-TD.Proof: (1) Assume that ETM is decidable by TM R.

R(< M >) =

accept L(M) = /0

reject L(M) 6= /0

(2) Use R to construct TM S that decides ATM , i.e., ATM ≤ ETM .TM S = ”On input < M,w >,I Construct TM M1 = ”On input x

If x 6= w reject else run M on w”I Run R on < M1 >.I If R accepts, reject; and if R rejects, accept”

(3) Why does S decide ATM? L(M1) = /0 if M does not accept w ;and L(M1) = w if M accepts w . I.e., L(M1) = /0 iff w 6∈ L(M).So R accepts < M1 > iff L(M1) = /0 iff w 6∈ L(M) iff S rejects.(4) TM S decides the non-TD ATM . A contradiction.

147

A graphical explanation of the undecidability proof of ETM

S

R<M>a if L(M)=empty

r otherwise

Ra

r

a

r<M, w> <M >1

Figure 31: Reduction from ATM to ETM

Important questions to answer:I Input: how to define M1 (the input to R) using < M,w >

(the input to S)?I Output: how the output from R implies the output from S?

Goal: Design M1 such that the output from R defines that of S.

148

Prove that NETM = < M > |L(M) 6= /0 is TR but non-TD.(1) To prove NETM is TR, we give a NTM N to recognize NETM .NTM N = ”On input < M >∈ NETM

I Guess a string wI Run M on wI If M accepts, accept”

We can also use a deterministic TM to recognize NETM .TM D = ”On input < M >∈ NETMRecall the binary sequence w1,w2,w3, . . .

I Systematically generates strings: ε, 0, 1, 00, 01, . . .I for i = 1,2,3, . . .

Run M on w1, · · · ,wi , each for i stepsI If in the loop above, M ever accepts some wj , then accept”

149

An explanation of the TM D that recognizes NETM :

Assume w9 is accepted by M in 7 steps.Assume w10 is accepted by M in 12 steps.

i = 1: Run M on w1 for 1 step;i = 2: Run M on w1,w2 each for 2 steps;i = 3: Run M on w1,w2,w3 each for 3 steps;· · · · · ·i = 9: Run M on w1,w2, · · · ,w9 for 9 steps; (accepted)· · · · · ·i = 12: Run M on w1,w2, · · · ,w10, · · · ,w12 for 12 steps(accepted)

150

(2) To prove NETM = < M > |L(M) 6= /0 is non-TD, assume it isdecided by TM R. Then R accepts < M > iff L(M) 6= /0.Construct a TM S that decides the undecidable ATM . Then acontradiction.TM S = ”On input < M,w >

1. Construct TM M1=”On input xIf x 6= w , reject else Run M on w”

2. Run R on < M1 >

3. If R accepts, accept; else reject”Why does S accept ATM?L(M1) = /0 if w 6∈ L(M) and L(M1) = w if w ∈ L(M). In otherwords, L(M1) 6= /0 iff w ∈ L(M).R accepts < M1 > iff L(M1) 6= /0 iff w ∈ L(M) iff < M,w >∈ ATMiff S accepts < M,w >.So ATM is TD. A contradiction.

151

About ETM and its complement NETM

We proved: ETM is non-TD. NETM is TR.

Recall the theorem on page 120 . For A and A,1. Both are TD; (Both are TR)2. Neither is TR;3. One is TR but non-TD, the other is non-TR

We immediately have the following results.

(1) NETM is non-TD (If NETM is TD, so is ETM )

(2) ETM is non-TR (If ETM is TR, both ETM and NETM are TD)

152

Theorem 5.4 EQTM = < M1,M2 > |L(M1) = L(M2) is non-TD.Reduce from ETM = < M > |L(M) = /0.

1. Assume EQTM is decided by TM R.

R(< M1,M2 >) =

accept L(M1) = L(M2)

reject L(M1) 6= L(M2)

2. Construct TM S that decides the undecidable ETM .TM S =”On input < M >

Construct TM M1 =”On input xImmediately reject” //L(M1) = /0

Run R on < M1,M >If R accepts, accept; else reject”

3. Why does S decides ETM? R accepts < M1,M > iffL(M1) = L(M) iff L(M) = /0 iff S accepts < M >.

4. S decides ETM . So ETM is TD. A contradiction.

153

10.7 Post’s correspondence problem (PCP) (Sipser 5.2)

INPUT: P = t1b1, t2

b2, . . . , tk

bk, where t1, t2, . . . , tk and b1,b2, . . . ,bk

are strings over alphabet Σ. (P is a collection of dominos, eachcontaining two strings, with one stacked on top of the other.)

QUESTION: Does P contain a match?Or, is there i1, i2, . . . , il ∈ 1,2, . . . ,k with l ≥ 1 such thatti1 ti2 · · · til = bi1bi2 · · ·bil ?

Equivalently, defined as a language, we haveLPCP = < P > |P is an instance of PCP with a match.

For input P1 = bca ,

aab ,

caa , abc

c , sequence 2,1,3,2,4 indicates amatch. Since a

abbca

caa

aab

abcc , top=bottom=abcaaabc

For P2 = abcab , ca

a , accba , there is no match since all top strings

are longer than bottom strings..

PCP is non-TD for the binary alphabet.

154

A Summary of Computability Theory1. Definitions and concepts:I Turing machine, how it works, its language, its encoding,

Church-Turing ThesisI TRL and TDL, properties, how M accepts/decides a

language, implementation-level descriptionI Reduction, the meaning of A≤ B (A is no harder than B),

use reduction to prove undecidability2. Various proofs:I A language is TR/TD (prove by definition)I A language is non-TR/non-TD (prove by a combination of

contradiction, construction, and reduction)I Many examples to learn from

155

Complexity Theory:I Computability Theory is the study of what can or cannot be

computed by a TM/algorithm, among all problems.I Complexity Theory is the study of what can or cannot be

computed efficiently by a TM/algorithm, among alldecidable/solvable problems.

I For the set of all solvable problems, it is further classifiedinto various complexity classes based on the efficiency ofalgorithms solving these problems.

I Complexity Theory is the study of the definition andproperties of these classes.

156

PRL

CFL

CSL

TDL

TRL

ALL

All solvable problems

NP

NPC

If P is not equal to NP

TDLTDL

All solvable problems

If P is equal to NP

P=NP=NPC

Figure 32: Three complexity classes if P=NP or P6=NP

157

11.1 The class of P (Sipser 7.2)I Definition: P is the class of problems solvable in

polynomial time (number of steps) by deterministic TMs.Polynomial O(nc), where n is input size and c is a constant.Problems in P are ”tractable” (not so hard).

I Why use polynomial as the criterion?I If a problem is not in P, it often requires unreasonably long

time to solve for large-size inputs.I P is independent of all models of computation, except

nondeterministic TM.I Problems in P: Sorting, Searching, Selecting, Minimum

Spanning Tree, Shortest Path, Matrix Multiplication, etc.I Review of asymptotic notation: O, Ω, Θ

I Examples of polynomial and polylog functions: O(1), O(n),O(n2), O(nd ), O(logn), O((logn)c), O(n3 logn),O(nc(logn)d )

158

11.2 The class of NP (Sipser 7.3)I An NTM is an unrealistic (unreasonable) model of

computing which can be simulated by other models with anexponential loss of efficiency.It is a useful concept that has had great impact on thetheory of computation.

I NTM N = (Q,Σ,Γ,δ,q0,qaccept ,qreject ), whereδ : Q×Γ→ 2P for P = Q×Γ×L,R.

I δ(q,X ) is a set a moves. Which one to choose? This isnondeterminism. The computation can be illustrated by atree, with each node representing a configuration.

159

I Nondeterminism can be viewed as a kind of parallelcomputation wherein multiple independent processes orthreads can be running concurrently.When a nondeterministic machine splits into severalchoices, that corresponds to a process forking into severalchildren, each then proceeding separately. If at least oneprocess accepts, then the entire computation accepts.

I Time complexity of nondeterministic TMs (NTMs): Let Nbe an NTM that is a decider (where all computation pathshalt in the tree for any input). The time complexity of N,f (n), is the maximum number of steps that N uses on anycomputation path for any input of length n. In other words,f (n) is the maximum height of all computation trees for allinput of length n.

160

I An unreasonable model of computation:

Theorem; Every T (n)-time multi-tape TM has anequivalent O(T 2(n))-time single-tape TM.

Theorem: Every T (n)-time single-tape NTM has anequivalent O(2O(T (n)))-time single-tape DTM.

I Definition: NP is the class of problems solvable inpolynomial time by nondeterministic TMs.

I Another definition of nondeterministic TMs (algorithms):I Guessing phase: Guess a solution (always on target).I Verifying phase: Verify the solution.

161

Example: TSP (DEC) is in NP.

INSTANCE: An edge-weighted graph G(V ,E ,w) and a boundB ≥ 0.

QUESTION: Is there a tour (a cycle that passes through eachnode exactly once) in G with total weight no more than B?Define the following NTM N to solve TSP in polynomial time.

NTM N =”On input < G,B >

1. Nondeterministically guess a tour T O(|V |)2. Verify if T includes every node once O(|V |)3. Compute sum← ∑e∈T w(e) O(|V |)4. Verify if sum ≤ B. If true, answer yes; else No O(1)

Note 1: The time complexity of N is O(|V |).Note 2: If the answer to the QUESTION is ”Yes”, N guaranteesthat the right T will be guessed in step 1.

162

Note 3: The acceptance of an input by a nondeterministicmachine is determined by whether there is an acceptingcomputation among all, possibly exponentially many,computations.

In the above proof, if there is a solution, i.e., a tour with totalweight no more than B, it will always be generated by theTuring machine. This is like that a nondeterministic machinehas a guessing power.

A tour can only be found by a deterministic machine inexponential time, however, it can be found by a nondeterministicmachine in just linear steps. Any nondeterministic proof shouldalways contain two stages: Guessing and verifying.

What needs to be guessed? What needs to be verified? Whatis the time complexity?

163

Example: Graph Coloring (GC) is in NP.

INSTANCE: Graph G = (V ,E), and B ≥ 0

QUESTION: Is there a coloring scheme of the nodes that usesno more than B colors such that no two nodes connected by anedge are given the same color?

NTM N =”On input < G,B >

1. Guess a coloring scheme (in polynomial time) c : V → C2. Verify if (1) |C| ≤ B and (2) ∀(u,v) ∈ E ,c(u) 6= c(v)

3. if true, answer yes; else answer noSo we have a nondeterministic algorithm (or TM) thatguesses a coloring scheme (or function) and verifies that (1)for any (u,v) ∈ E , c(u) 6= c(v) and that (2) the number of colorsused is no more than B, and further, all these can be done inpolynomial time of O(|V |) + O(|E |). So GC is in NP.

164

I Theorem: P⊆NP.(Two possibilities: P ⊂ NP or P = NP)Any deterministic TM is a special case of nondeterministicTMs.

I Theorem: Any Π ∈NP can be solved by a deterministic TMin time O(cp(n)) for some c > 0 and polynomial p(n).

I Open problem: P=NP?The west wall bricks on the CS building at Princeton,1989:x1010000xx0111101xx1001110xx1010000xx0111111x

165

11.3 Polynomial reduction (Sipser 7.4)I Definition of Polynomial Reduction ≤p (cf. ≤m and ≤)

Let Π1 and Π2 be two decision problems, and I1 and I2be sets of instances for Π1 and Π2, respectively.We say there is a polynomial reduction from Π1 to Π2, orΠ1 ≤p Π2, if there is f : I1→ I2 such that(1) f can be computed in polynomial time and(2) I1 has a “yes” solution if and only if f (I1) has a “yes”solution.

I Theorem: If Π1 ≤p Π2, then Π2 ∈P implies Π1 ∈P.I Theorem: If Π1 ≤p Π2 and Π2 ≤p Π3, then Π1 ≤p Π3.I Remark: ≤p means “no harder than”.

π2

Yes

No

Yes

NoIπ1 fIπ2 Alg

Algπ1

Figure 33: Polynomial reduction Π1 ≤p Π2

166

11.4 The class of NPC (Sipser 7.4)I Definition 1: NPC (NP-complete) is the class of the hardest

problems in NPI Definition 2: Π ∈NPC if Π ∈NP and ∀Π′ ∈NP, Π′ ≤p Π.I Definition 3: Π ∈NPC if Π ∈NP and ∃Π′ ∈NPC such that

Π′ ≤p Π

I Theorem: If ∃Π ∈NPC such that Π ∈P, then P=NP.I Theorem: If ∃Π ∈NPC such that Π 6∈ P, then P6=NP.

167

Some most important classes: Definitions and proofsI P: class of problems solvable in polynomial-time by DTM.

To prove Π ∈P, design a polynomial-time algorithm.I NP: class of problems solvable in polynomial-time by NTM.

To prove Π ∈NP, design a polynomial-timenondeterministic algorithm of two steps: guess and verify.

I NPC: class of all hardest problems in NP.To prove Π ∈NPC, prove (1) Π ∈NP and (2) ∃Π′ ∈ NPC s.t.Π′ ≤p Π

I NP-hard: A problem X is NP-hard, if there is anNP-complete problem Y, such that Y is reducible to X inpolynomial time. (Note X does not need to be in NP)

Possible relations among P, NP, NP-complete, NP-hard:P⊂NP, NP∩ NP-hard = NP-complete, P∪ NP-complete= /0

168

Satisfiability (SAT):

INSTANCE: A boolean formula φ in CNF with variablesx1, . . . ,xn and clauses c1, . . . ,cm

QUESTION: Is φ satisfiable? (Is there a truth assignment A tox1, . . . ,xn such that φ is true?)

LSAT = < φ > |∃A that satisfies φExample of an instance for SAT:I Variables: x1,x2,x3,x4

I Literals: Any variables and their negations, such as x1, x3

I Clauses: c1 = x1∨x2∨x3, c2 = x1∨x2, c3 = x1∨x2∨x3∨x4

I Function/formula: φ = c1∧c2∧c3

I The instance φ is T by assignment x1 = T ,x2 = x3 = x4 = F .Note: Many assignments satisfy φ, but we only need one.

169

Cook’s Theorem: SAT∈NPC. (Need to prove (1) SAT∈NP and(2) ∀Π ∈NP, Π≤pSAT.)

First Step: How to prove SAT is in NP?NTM N = ”On input < φ > in CNF

1. Guess a truth assignmnet A O(n)

2. Verify if φ = T under A O(n + m)

3. If T , accept; else reject”SAT is solvable by a NTM in polynomial time, thus in NP.

Second step: How to prove ∀Π ∈NP, Π≤pSAT, or equivalently,for any polynomial-time NTM M, L(M)≤p LSAT ?

Will not discuss this proof. But if interested, go to the final sixpages of this slide set for details.

170

11.5 NP-complete problems(Sipser 7.5, pp.310-322)How to prove Π2 is NP-complete:I Show that Π2 ∈NP.I Choose a known NP-complete Π1.I Construct a reduction f from Π1 to Π2.I Prove that f is a polynomial reduction by showing (1) f can

be computed in polynomial time and (2) ∀I1 for Π1, I1 is ayes-instance for Π1 if and only if f (I1) is a yes-instance forΠ2.

π2

Yes

No

Yes

NoIπ1 fIπ2 Alg

Algπ1

Figure 34: Polynomial reduction Π1 ≤p Π2

171

Seven basic NP-complete problems.I 3-Satisfiability (3SAT): (Reduced from SAT)

INSTANCE: A formula α in CNF with each clause havingthree literals.QUESTION: Is α satisfiable?

I 3-Dimensional Matching (3DM): (Reduced from 3SAT)INSTANCE: M ⊆ X ×Y ×Z , where X ,Y ,Z are disjoint andof the same size.QUESTION: Does M contain a matching, which is M ′ ⊆Mwith |M ′|= |X | such that no two triples in M ′ agree in anycoordinate?

I PARTITION: (Reduced from 3DM)INSTANCE: A finite set A of numbers.QUESTION: Is there A′ ⊆ A such that ∑a∈A′ a = ∑a∈A−A′ a?

172

I Vertex Cover (VC): (Reduced from 3SAT)INSTANCE: A graph G = (V ,E) and 0≤ k ≤ |V |.QUESTION: Is there a vertex cover of size ≤ k , where avertex cover is V ′ ⊆ V such that ∀(u,v) ∈ E , either u ∈ V ′

or v ∈ V ′?I Hamiltonian Circuit (HC): (Reduced from VC)

INSTANCE: A graph G = (V ,E).QUESTION: Does G have a Hamiltonian circuit, i.e., a tourthat passes through each vertex exactly once?

I CLIQUE: (Reduced from VC)INSTANCE: A graph G = (V ,E) and 0≤ k ≤ |V |.QUESTION: Does G contain a clique (complete subgraph)of size ≥ k?

173

PARTITION

SAT

3SAT CLIQUE

VC 3DM

HC

Figure 35: Seven basic NP-complete problems

174

Test of your understanding of the complexity classesI If Π1 ≤p Π2 and Π1 ∈NP, is Π2 ∈NP? Maybe.I If Π1 ≤p Π2 and Π1,Π2 ∈NPC, is Π2 ≤p Π1? Yes.I If Π1 ≤p Π2 and Π1 6∈NP, is Π1 ∈P? No.

175

Example to prove reduction: KNAPSACK Problem

INSTANCE: U = u1, . . . ,un, W (max weight knapsack holds),functions w : U→ R+ and v : U→ R+, bound B ≥ 0.QUESTION: Is there U ′ ⊆ U s.t. ∑ui∈U ′w(ui)≤W , and∑ui∈U ′ v(ui)≥ B?

Show PARTITION≤p KNAPSACK.For any instance a1, . . . ,an in PARTITION, define thefollowing instance for KNAPSACK:I U = u1, . . . ,unI w(ui) = ai and v(ui) = ai , for all iI W = B = 1

2 ∑ni=1 ai

Show that (1) the above construction can be done in polynomialtime and (2) there is a partition A′ ⊆ A iff there is a subset U ′

s.t. ∑ui∈U ′w(ui)≤ 12 ∑

ni=1 ai and ∑ui∈U ′ v(ui)≥ 1

2 ∑ni=1 ai .

176

Example to prove reduction: Hitting Set (HS) problem

INSTANCE: Set S, collection C of subsets of S, K ≥ 0QUESTION: Does S contain a HS S′ of size ≤ K ? (S′ is a HS ifS′ contains at least one element from each subset in C.)

VC ≤p HS.Instance for VC: G = (V ,E) and B ≥ 0Instance for HS: S = V , C = u,v|∀e = (u,v) ∈ E, K = BGoal: G has a VC of size ≤ B iff S has a HS of size ≤ K .

HS S’=1, 3, 5

Vertex Cover Hitting Set

1 2 3

4 5 6 7

B=3 and |E|=8

S=1,2,3,4,5,6,7

VC=1,3,5

C=1,2, 1,4, 2,3, 2,5, 3,5, 3,7, 3,6, 5,6

K=3

Figure 36: An example of reduction from VC to HS

177

Proof techniques, languages, complexity classesI Contradiction, construction, reduction, polynomial

reduction, and diagonalizationI RL, CFL, TDL, TRL, non-TR (Chomsky hierarchy)I DFA/NFA, PDA, TM(decides), TM(accepts), NTM(guesses)I Closure properties for the above sets of languagesI Pumping Lemmas for RL and CFLI P, NP, NPC, NP-Hard (relation based on P6=NP, P=NP)I TD, non-TD, TR, non-TRI Prove a language is TD, TR, non-TD, or non-TRI Prove a DEC is in NP.I A good understanding of AD, ATM , HALTTM , SAT, 3SAT,

VC, PARTITION, HC, and those languages and DECsdiscussed in examples.

178

Review of Computability TheoryI TDL: How to prove a language is TDLI The closure properties of TDLs: union, intersection,

concatenation, star, complementI A TM that decides (accept, reject)I TRL: How to prove a language is TRLI The closure properties of TRLs: union, intersection,

concatenation, star, homomorphismI A TM that accepts/recognizesI A language and its complement: Three scenarios (both

TD, one TR but non-TD other non-TR, both non-TR)I Important languages: AD, ATM , HALTTM , etc.I Reduction A≤ B is to show any TM that decides B can be

used to define a TM that decides A. (A is no harder than Bor B is at least as hard as A.)

179

Review of Complexity TheoryI Three classes: P, NP, NPC (Also NP-hard)I Polynomial reduction A≤p B is to show any algorithm that

solves B can be used to define an algorithm that solves A.(A is no harder than B or B is at least as hard as A.)

I Important NP-complete problems: SAT, 3SAT, VC, HC,PARTITION, 3DM, CLIQUE, COLOR, KNAPSACK, HS

180

Some sample problemsI The universal language, ATM , is a proper (non-equal)

subset of the halting language, HALTTM .I The Post Correspondence Problem is decidable for the

unary alphabet.I If L is TR but non-TD, then L is non-TR.

TDTR

non−TR

L

Figure 37: Venn diagram for TD, TR, and non-TR

I If A is non-TD and A≤ C and D ≤ C, then D must benon-TR.

181

I A problem that can be solved by a deterministic TM in 2n

steps must be intractable.I If Π1 is not in NP, then Π1 may still be in PI If A≤p B, then it is possible that A ∈ P but B 6∈ NP.I In order to prove a problem X is NP-hard, one needs to

develop a polynomial reduction from X to a knownNP-hard problem.

I Prove that L = < M > | M accepts 01101 is non-TDI Prove that 3DM is in NPI Prove there is a polynomial reduction from SAT to 3SAT

182

A proof that 3SAT is NP-complete:First, 3SAT is obvious in NP.Next, we show that SAT≤p3SAT.Given any instance of SAT, f (x1, . . . ,xn) = c1∧·· ·∧cm, where ciis a disjunction of literals. To construct an instance for 3SAT, weneed to convert any ci to an equivalent c′i , a conjunction ofclauses with exactly 3 literals.Case 1. If ci = z1 (one literal), define y1

i and y2i . Let c′i =

(z1∨y1i ∨y2

i )∧(z1∨y1i ∨¬y2

i )∧(z1∨¬y1i ∨y2

i )∧(z1∨¬y1i ∨¬y2

i ).Case 2. If ci = z1∨z2 (two literals), define y1

i . Letc′i = (z1∨z2∨y1

i )∧ (z1∨z2∨¬y1i ).

Case 3. If ci = z1∨z2∨z3 (three literals), let c′i = ci .

183

Case 4. If ci = z1∨z2∨·· ·∨zk (k > 3), define y1i ,y

2i , . . . ,y

k−3i .

Let c′i = (z1∨z2∨y1i )∧ (¬y1

i ∨z3∨y2i )∧ (¬y2

i ∨z4∨y3i )∧·· ·∧

(¬yk−3i ∨zk−1∨zk ).

If ci is satisfiable, then there is a literal zl = T in ci . If l = 1,2, lety1

i , . . . ,yk−3i = F . If l = k −1,k , let y1

i , . . . ,yk−3i = T . If

3≤ l ≤ k−2, let y1i , . . . ,y

l−2i = T and y l−1

i , . . . ,yk−3i = F . So c′i is

satisfiable.If c′i is satisfiable, assume zl = F for all l = 1, . . . ,k . Theny1

i , . . . ,yk−3i = T . So the last clause (¬yk−3

i ∨zk−1∨zk ) = F .Therefore, c′i is not satisfiable. Contradiction.The instance of 3SAT is therefore f ′(x1, . . . ,xn, . . .) = c′1∧·· ·∧c′m,and f is satisfiable if and only if f ′ is satisfiable.

184

Cook’s Theorem: SAT is NP-complete.Proof. SAT is clearly in NP since a NTM exists that guesses atruth assignment and verifies its correctness in polynomial time.Now we wish to prove ∀Π ∈NP, Π≤pSAT, or equivalently, forany polynomial-time NTM M, L(M)≤p LSAT .For any NTM M, assume Q = q0,q1(accept),q2(reject), . . . ,qrand Γ = s0,s1,s2, . . . ,sv. Also assume that the time isbounded by p(n), where n is the length of the input.We wish to prove that there is a functionfM : Σ∗→instances of SAT such that ∀x ∈ Σ∗, x ∈ L(M) ifffM(x) is satisfiable. In other words, we wish to use a Booleanexpression fM(x) to describe the computation of M on x .

185

Variables in fM(x):— State: Q[i ,k ]. M is in qk after the i th step of computation (attime i).— Head: H[i , j]. Head points to tape square j at time i .— Symbol: S[i , j , l]. Tape square j contains sl at time i .(Assume the tape is one-way infinite and the leftmost square islabeled with 0.)For example, initially i = 0. Assume the configuration is q0abba.Let s0 = B, s1 = a, and s2 = b. Therefore, we set the followingBoolean variables to be true: Q[0,0], H[0,0], S[0,0,1],S[0,1,2], S[0,2,2], S[0,3,1] and S[0, j ,0] for j = 4,5, . . .. Aconfiguration defines a truth assignment, but not vice versa.

186

Clauses in fM(x):— At any time i , M is in exactly one state.

Q[i ,0]∨·· ·∨Q[i , r ] for 0≤ i ≤ p(n).¬Q[i ,k ]∨¬Q[i ,k ′] for 0≤ i ≤ p(n) and 0≤ k < k ′ ≤ r .

— At any time i , head is scanning exactly one square.H[i ,0]∨·· ·∨H[i ,p(n)] for 0≤ i ≤ p(n).¬H[i , j]∨¬H[i , j ′] for 0≤ i ≤ p(n) and 0≤ j < j ′ ≤ p(n).

— At any time i , each square contains exactly one symbol.S[i , j ,0]∨·· ·∨S[i , j ,v ] for 0≤ i ≤ p(n) and 0≤ j ≤ p(n).¬S[i , j , l]∨¬S[i , j , l ′] for 0≤ i ≤ p(n), 0≤ j ≤ p(n) and

0≤ l < l ′ ≤ v .

187

— At time 0, M is in its initial configuration. Assumex = sl1 · · ·sln .

Q[0,0].H[0,0].S[0,0, l1], . . . ,S[0,n−1, ln].S[0, j ,0] for n ≤ j ≤ p(n).

— By time p(n), M has entered q1 (accept)). (If M halts in lessthan p(n) steps, additional moves can be included in thetransition function.)

Q[p(n),1].

188

— Configuration at time i → configuration at time i + 1. Assumeδ(qk ,sl) = (qk ′ ,sl ′ ,D), where D =−1,1.

If the head does not point to square j , symbol on j is notchanged from time i to time i + 1.

H[i , j]∨¬S[i , j , l]∨S[i + 1, j , l] for 0≤ i ≤ p(n), 0≤ j ≤ p(n),and 0≤ l ≤ v .

If the current state is qk , the head points to square j whichcontains symbol sl , then changes are made accordingly.¬H[i , j]∨¬Q[i ,k ]∨¬S[i , j , l]∨H[i + 1, j + D],¬H[i , j]∨¬Q[i ,k ]∨¬S[i , j , l]∨Q[i + 1,k ′], and¬H[i , j]∨¬Q[i ,k ]∨¬S[i , j , l]∨S[i + 1, j , l ′], for 0≤ i ≤ p(n),

0≤ j ≤ p(n), 0≤ k ≤ r , and 0≤ l ≤ v .

189

Let fM(x) be the conjunction of all the clauses defined above.Then x ∈ L(M) iff there is an accepting computation of M on xiff fM(x) is satisfiable. fM can be computed in polynomial timesince |fM(x)| ≤ (number of clauses) ∗( number ofvariables)= O(p(n)2)∗O(p(n)2) = O(p(n)4).So there is a polynomial reduction from any language in NP toSAT. So SAT is NP-complete.

190