Upload
georgiana-patterson
View
219
Download
0
Embed Size (px)
Citation preview
1
CIS 461Compiler Design & Construction
Fall 2012 slides derived from Tevfik Bultan, Keith Cooper, and
Linda Torczon
Lecture-Module #12Parsing 4
2
Parsing Techniques
Top-down parsers (LL(1), recursive descent)
• Start at the root of the parse tree from the start symbol and grow toward leaves (similar to a derivation)
• Pick a production and try to match the input
• Bad “pick” may need to backtrack
• Some grammars are backtrack-free (predictive parsing)
Bottom-up parsers (LR(1), operator precedence)
• Start at the leaves and grow toward root
• We can think of the process as reducing the input string to the start symbol
• At each reduction step a particular substring matching the right-side of a production is replaced by the symbol on the left-side of the production
• Bottom-up parsers handle a large class of grammars
3
S
lookahead input string
S
A B
C
?
?
lookahead
start symbol fringe of the parse tree
A ?
lookahead
upper fringe of the parse tree
left-to-rightscan
Bottom-up Parsing
left-most derivation
Top-down Parsing
D
D
C
S
right-most derivation in reverse
4
Handle-pruning, Bottom-up Parsers
The process of discovering a handle & reducing it to the appropriate left-hand side is called handle pruning
Handle pruning forms the basis for a bottom-up parsing method
To construct a rightmost derivation
S 0 1 2 … n-1 n w
Apply the following simple algorithm
for i n to 1 by -1
Find the handle < i i , ki > in i
Replace i with i to generate i-1
5
Example
The expression grammar
Handles for rightmost derivation of input string:
x – 2 * y
Sentential Form Handle Prod’n , Pos’n
S —Expr 1,1Expr – Term 3,3Expr – Term * Factor 5,5Expr – Term * <id,y> 9,5Expr – Factor * <id,y> 7,3Expr – <num,2> * <id,y> 8,3Term – <num,2> * <id,y> 4,1Factor – <num,2> * <id,y> 7,1<id,x> – <num,2> * <id,y> 9,1
1 S Expr2 Expr Expr + Term3 | Expr – Term4 | Term5 Term Term * Factor6 | Term / Factor7 | Factor8 Factor num9 | id
6
Handle-pruning, Bottom-up Parsers
One implementation technique is the shift-reduce parser
push $lookahead = get_ next_token( )repeat until (top of stack == start symbol and lookahead == $) if the top of the stack is a handle then /* reduce to */ pop || symbols off the stack push onto the stack else if (lookahead $) then /* shift */ push lookahead lookahead = get_next_token( )
How do errors show up?
• failure to find a handle
• hitting $ and needing to shift (final else clause)
Either generates an error
7
Example, Corresponding Parse Tree
S
<id,x>
Term
Fact.
Expr –
Expr
<id,y>
<num,2>
Fact.
Fact.Term
Term
*
1. Shift until top-of-stack is the right end of a handle2. Pop the left end of the handle & reduce
5 shifts + 9 reduces + 1 accept
8
Shift-reduce Parsing
Shift reduce parsers are easily built and easily understood
A shift-reduce parser has just four actions
• Shift — next word is shifted onto the stack
• Reduce — right end of handle is at top of stack
Locate left end of handle within the stack
Pop handle off stack & push appropriate lhs
• Accept — stop parsing & report success
• Error — call an error reporting/recovery routine
Accept & Error are simple
Shift is just a push and a call to the scanner
Reduce takes |rhs| pops & 1 push
If handle-finding requires state, put it in the stack
Handle finding is key
• handle is on stack
• finite set of handles
use a DFA !
9
LR Parsers
• LR(k) parsers are table-driven, bottom-up, shift-reduce parsers that use a limited right context (k-token lookahead) for handle
recognition• LR(k): Left-to-right scan of the input, Rightmost derivation in reverse
with k token lookahead
A grammar is LR(k) if, given a rightmost derivation
S 0 1 2 … n-1 n sentence
We can
1. isolate the handle of each right-sentential form i , and
2. determine the production by which to reduce,
by scanning i from left-to-right, going at most k symbols beyond
the right end of the handle of i
10
LR Parsers
A table-driven LR parser looks like
ScannerTable-driven
Parser
ACTION & GOTOTables
ParserGenerator
sourcecode
grammar
IR
Stack
11
LR Shift-Reduce Parsers push($); // $ is the end-of-file symbolpush(s0); // s0 is the start state of the DFA that recognizes handleslookahead = get_next_token();repeat forever s = top_of_stack(); if ( ACTION[s,lookahead] == reduce ) then pop 2*|| symbols; s = top_of_stack(); push(); push(GOTO[s,]); else if ( ACTION[s,lookahead] == shift si ) then push(lookahead); push(si); lookahead = get_next_token(); else if ( ACTION[s,lookahead] == accept and lookahead == $ ) then return success; else error();
The skeleton parser
•uses ACTION & GOTO
• does |words| shifts
• does |derivation| reductions • does 1 accept
12
To make a parser for L(G), we need a set of tables
The grammar
The tables
LR Parsers (parse tables)
1 S Z 2 Z Z z3 | z
ACTION State $ z0 — shift 21 accept shift 32 reduce 3 reduce 33 reduce 2 reduce 2
GOTO State Z 0 1 123
13
Example ParsesThe string “z”
The string “zz”
Stack Input Action$ s0 z $ shift 2$ s0 z s2 $ reduce 3$ s0 Z s1 $ accept
Stack Input Action$ s0 z z $ shift 2$ s0 z s2 z $ reduce 3 $ s0 Z s1 z $ shift 3$ s0 Z s1 z s3 $ reduce 2$ s0 Zs1 $ accept
14
LR Parsers
How does this LR stuff work?
• Unambiguous grammar unique rightmost derivation
• Keep upper fringe on a stack– All active handles include TOS– Shift inputs until TOS is right end of a handle
• Language of handles is regular– Build a handle-recognizing DFA
– ACTION & GOTO tables encode the DFA
• To match subterms, recurse and leave DFA’s state on stack
• Final states of the DFA correspond to reduce actions– New state is GOTO[lhs , state at TOS]
– For Z, this takes the DFA to S1
S0
S3
S2
S1
z
zZ
Control DFA for the simple example
Reduce action
Reduce action
15
Building LR Parsers
How do we generate the ACTION and GOTO tables?• Use the grammar to build a model of the handle recognizing DFA
• Use the DFA model to build ACTION & GOTO tables• If construction succeeds, the grammar is LR
How do we build the handle-recognizing DFA ?
• Encode the set of productions that can be used as handles in the DFA state: Use LR(k) items
• Use two functions goto( s, ) and closure( s )
– goto() is analogous to move() in the DFA to NFA conversion
– closure() is analogous to -closure• Build up the states and transition functions of the DFA
• Use this information to fill in the ACTION and GOTO tables
16
LR(k) items
An LR(k) item is a pair [A , B], where
A is a production with a • at some position in the rhs
B is a lookahead string of length ≤ k (terminal symbols or $)
Examples: [• , a], [• , a], [• , a], & [• , a]
The • in an item indicates the position of the top of the stack
• LR(0) items [ • ] (no lookahead symbol)
• LR(1) items [ • , a ] (one token lookahead)
• LR(2) items [ • , a b ] (two token lookahead) ...
17
LR(k) items
The • in an item indicates the position of the top of the stack
[• , a] means that the input seen so far is consistent with the use of immediately after the symbol on top of the stack
[• , a] means that the input seen so far is consistent with the use of at this point in the parse, and that the parser has already recognized .
[• , a] means that the parser has seen , and that a lookahead a is consistent with reducing to (for LR(k) parsers a is a string of terminal symbols of length k)
The table construction algorithm uses items to represent valid configurations of an LR(1) parser
18
LR(1) Items
The production •, with lookahead a, generates 4 items
[• , a], [• , a], [• , a], & [• , a]
The set of LR(1) items for a grammar is finite
What’s the point of all these lookahead symbols?
• Carry them along to choose correct reduction
• Lookaheads are bookkeeping, unless item has • at right end
– Has no direct use in [• , a]
– In [• , a], a lookahead of a implies a reduction by
– For { [• , a],[• , b] }
lookahead = a reduce to ;
lookahead FIRST() shift Limited right context is enough to pick the actions
19
Back to Finding Handles
Parser in a state where the stack (the fringe) was
Expr – Term
With lookahead of *
How did it choose to expand Term rather than reduce to Expr?
• Lookahead symbol is the key
• With lookahead of + or –, parser should reduce to Expr
• With lookahead of * or /, parser should shift
• Parser uses lookahead to decide
• All this context from the grammar is encoded in the handle- recognizing mechanism
20
Back to x - 2 * y
1. Shift until TOS is the right end of a handle2. Find the left end of the handle & reduce
shift here
reduce here
21
High-level overview Build the handle-recognizing DFA (aka Canonical Collection of sets of LR(1) items), C =
{ I0 , I1 , ... , In }
a Introduce a new start symbol S’ which has only one production
S’ S
b Initial state, I0 should include
• [S’ •S, $], along with any equivalent items
• Derive equivalent items as closure( I0 )
c Repeatedly compute, for each Ik , and each grammar symbol , goto(Ik , )
• If the set is not already in the collection, add it
• Record all the transitions created by goto( )
This eventually reaches a fixed point
2 Fill in the ACTION and GOTO tables using the DFA
The canonical collection completely encodes the transition diagram for the handle-finding DFA
LR(1) Table Construction
22
Computing Closures
closure(I) adds all the items implied by items already in I
• Any item [ , a] implies [ , x] for each production with on the lhs, and x FIRST(a)
• Since is valid, any way to derive is valid, too
The algorithm
Closure( I ) while ( I is still changing ) for each item [ • , a] I for each production P for each terminal b FIRST(a) if [ • , b] I then add [ • , b] to I
Fixpoint computation
23
Example Grammar
Initial step builds the item [S • A ,$]and takes its closure( )
Closure( [S • A , $] )
So, initial state s0 is { [S • Z ,$], [Z • Z z, $],[Z• z , $], [Z • Z z , z], [Z • z , z] }
1 S Z 2 Z Z z3 | z
Item From[S • Z , $] Original item[Z • Z z , $] 1, a is $[Z • z , $] 1, a is $[Z • Z z , z] 2, a is z $[Z • z , z] 2, a is z $
24
Computing Gotos
goto(I , x) computes the state that the parser would reach if it recognized an x while in state I
• goto( { [ , a] }, ) produces [ , a]
• It also includes closure( [ , a] ) to fill out the state
The algorithm
Goto( I, x ) new = Ø for each [ • x , a] I new = new [ x • , a]
return closure(new)
• Not a fixpoint method• Uses closure
25
Example Grammar
s0 is { [S • Z ,$], [Z • Z z, $],[Z • z , $], [Z • Z z , z], [Z • z , z] }
goto( S0 , z )
• Loop produces
• Closure adds nothing since • is at end of rhs in each item
In the construction, this produces s2
{ [Z z • , {$ , z}]}
New, but obvious, notation for two distinct items
[Zz • , $] and [Zz • , z]
Item From[Z z • , $] Item 3 in s0
[Z z • , z] Item 5 in s0