Embed Size (px)
DESCRIPTION
Integrated development environments (IDEs) increase programmer productivity, providing rapid, interactive feedback based on the syntax and semantics of a language. A heavy burden lies on developers of new languages to provide adequate IDE support. Code generation techniques provide a viable, efficient approach to semi-automatically produce IDE plugins. Key components for the realization of plugins are the language's grammar and parser. For embedded languages and language extensions, constituent IDE plugin modules and their grammars can be combined. Unlike conventional parsing algorithms, scannerless generalized-LR parsing supports the full set of context-free grammars, which is closed under composition, and hence can parse language embeddings and extensions composed from separate grammar modules. To apply this algorithm in an interactive environment, this paper introduces a novel error recovery mechanism, which allows it to be used with files with syntax errors -- common in interactive editing. Error recovery is vital for providing rapid feedback in case of syntax errors, as most IDE services depend on the parser -- from syntax highlighting to semantic analysis and cross-referencing. We base our approach on the principles of island grammars, and derive permissive grammars with error recovery productions from normal SDF grammars. To cope with the added complexity of these grammars, we adapt the parser to support backtracking. We evaluate the recovery quality and performance of our approach using a set of composed languages, based on Java and Stratego.
Citation preview
Providing Rapid Feedback inGenerated Modular Language
Environments
Lennart Kats (me)Maartje de JongeEmma Nilsson-NymanEelco Visser
Software Engineering Research Group
OOPSLA 2009 October 29, 2009
2
Domain-Specific Languages
• Encapsulate domain knowledge
• Eliminate boilerplate code
• Domain-specific analysis / verification / optimization
3
UIData
models
Integration and Composition of DSLs
HQL
WorkflowAccesscontrol
Datavalidation
Actions
(www.webdsl.org)
4Markus Voelter [Practical Product Lines 2009]
DSLs without IDE supportare a nonstarter!
5
Spoofax/IMP:Efficient Editor Development
• Existing platform: Eclipse with IMP
• DSLs to define editor components
• Grammar and a parser generator: SDF, (J)SGLR
6
Code Folding
Semantic errors
Outline
References
HQL Brace matching
7
FOLDING/OUTLINE
entity User { ...}
entity BlogPost { poster : User body : String}
REFERENCE
8
entity User { name : String password : String homepage}
entity BlogPost { poster : User body : String}
What aboutincorrect and incomplete programs?
NORMAL PARSING
}
9
entity User { name : String password : String homepage}
entity BlogPost { poster : User body : String}
What aboutincorrect and incomplete programs?
NORMAL PARSING ERROR
NEVER PARSED
Parse failure:No abstract syntax tree
10
Mini-Demo
Error Recovery ina data modeling language
11
Parsing with SDF and SGLR
• Language composition withoutshift/reduce conflicts
• Ambiguities can be inspected
• Declarative disambiguation
12
token token token token token
token token token token token
token x
x
x
token x
Recovery:Backtracking,
skip/insert tokens, etc.
Normal LR Parser
(S)GLR Parser
Recovery: ?
13
Do we really need to dive into thisintimidating algorithm?
14
Island Grammars
IDENTIFICATION DIVISION.PROGRAM-ID. EXAMPLE.PROCEDURE DIVISION. CALL X. YADA. YADA YADA. CALL Y.
CALL X.
CALL Y.
WATER
WATER
• Parse only interesting bits (Islands)• Skip the remaining characters (Water)
[Van Deursen & Kuipers, 1999]
Grammar-based“error recovery”!
15
Island Grammars
~[\ \t\n]+ → WATER {avoid}
[Van Deursen & Kuipers, 1999]
16
Running Example: Java
17
1. Take the entire Java grammar2. Add water3. Mix it together
Error Recovery Recipe
18
Mixing Java with Water
public class TemperatureMonitor { private Fridge fridge; public void trigger(Temperature temp) { if (temp.greaterThan(MAX)) // missing { fridge.startCooling(); } return; }
public TemperatureMonitor(Fridge fridge) { this.fridge = fridge; }}
ERROR
19
Mixing Java with Water
public class TemperatureMonitor { private Fridge fridge; public void trigger(Temperature temp) { if (temp.greaterThan(MAX)) // missing { fridge.startCooling(); } return; }
public TemperatureMonitor(Fridge fridge) { this.fridge = fridge; }}
NORMAL PARSING
ERROR
20
Mixing Java with Water
public class TemperatureMonitor { private Fridge fridge; public void trigger(Temperature temp) { if (temp.greaterThan(MAX)) // missing { fridge.startCooling(); } return; }
public TemperatureMonitor(Fridge fridge) { this.fridge = fridge; }}
} NORMAL PARSING ERROR
21
Mixing Java with Water
public class TemperatureMonitor { private Fridge fridge; public void trigger(Temperature temp) { if (temp.greaterThan(MAX)) // missing { fridge.startCooling(); } return; }
public TemperatureMonitor(Fridge fridge) { this.fridge = fridge; }}
} NORMAL PARSING ERROR
UNEXPECTED KEYWORD
22
Mixing Java with Water
public class TemperatureMonitor { private Fridge fridge; public void trigger(Temperature temp) { if (temp.greaterThan(MAX)) // missing { fridge.startCooling(); } return; }
public TemperatureMonitor(Fridge fridge) { this.fridge = fridge; }} New production:
WATER → MethodDec {cons(“WATER”)}New production: WATER → MethodDec {cons(“WATER”),recover}
23
New production: WATER → MethodDec {cons(“WATER”),recover}New production: WATER → Stm {cons(“WATER”),recover}
Mixing Java with Water
public class TemperatureMonitor { private Fridge fridge; public void trigger(Temperature temp) { if (temp.greaterThan(MAX)) // missing { fridge.startCooling(); } return; }
public TemperatureMonitor(Fridge fridge) { this.fridge = fridge; }}
WATER
24
Mixing Java with Water
public class TemperatureMonitor { private Fridge fridge; public void trigger(Temperature temp) { if (temp.greaterThan(MAX)) // missing { fridge.startCooling(); } return; }
public TemperatureMonitor(Fridge fridge) { this.fridge = fridge; }} New production:
WATER → Stm {cons(“WATER”),recover}
WATER
25
Mixing Java with Water
public class TemperatureMonitor { private Fridge fridge; public void trigger(Temperature temp) { if (temp.greaterThan(MAX)) // missing { fridge.startCooling(); } return; }
public TemperatureMonitor(Fridge fridge) { this.fridge = fridge; }}
WATER LAYOUT
PROBLEMATIC TOKEN
26
Mixing Java with Water
public class TemperatureMonitor { private Fridge fridge; public void trigger(Temperature temp) { if (temp.greaterThan(MAX)) // missing { fridge.startCooling(); } return; }
public TemperatureMonitor(Fridge fridge) { this.fridge = fridge; }} New production:
WATER → LAYOUT {cons(“WATER”),recover}
WATER LAYOUT
PROBLEMATIC TOKEN
27
Mixing Java with Water
public class TemperatureMonitor { private Fridge fridge; public void trigger(Temperature temp) { if (temp.greaterThan(MAX)) // missing { fridge.startCooling(); } return; }
public TemperatureMonitor(Fridge fridge) { this.fridge = fridge; }}
WATER
New production: WATER → LAYOUT {cons(“WATER”),recover}
LAYOUT
28
Reflection:Water-based Recovery Rules
•• Works with any existing grammar
• Can only remove fragments
29
Danger of Floods
public class Fridge { public void startCooling() { cooling.commence(); // missing }
public void onResetButtonPress() { log.message(“Reset button pressed”); power.reboot(); }}
30
Danger of Floods
public class Fridge { public void startCooling() { cooling.commence(); // missing }
public void onResetButtonPress() { log.message(“Reset button pressed”); power.reboot(); }}
WATER
So why not parse methodswith missing closing brackets?
31
“if” “(” Expr Stm → Stm {cons(“If”), recover}
Productions for Missing Literals
“if” Expr “)” Stm → Stm {cons(“If”), recover}“if” Expr Stm → Stm {cons(“If”), recover}“while” “(“ Expr Stm → Stm {cons(“While”), ...}...
and this, and this, and this:
why not add rules like this:
not going to scale.
32
IF BOPEN … BCLOSE … → …[\i][\f] → IF[\(] → BOPEN[\)] → BCLOSE
Productions for Missing Literals
→ BCLOSE {recover} → “}” {recover}
“)”
so, we can write (using the literal instead of BCLOSE):
what it means internally:
“if” “(” Expr “)” Stm → Stm {cons(“If”)}
33
Applying Insertion Rules
public class Fridge { public void startCooling() { cooling.commence(); // missing }
public void onResetButtonPress() { log.message(“Reset button pressed”); power.reboot(); }}
New production: → “}” {recover}
34
Applying Insertion Rules
public class Fridge { public void startCooling() { cooling.commence(); // missing }
public void onResetButtonPress() { log.message(“Reset button pressed”); power.reboot(); }}
INSERT }
New production: → “}” {recover}
35
So who's gonnawrite all thoserecovery rules?
•• Water•• Closing brackets (“}”)•• Opening brackets (“{”)•• Separators (“,”)•• Comments•• String literals
Recovery Rules
We derivethem from the
grammar!
36
Customization of Recovery Rules
→ “class” {reject}
“[|” → “|[“ {recover}
“|]” → “]|” {recover}
37
Putting Things Together
y = f ( x + 1 ;y = f ( x + 1 ) ;y = f ( x ) + 1 ;y = f + 1 ;y = f ( 1 ) ;y = ( x ) + 1 ;y = ( x + 1 ) ;y = x + 1 ;y = f ;y = ( x ) ;y = f ( ) ;
• Water• Insertion: “)”
y = x ; f ( x + 1 ) ; f ( 1 ) ;y ( x + 1 ) ;y ( x ) ;y () ;y ( 1 ) ;y = 1 ; ;
38
Putting Things Together
For recovery, parallel parsing does not scale...
39
Putting Things Together
Why not do backtracking for recovery?
40
Recovery Using Backtracking
public class TemperatureMonitor { private Fridge fridge; public void trigger(Temperature temp) { if (temp.greaterThan(MAX)) // missing { fridge.startCooling(); } return; }
public TemperatureMonitor(Fridge fridge) { this.fridge = fridge; }}
ERROR
41
Recovery Using Backtracking
public class TemperatureMonitor { private Fridge fridge; public void trigger(Temperature temp) { if (temp.greaterThan(MAX)) // missing { fridge.startCooling(); } return; }
public TemperatureMonitor(Fridge fridge) { this.fridge = fridge; }}
1. NORMAL PARSING
ERROR
42
Recovery Using Backtracking
public class TemperatureMonitor { private Fridge fridge; public void trigger(Temperature temp) { if (temp.greaterThan(MAX)) // missing { fridge.startCooling(); } return; }
public TemperatureMonitor(Fridge fridge) { this.fridge = fridge; }}
1. NORMAL PARSING ERROR
POINT OF DETECTION
43
Recovery Using Backtracking
public class TemperatureMonitor { private Fridge fridge; public void trigger(Temperature temp) { if (temp.greaterThan(MAX)) // missing { fridge.startCooling(); } return; }
public TemperatureMonitor(Fridge fridge) { this.fridge = fridge; }}
2. BACKTRACKING 1. NORMAL PARSING
44
Recovery Using Backtracking
public class TemperatureMonitor { private Fridge fridge; public void trigger(Temperature temp) { if (temp.greaterThan(MAX)) // missing { fridge.startCooling(); } return; }
public TemperatureMonitor(Fridge fridge) { this.fridge = fridge; }}
1 RULE1 RULE
2 RULES
1 RULE2 RULES3 RULES
1 RULE2 RULES3 RULES4 RULES
{
45
Evaluation
Implementation:• JSGLR• Spoofax/IMP (WebDSL, Stratego, PIL, ...)
Test grammars:• Java• Java-SQL• Stratego• Stratego-Java
46
Evaluation
Measurements:
• qualitative [Penello & DeRemer '78]:excellent / good / poor
• tree alignment distance [Jian et al '94]
• textual 'diffs'
47
Recovery Quality (Stratego-Java)
48
Continued Work
“Natural and Flexible Error Recovery for Generated Parsers” [SLE 2009]
• indentation-based region selection• fall-back mechanism• bridge parsing [Nilsson-Nyman et al, 2008]
(www.strategoxt.org/Stratego/PermissiveGrammars)
49
Continued Work
“Natural and Flexible Error Recovery for Generated Parsers” [SLE 2009]
• comparison with JDT• performance, quality• synergy between recovery techniques
(www.strategoxt.org/Stratego/PermissiveGrammars)
www.strategoxt.org/Stratego/PermissiveGrammarswww.strategoxt.org/Stratego/SpoofaxIMPwww.lennartkats.net
Software Engineering Research Group
Conclusion
SGLR/SDF• modular specifications• composable languages
Recovery production rules• transparent• flexible• language-independent
