Upload
aaron-patterson
View
12.780
Download
3
Tags:
Embed Size (px)
DESCRIPTION
RubyConf Argentina 2011
Citation preview
Who makes the best Asado?
What country has the best footballers?
Argentina!
river
Nacional B???
river
NOOOOOOOO!!!!
Aaron Patterson
@tenderlove
Ruby core team
Rails core team
! WARNING !
ZOMG!
We are in Argentina!
YoJosé
WWFMD?
me gusta
¿Por qué Maria?
THANKS!!
ResourceManagement
Path Management
PatternMatching
Move between theory and practice
worldhello greeting
digraph nfa { rankdir=LR;
world [shape = doublecircle]; hello [shape = circle]; hello -> world [label="greeting"];}
graph.dot
digraph nfa { rankdir=LR;
world [shape = doublecircle]; hello [shape = circle]; goodbye [shape = circle];
hello -> world [label="greeting"]; goodbye -> world}
graph.dot
world
hellogreeting
goodbye
2 3.
4
/
/articles(.:format)
5
6 /articles/new(.:format)
0 1/ articles (?-mix:[^./?]+)
new
Journey
JourneyYes, it's named after the '70s rock sensation
What is a router?
Provides1: URL generation2: Path Recognition3: Route parsing
Why a new router?
Maintenance
0
750
1500
2250
3000
Journey Journey-2 Rack-Mount
LOC
Known algorithms
References
•Compilers: Principles, Techniques, & Tools (Aho, Lam, Sethi, Ullman)
• Intro to Formal Languages and Automata (Linz)
Predictability
CPU Time
Memory Usage
Current Performance
url generation path recognition route parsing
rack-mount Journey
me gusta
Patterns
Why patterns matter
Regular Expressions/om(g)!/
Rails Routes
resource :articles
/articles(.:format) /articles/new(.:format) /articles/:id/edit(.:format) /articles/:id(.:format)
Parens don't capture
:whatever => /([^./?]+)/
/articles(.:format)
/\/articles(?:\.([^.\/?]+))?$/
/\/articles(?:.([^.\/?]+))?$/
/\/articles\/new(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/
/\/articles(?:.([^.\/?]+))?$/
/\/articles\/new(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/
GET /articles/new
/\/articles(?:.([^.\/?]+))?$/
/\/articles\/new(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/
GET /articles/new
/\/articles(?:.([^.\/?]+))?$/
/\/articles\/new(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/
GET /articles/new
200 OK!
/\/articles(?:.([^.\/?]+))?$/
/\/articles\/new(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/
/\/articles(?:.([^.\/?]+))?$/
/\/articles\/new(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/
GET /foos/new
/\/articles(?:.([^.\/?]+))?$/
/\/articles\/new(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/
GET /foos/new
/\/articles(?:.([^.\/?]+))?$/
/\/articles\/new(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/
GET /foos/new
/\/articles(?:.([^.\/?]+))?$/
/\/articles\/new(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/
GET /foos/new
/\/articles(?:.([^.\/?]+))?$/
/\/articles\/new(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/
GET /foos/new
/\/articles(?:.([^.\/?]+))?$/
/\/articles\/new(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)\/edit(?:.([^.\/?]+))?$/
/\/articles\/([^.\/?]+)(?:.([^.\/?]+))?$/
GET /foos/new
404 Not Found
How long does this take?
r = routes.length
x = regexp compare
O(r ⨉ x)
Can we do better?
Finite State Machines
Parse Trees
Automata
Parse Trees
Parsing regexp
(a|b)*abb
○○ b
○ b
a *
a b
()
|
Describe node types
Parsing rails routes
Journey::Parserparser = Journey::Parser.newast = parser.parse '/articles(.:format)'
puts ast.to_dot
○
○ ()
/ articles ○
. :format
To String!
parser = Journey::Parser.newast = parser.parse '/articles(.:format)'puts ast.to_sputs ast.to_regexp
OR ASTparser = Journey::Parser.new
trees = [ '/articles(.:format)', '/articles/new(.:format)',].map { |s| parser.parse s }
ast = Journey::Nodes::Or.new trees
puts ast.to_dot
|
○ ○
○ ()
/ articles ○
. :format
○ ()
○ new
○ /
/ articles
○
. :format
SECRET FEATUREparser = Journey::Parser.newast = parser.parse '/articles|books(.:format)'
SECRET FEATUREparser = Journey::Parser.newast = parser.parse '/articles|books(.:format)'
SUPER SECRET!
Automata
(a|b)*abb
3
0b2a
b
1aa
b
b
a
Double circle is acceptance state
baabb
3
0b2a
b
1aa
b
b
a
3
0b2a
b
1aa
b
b
a
b
3
0b2a
b
1aa
b
b
a
ba
3
0b2a
b
1aa
b
b
a
baa
3
0b2a
b
1aa
b
b
a
baabb
aab
3
0b2a
b
1aa
b
b
a
Deterministic Finite Automaton
Only one path
Storageclass DFA attr_reader :table
def initialize @table = Hash.new { |h, from| h[from] = {} } @table[0]['a'] = 1 @table[0]['b'] = 0 @table[1]['a'] = 1 @table[1]['b'] = 2 @table[2]['a'] = 1 @table[2]['b'] = 3 @table[3]['b'] = 0 @table[3]['a'] = 2 endend
FSM Simulation
class Simulator def initialize(dfa) @dfa = dfa end
def simulate(symbols) state = 0 until symbols.empty? state = @dfa.move(state, symbols.shift) end state endend
class DFA ...
def move(from, symbol) @table[from][symbol] endend
move function
irb> sim = Simulator.new(DFA.new)=> #<Simulator:0x007f95929a82e0 ...>irb> sim.simulate %w{ b a a b b }=> 3irb> sim.simulate %w{ a a b }=> 2
Time: O(n)n = string.length
me gusta
Space: S + Tstates.length + transitions.length
Nondeterministic Finite Automaton
Has nil edges
Can't tell direction
Simulation of NFA
602ε
4ε
3a
5b
εε
a|b
602ε
4ε
3a
5b
εε
nil-closure
602ε
4ε
3a ε
5b ε
a
Storageclass NFA def initialize @table = Hash.new { |h, from| h[from] = Hash.new { |i,sym| i[sym] = [] } }
@table[0][nil] << 2 @table[0][nil] << 4 @table[2]['a'] << 3 @table[4]['b'] << 5 @table[3][nil] << 6 @table[5][nil] << 6 end
def nil_closure(states) states.map { |s| @table[s][nil] }.flatten endend
FSM Simulationclass Simulator def initialize(nfa) @nfa = nfa end
def simulate(symbols) states = @nfa.nil_closure([0])
until symbols.empty? next_s = @nfa.move(states, symbols.shift) states = @nfa.nil_closure(next_s) end
states endend
Move function
class NFA ...
def move(states, symbol) states.map { |s| @table[s][symbol] }.flatten endend
irb> sim = Simulator.new(NFA.new)=> #<Simulator:0x007faa188a5f88 ...>irb> sim.simulate %w{ a }=> [6]irb> sim.simulate %w{ b }=> [6]irb> sim.simulate %w{ b b }=> []irb> sim.simulate %w{ c }=> []
Time: O(r ⨉ x)r = operators.length, x = string.length
Who cares about NFA?
NFA Construction
○
○ ()
/ articles ○
. :format
/articles(.:format)
cat nodes
10 /
/articles
20 1/ articles
Optional
30ε
1ε
2????? ε
60 1/ 2articlesε
3ε
4. 5:format ε
parser = Journey::Parser.newast = parser.parse '/articles(.:format)'nfa = Journey::NFA::Builder.new ast
tt = nfa.transition_tableputs tt.to_dot
Journey::NFA
ConvertingNFA to DFA
Eliminate nil transitions
Collapse duplicate edges
60 1/ 2articlesε
3ε
4. 5:format ε
/articles(.:format)
2 3. 40 1/ articles :format
/articles(.:format)
CODES!
parser = Journey::Parser.newast = parser.parse '/articles(.:format)'nfa = Journey::NFA::Builder.new ast
tt = nfa.transition_table.generalized_tableputs tt.to_dot
SHORTER CODES!
parser = Journey::Parser.newast = parser.parse '/articles(.:format)'dfa = Journey::GTG::Builder.new ast
tt = dfa.transition_tableputs tt.to_dot
ANY NFAconverts to DFA
O(r ⨉ x) => O(x)r = operations.length, x = string.length
me gusta
Converting Automata to Regexp
2 3. 40 1/ articles :format
/articles(.:format)
2 3. 40 /articles :format
/articles(.:format)
/articles(.:format)
2 4.:format0 /articles
/articles(.:format)
40 /articles(.:format)?
Generalized Transition Graph
/users(.:format) /users/new(.:format) /users/:id/edit(.:format) /users/:id(.:format)
resource :users
2
3.
4/
5
6 8.
7 9/
10
.
11
12 14.
13
15
0 1/ users
(?-mix:[^./?]+)
new(?-mix:[^./?]+)
(?-mix:[^./?]+)
edit
(?-mix:[^./?]+)
(?-mix:[^./?]+)
resource :users
The Plan?
Combine All Routes
Produce DFA
Simulate in O(n) Time
Routing To The Future
JS Simulation
Table => JSON
parser = Journey::Parser.newast = parser.parse '/articles(.:format)'dfa = Journey::GTG::Builder.new ast
tt = dfa.transition_tableputs tt.to_json
Table => SVG
parser = Journey::Parser.newast = parser.parse '/articles(.:format)'dfa = Journey::GTG::Builder.new ast
tt = dfa.transition_tableputs tt.to_svg
JS Tokenizer
function tokenize(input, callback) { while(input.length > 0) { callback(input.match(/^[\/\.\?]|[^\/\.\?]+/)[0]); input = input.replace(/^[\/\.\?]|[^\/\.\?]+/, ''); }}
JS Simulator tokenize(input, function(token) { var new_states = []; for(var key in states) { var state = states[key];
if(string_states[state] && string_states[state][token]) { var new_state = string_states[state][token]; highlight_edge(state, new_state); highlight_state(new_state); new_states.push(new_state); } }
if(new_states.length == 0) { return; } states = new_states; });
d3.js
Rails Console
irb> File.open('out.html', 'wb') { |f|irb* f.write(irb* Wot::Application.routes.router.visualizerirb> )}=> 69074
routes.rb marshalling
Test suggestions
Test coverage
Usage Heat Maps
REGEXP
AST
NFA
DFA
ROFLSCALE!!!
DFA => YACC
DFA => RACC
DFA => Ragel
Open Questions
Is our GTG deterministic?
/users/new|/users/:id
4
5
0 1/ 2users 3/new
(?-mix:[^./?]+)
"new" =~ /new|[^.\/?]+/
Can we make it deterministic?
L1 = {new}L2 = {[^./?]+}
/users/new|/users/:id
4
5
0 1/ 2users 3/L1
L2 - L1
Is it worth our effort?
REGEXP
AST
NFA
DFA
Is it worth our effort?
Thank You!!
<3<3<3<3<3