Upload
walter-boone
View
214
Download
0
Embed Size (px)
Citation preview
Design and Implementation of the Recursive Virtual Address Space Model
for Small Scale Multiprocessor Systems
Diploma Thesis
Marcus VölpSupervisor: Volkmar Uhlig
Universität Karlsruhe
September 2002 Marcus Völp Universität Karlsruhe
2
recursive virtual address space model
microkernel
Policy S1 Policy S2
Policy S0
Policy S1a Policy S1bPolicy S2a Policy S2b
September 2002 Marcus Völp Universität Karlsruhe
6
Policy S1 Policy S2
Policy S0
Policy S1a Policy S1bPolicy S2a Policy S2b
recursive virtual address space model
microkernel
Region Mapper
Pager
Pager Pager
Pager
Pager Pager
Pager
September 2002 Marcus Völp Universität Karlsruhe
7
recursive virtual address space model
mapping databasekeep track of mappings
to implement unmap
No long interrupt latencies
No unbounded priority inversion
No helping
0
Hazelnut + Pistachio unmap with interrupts disabled
September 2002 Marcus Völp Universität Karlsruhe
8
recursive virtual address space model
mapping databasekeep track of mappings
to implement unmap
No long interrupt latencies
No unbounded priority inversion
No helping
0
unbounded priority inversion
Hazelnut + Pistachio (if interrupts enabled)
Fiasco (if helping disabled)
September 2002 Marcus Völp Universität Karlsruhe
9
recursive virtual address space model
mapping databasekeep track of mappings
to implement unmap
No long interrupt latencies
No unbounded priority inversion
No helping
0
Helping problematic for SMP- xcpu time donation- migration to helper cpu
September 2002 Marcus Völp Universität Karlsruhe
10
Outline
Motivation Problem analysis
Unbouned priority inversion inherent to
pre-order tree traversal (if not helped out)
Unbounded priority inversion inherent tonon-post-order tree traversal
Mapping database Data Structures Locking Scheme Restart Point Tracking
Performance Conclusion
September 2002 Marcus Völp Universität Karlsruhe
11
Problem Analysis
Pre-Order tree traversal b in b unmap ( b ) remove node c preemption a unmap ( a )
Priority: a > b
Solutions roll forward unmap (b) help b
keep node c
(last thread removes it)
a
D
b
c
September 2002 Marcus Völp Universität Karlsruhe
12
Problem Analysis
Unbound priority inversion is inherent to non post-order tree traversal
General traversal algorithm: i child nodes before parent
k child nodes after parent
Post order traversal: k = 0 for all nodes
D
b
… …1 i 1 k
a
September 2002 Marcus Völp Universität Karlsruhe
13
Conclusion from problem analysis
Post order tree traversal
Find leaf node to start with
Collapse tree from leaf to root of subtree
September 2002 Marcus Völp Universität Karlsruhe
14
Mapping Database
Datastructure fast insertion (map)
find start-leafnode
post order traversal
Synchronization SMP safe
avoid Priority inversion
avoid Starvation
=> Restart Point Tracking
September 2002 Marcus Völp Universität Karlsruhe
15
Data Structure
LL
Number of Pointers: 3 + depthFind first leaf: O (N)
a
b c
e
d
gf
h i k
0
1
2
3
Post order sorted doubly linked list
(page reference info+ overmapping)
September 2002 Marcus Völp Universität Karlsruhe
16
Data Structure
Treea
b c
e
d
gf
h i k
(left child, right sibling)
September 2002 Marcus Völp Universität Karlsruhe
17
Data Structure
LL Tree
Number of Pointers: 5Find first leaf: O (D)
a
b c
e
d
gf
h i k
September 2002 Marcus Völp Universität Karlsruhe
18
Data Structure
LL O1
Number of Pointers: 6Find first leaf: O (1) best
O (D) worst
a
b c
e
d
gf
h i k
September 2002 Marcus Völp Universität Karlsruhe
19
Data Structure
LL O1
Number of Pointers: 6Find first leaf: O (1) best
O (D) worst
a
b c
e
d
gf
h i k
September 2002 Marcus Völp Universität Karlsruhe
20
Data Structure
Representation Number of Pointers
Find first leaf
LL 3 + depth O (N)
LL Tree 5 O (D)
LL O1 6 O (1) best
O (D)
September 2002 Marcus Völp Universität Karlsruhe
21
Synchonization
Datastructure consistency Atomic modifiaction
No Priority Inversion
=> Scheduler Conscious Locks
Scalability Maximal possible parallelism
=> Frame granular locking
September 2002 Marcus Völp Universität Karlsruhe
22
Synchronization
acquire Lock release Lock
unpreemptable
preemption pending? => yield
Roll forward to Preemption point
backoff code
September 2002 Marcus Völp Universität Karlsruhe
23
Synchronization
Frame granular locking (the real world)
a
b c
d e
4MB
fg
i j
h
k l
1MB
…
September 2002 Marcus Völp Universität Karlsruhe
24
Restart Point Tracking
(rw)
d(r)
a
b c1
(rw)
(rw)(rw)
d(r)
e(rw)
a
b c1
(rw)
(rw)
Why Restart Point Tracking ?
Required Properties Low overhead on
map / unmap
Guarentee fwd progress to avoid starvation
September 2002 Marcus Völp Universität Karlsruhe
25
Restart Point Tracking
Why Restart Point Tracking ?
Required Properties Low overhead on
map / unmap
Guarentee fwd progress to avoid starvation
(rw)
d(r)
a
b c1
(rw)
(rw)
a
b c1
(rw)
(rw)
September 2002 Marcus Völp Universität Karlsruhe
26
Restart Point Tracking
Update restart point Find restart point information Update restart point information
On Wakeup Find node to proceed ! Guarantee progress of find !
September 2002 Marcus Völp Universität Karlsruhe
27
Restart Point Tracking
4 2 33 2
Token Based Preempted Thread List
September 2002 Marcus Völp Universität Karlsruhe
28
Token Based Preempted Thread List
Restart Point Tracking
4 2 3
1 2
September 2002 Marcus Völp Universität Karlsruhe
29
Token Based Preempted Thread List
Restart Point Tracking
4 2 3
1 2
September 2002 Marcus Völp Universität Karlsruhe
30
Restart Point Tracking
Rootnode overrun detection
4 2 3
1 2
check overrun flag in mdb node
September 2002 Marcus Völp Universität Karlsruhe
31
Summary Mapping Database
Post order traversal Data Structures
LL LL-Tree LL-O1
Synchronization Scheduler Conscious Locks Roll forward to preemption point
Token based preempted thread list Guarantee progress of preempted operations Root overrun detection
size search time for first leaf
September 2002 Marcus Völp Universität Karlsruhe
32
Performance Evaluation2683
2616
200
2693
264
2860
333
2739
2470
2693
0500
1000
1500
2000
25003000
Ex
ec
uti
on
tim
e
(cy
cle
s)
Pis
tach
io
LL
pu
re M
DB
LL
-Tre
e
pu
re M
DB
LL
-O1
pu
re M
DB
Fia
sco
LL
-op
t
LL
-O1
-op
t
Page fault resolution
Best case performance 5% Quantile
Comparable performance
for map systemcall
(prepare IPC, map page,
resume faulting instruction)
Repre-sentation
# pointer Search time leaf node
Mapping node size
LL 3 + depth O(N) 32 byte
(24 byte)
LL-Tree 5 O(D) 32 byte
LL-O1 6 O(1) best
O(D) worst
40 byte
(32 byte)
on PIII 500 MHz, 256 KB L2 Cache
September 2002 Marcus Völp Universität Karlsruhe
33
Performance Evaluation
Unmap wide mapping tree
Performance of unmap systemcall
Performance unmapping a wide subtree (4KB pages)
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
13000
14000
15000
16000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Width
Exe
cuti
on
tim
e o
f u
nm
ap (
in
cycl
es)
Pistachio LL LL-Tree LL-O1Fiasco LL-opt LL-O1-opt
…
1 2 n
September 2002 Marcus Völp Universität Karlsruhe
34
Overhead to Interrupt Latencies
Map: insert mapping update PTE
Unmap: 3 Preemption Points
1 step:search start point
remove mappingupdate PTE
finish unmap
2709 30
79
2524
6613
2948
2077
6522
2934
0
1000
2000
3000
4000
5000
6000
7000
time
(cyc
les)
1
Worst case overhead added to interrupt handling
Map 4K
Map 4M
Unmap 4K: start->PP(ff l)
Unmap 4K: backoff->PP(tr-up)
Unmap 4K: backoff->end
Unmap 4M: start->PP(ff l)
Unmap 4M: backoff->PP(tr-up)
Unmap 4M: backoff->end
Mapping Database responsible for 13ms interrupt latency(largest distance between preemption points)
September 2002 Marcus Völp Universität Karlsruhe
35
Conclusion
Search time optimization for leaf node does not pay
Good preemptability without helping
No unbounded priority inversion No need for prevention !!!
approx. 12.2% - 26% loss of performance for unmap
Open: explaination of Fiasco performance
bounded tree-size N for real-time unmap
September 2002 Marcus Völp Universität Karlsruhe
37
Page Reference Information / Overmapping
[rw]
[r]
[r] [r] [r]
[rw]
Increase rights omitting prior flush: • same source address space• same virtual source address• same virtual destination address• same physical page
parent link
September 2002 Marcus Völp Universität Karlsruhe
38
Page Reference Information /Overmapping
0
a
cb
d
pa [rw]
pb [rw]
pd [w]
pc [w]
<R>
<W>
0
a
cb
d
pa [rw]
pb [r]
pd [r]
0
a
cb
d
pa [rw]
pb [r]
{R}
{RW}
September 2002 Marcus Völp Universität Karlsruhe
40
Motivation
Problems in Existing Implementations Long Interrupt Latencies
Orangepip, Hazelnut
Unbounded Priority Inversion Calypso, Hazelnut (two phase lock)
Complex Helping or Timeslice Donation Fiasco, L4 Alpha
September 2002 Marcus Völp Universität Karlsruhe
41
Guaranteeing Forward Progress
Why Restart Point Tracking ?a
b c
e
d
gf
h i k
September 2002 Marcus Völp Universität Karlsruhe
42
Guaranteeing Forward Progress
Why Restart Point Tracking ?
a
b c1
(rw)
(rw)(rw)
d(r)
e(rw)
a
b c1
(rw)
(rw)(rw)
d(r)
a
b c1
(rw)
(rw)
September 2002 Marcus Völp Universität Karlsruhe
43
Guaranteeing Forward Progress
Restart Point Tracking
September 2002 Marcus Völp Universität Karlsruhe
44
Guaranteeing Forward Progress
Restart Point Tracking
September 2002 Marcus Völp Universität Karlsruhe
45
Guaranteeing Forward Progress
Restart Point Tracking
September 2002 Marcus Völp Universität Karlsruhe
46
Guaranteeing Forward Progress
4 2 33 2
Restart Point Tracking Token Based Preempted Thread List
September 2002 Marcus Völp Universität Karlsruhe
47
Experimental Results
Performance: Map 4KB to handle Pagefault
Unmap Deep + Wide subtree
Interrupt Latencies Time OPs are rolled forward
Intel Pentium III 500 MHz , 512 KB L2, 2x 16 KB L1, 64 / 32 entry 4KB TLB (data / code), 8 / 2 entry 4MB TLB
September 2002 Marcus Völp Universität Karlsruhe
48
Experimental Results
2683
2616
200
2693
264
2860
333
2739
2470
2693
0500
1000
1500
2000
25003000
Ex
ec
uti
on
tim
e
(cy
cle
s)
Pis
tach
io
LL
pu
re M
DB
LL
-Tre
e
pu
re M
DB
LL
-O1
pu
re M
DB
Fia
sco
LL
-op
t
LL
-O1
-op
t
Page fault resolution
Best case performance 5% Quantile
September 2002 Marcus Völp Universität Karlsruhe
49
Experimental Results
Performance unmapping a deep subtree (4KB pages)
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
13000
14000
15000
16000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Depth
Exe
cuti
on
tim
e o
f u
nm
ap (
in
cycl
es)
Pistachio LL LL-Tree LL-O1
LL-opt LL-O1-opt
Performance unmapping a wide subtree (4KB pages)
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
13000
14000
15000
16000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Width
Exe
cuti
on
tim
e o
f u
nm
ap (
in
cycl
es)
Pistachio LL LL-Tree LL-O1Fiasco LL-opt LL-O1-opt
September 2002 Marcus Völp Universität Karlsruhe
50
Experimental Results
2709 30
79
2524
6613
2948
2077
6522
2934
0
1000
2000
3000
4000
5000
6000
7000
tim
e (c
ycle
s)
1
Worst case overhead added to interrupt handling
Map 4K
Map 4M
Unmap 4K: start->PP(ff l)
Unmap 4K: backoff->PP(tr-up)
Unmap 4K: backoff->end
Unmap 4M: start->PP(ff l)
Unmap 4M: backoff->PP(tr-up)
Unmap 4M: backoff->end
September 2002 Marcus Völp Universität Karlsruhe
51
Experimental Results
Worst case interrupt latency
find first leaf
September 2002 Marcus Völp Universität Karlsruhe
52
Experimental Results
2709 30
79
2524
6613
2948
2077
6522
2934
0
1000
2000
3000
4000
5000
6000
7000
tim
e (c
ycle
s)
1
Worst case overhead added to interrupt handling
Map 4K
Map 4M
Unmap 4K: start->PP(ff l)
Unmap 4K: backoff->PP(tr-up)
Unmap 4K: backoff->end
Unmap 4M: start->PP(ff l)
Unmap 4M: backoff->PP(tr-up)
Unmap 4M: backoff->end
September 2002 Marcus Völp Universität Karlsruhe
53
Interrim Measurements
0
100
200
300
400
500
600
Performance (cycles)
LL-Min LL-Tree-Min
LL-O1-Min
Representation
Interrim Measures
set entry
contents
next map link
insert into ds
alloc mdb node
grant
overmap check
September 2002 Marcus Völp Universität Karlsruhe
54
Interrim Measurements
0
100
200
300
400
500
600
Performance (cycles)
LL -Min LL-Tree-Min
LL-O1-Min
Representation
Interrim Measures unmap
root overrun
traverse up
check end
remove node
flush entry
ref info
flip lock
traverse down
find first leaf
preparation