DistributedMarkov Chains

Ratul SahaNational University of Singapore

Joint work with:Javier Esparza, Sumit K. Jha,Madhavan Mukund, and P. S. Thiagarajan

THE MODEL

.DistributedMarkov Chains (DMC)

→ Network of communicating probabilistictransition systems

→ Synchronize on shared actions

→ Followed by joint probabilistic move

→ Key restriction: no two enabledsynchronizations will involve the same agent

→ Enforced syntactically (will explain soon)

→ Efficient model checking using aninterleaved semantics

..3

.DMC: Synchronization

→ Joint probabilistic move after thesynchronization action

....

Agent 1

.s1 .

s′1

.

s′′1

..

Agent 2

. s2.

s′2

.

s′′2

. a....0.2

....0.8

..

..4

.DMC: Key Restriction

→ Any two simultaneously enabled actionsinvolve disjoint sets of agents

→ Syntactically, local state uniquelydetermines its communicating partners

..5

.DMC: This is allowed

....

Agent 1

.s1 ..

Agent 2

.

s2

.

s′2

.

b

.

a

....

..6

.DMC: This is not allowed!

....

Agent 1

.s1 .

s2

.

s3

.

b

.

a

......

Agent 2

..

Agent 3

..7

.DMC: Events

→ Event: One synchronization is executed at atime, followed by a probabilistic move by theparticipating agents

..s1. s2.

0.3

.

0.2

.

0.5

..

a

.

s′1

.

s′2

.

s′′1

.

s′′2

.

s′′′1

.

s′′′2

............

e = ((s1, s2),a, (s′

1, s′

2)) is an event, pe = 0.3

..8

.DMC: Coin Toss Example

→ Two players. Each toss a fair coin

→ Outcomes are the same: they toss again

→ Outcomes are different: who tosses Heads wins

..H1

.

IN1

..

W1

. T1.

L1

. 0.5. 0.5.a1

.ht

.hh

.

w1

.th

.tt

.

w2

Agent 1

..H2

.

IN2

..

W2

. T2.

L2

. 0.5. 0.5.a2

.th

.hh

.

w2

.ht

.tt

.

w1

Agent 2

..9

.Global Transition System

→ Associate a global transition system basedon event occurrences

→ This is interleaved semantics

..10

.Global Transition System: Coin Toss

..agent 1 tossing T..

(T1, IN2)

.

(IN1, IN2)

...

e1t , 0.5

..11

.Global Transition System: Coin Toss

..agent 1 tossing T. (IN1, T2). agent 2 tossing T.

(T1, IN2)

.

(IN1, IN2)

.....

e1t , 0.5

.e2t , 0.5

..12

.Global Transition System: Coin Toss

..(T1, T2). (IN1, T2)..

(T1, IN2)

.

(IN1, IN2)

. e1t , 0.5.

e1t , 0.5

.e2t , 0.5

.e2t , 0.5

..13

.Global Transition System: Coin Toss

..(T1, T2). (IN1, T2).

(T1, IN2)

.

(IN1, IN2)

.

(IN1,H2)

. e1t , 0.5.

e1t , 0.5

.e2t , 0.5

.e2t , 0.5

.

e2h, 0.5

..14

.Global Transition System: Coin Toss

..(T1, T2). (IN1, T2).

(T1, IN2)

.

(IN1, IN2)

.

(T1,H2)

.

(IN1,H2)

. e1t , 0.5.

e1t , 0.5

.

e1t , 0.5

.e2t , 0.5

.e2t , 0.5

.

e2h, 0.5

.

e2h, 0.5

..15

.Global Transition System: Coin Toss

..(T1, T2). (IN1, T2). (H1, T2).

(T1, IN2)

.

(IN1, IN2)

.

(H1, IN2)

.

(T1,H2)

.

(IN1,H2)

.

(H1,H2)

.

(both agents tossed)

. e1t , 0.5.

e1t , 0.5

.

e1t , 0.5

. e1h, 0.5.

e1h, 0.5

.

e1h, 0.5

.e2t , 0.5

.e2t , 0.5

.e2t , 0.5

.

e2h, 0.5

.

e2h, 0.5

.

e2h, 0.5

..16

.Global Transition System: Coin Toss

..(T1, T2). (IN1, T2). (H1, T2).

(T1, IN2)

.

(IN1, IN2)

.

(H1, IN2)

.

(T1,H2)

.

(IN1,H2)

.

(H1,H2)

.

(L1,W2)

. (W1, L2).

(full global transition system)

.ett

.

ehh

. e1t , 0.5.

e1t , 0.5

.

e1t , 0.5

. e1h, 0.5.

e1h, 0.5

.

e1h, 0.5

. eht.

eth

.

ew1

.

ew2

.e2t , 0.5

.e2t , 0.5

.e2t , 0.5

.

e2h, 0.5

.

e2h, 0.5

.

e2h, 0.5

(unmarked events have probability 1)

..17

.The Trajectory Space

→ We refer to paths in TS as trajectories

→ We wish to reason about the behavior of thesystem using the interleaved semantics

Problem: It is hard to define a probabilitymeasure over the set of maximal trajectories

..18

.The Trajectory Space

Due to mix of concurrency and stochasticity,TS is not a Markov chain in general

... (IN1, T2)..

(T1, IN2)

.

(IN1, IN2)

.

(H1, IN2)

.

(IN1,H2)

.

e1t , 0.5

.

e1h, 0.5

.e2t , 0.5

.

e2h, 0.5

Here, the sum of the probabilities of thetransitions from the state (IN1, IN2) is 2

..19

.The Solution..DMC.

Transition system

.

Markov chain

...

Path space (σ-algebragenerated by basic cylin-ders)

.

Probability measure forthe path space

..

Trajectory space (σ-algebra generated bybasic cylinders of equivclass of trajectories)

.

Probability measure forthe trajectory space

.

Trajectories

.

Equiv classes

.

of trajectories

.

Paths

.........

..20

.Equivalence Classes of Trajectories..DMC.

Transition system

.

Markov chain

...

Path space (σ-algebragenerated by basic cylin-ders)

.

Probability measure forthe path space

..

Trajectory space (σ-algebra generated bybasic cylinders of equivclass of trajectories)

.

Probability measure forthe trajectory space

.

Trajectories

.

Equiv classes

.

of trajectories

.

Paths

.

✓

.

✓

.........

..21

. Independence over Events

... (IN1, T2)..

(T1, IN2)

.

(IN1, IN2)

.

(H1, IN2)

.

(IN1,H2)

.

e1t , 0.5

.

e1h, 0.5

.e2t , 0.5

.

e2h, 0.5

→ e1t I e2h — agent 1 tossing tail and agent 2tossing head are independent

..22

.Equivalence over Event Sequences

... (IN1, T2)..

(T1, IN2)

.

(IN1, IN2)

.

(H1, IN2)

.

(T1,H2)

.

(IN1,H2)

.

e1t , 0.5

.

e1t , 0.5

.

e1h, 0.5

.e2t , 0.5

.

e2h, 0.5

.

e2h, 0.5

→ [e1t e2h] = {e1t e2h, e2he

1t }— equivalence class over

event sequences→ Lifts to equivalence over trajectories

..23

.Markov Chain Semantics..DMC.

Transition system

.

Markov chain

...

Path space (σ-algebragenerated by basic cylin-ders)

.

Probability measure forthe path space

..

Trajectory space (σ-algebra generated bybasic cylinders of equivclass of trajectories)

.

Probability measure forthe trajectory space

.

Trajectories

.

Equiv classes

.

of trajectories

.

Paths

.

✓

.

✓

.

✓

.........

..24

.Markov Chain Semantics

→ A step at a global state is a maximal set ofindependent enabled events

→ The transition relation using steps induces aMarkov chain

..25

.Markov Chain Semantics

... (IN1, T2)..

(T1, IN2)

.

(IN1, IN2)

.

e1t , 0.5

.e2t , 0.5 ⇒

..(T1, T2)...

(IN1, IN2)

.

{e1t , e2t }, 0.25

→ {e1t , e2t } is a maximal step at (IN1, IN2)

→ The probability of a step is the product ofprobabilities associated with the events inthe step

..26

.Markov Chain Semantics

..(T1, T2). (T1,H2). (H1,H2). (H1, T2).

(IN1, IN2)

.

{e1t , e2t }, 0.25

.

0.25

.

0.25

.

0.25

The maximal steps at (IN1, IN2) are{e1h, e

2h}, {e

1h, e

2t }, {e1t , e2h}, {e

1t , e2t }

..27

.Coin Toss: Global Markov Chain

..(T1, T2). (T1,H2). (H1, T2). (H1,H2).

(IN1, IN2)

.

(L1,W2)

.

(W1, L2)

.

0.25

.

0.25

.

0.25

.

0.25

......

(The unmarked transitions have probability 1)

..28

.Coin Toss: Global Markov Chain

What if there were k players?

..(T1, T2, · · · , Tk). (H1, · · · ,Hk−1,Hk).

(H1, T2, · · · , Tk)

. · · ·.

(H1, · · · ,Hk−1, Tk)

.

· · ·

.

(IN1, IN2, · · · , INk)

.

12k

.

12k

.

12k

.

12k

..

k parallel probabilistic moves generate 2k

global transitions..29

.Markov Chain Semantics

The number of transitions out of a globalstate can be (in number of agents)

exponentialin Markov chain

semantics⇒

polynomialin interleavedsemantics

..30

.Markov Chain Semantics..DMC.

Transition system

.

Markov chain

...

Path space (σ-algebragenerated by basic cylin-ders)

.

Probability measure forthe path space

..

Trajectory space (σ-algebra generated bybasic cylinders of equivclass of trajectories)

.

Probability measure forthe trajectory space

.

Trajectories

.

Equiv classes

.

of trajectories

.

Paths

.

✓

.

✓

.

✓

.

✓

.........

..31

.Markov Chain Semantics..DMC.

Transition system

.

Markov chain

...

Path space (σ-algebragenerated by basic cylin-ders)

.

Probability measure forthe path space

..

Trajectory space (σ-algebra generated bybasic cylinders of equivclass of trajectories)

.

Probability measure forthe trajectory space

.

Trajectories

.

Equiv classes

.

of trajectories

.

Paths

.

✓

.

✓

.

✓

.

✓

.

✓

.

✓

.

✓

.........

..32

.Defining the ProbabilityMeasure..DMC.

Transition system

.

Markov chain

...

Path space (σ-algebragenerated by basic cylin-ders)

.

Probability measure forthe path space

..

Trajectory space (σ-algebra generated bybasic cylinders of equivclass of trajectories)

.

Probability measure forthe trajectory space

.

Trajectories

.

Equiv classes

.

of trajectories

.

Paths

.

✓

.

✓

.

✓

.

✓

.

✓

.

1-1 correspondence

.........

..33

.Defining the ProbabilityMeasure..DMC.

Transition system

.

Markov chain

...

Path space (σ-algebragenerated by basic cylin-ders)

.

Probability measure forthe path space

..

Trajectory space (σ-algebra generated bybasic cylinders of equivclass of trajectories)

.

Probability measure forthe trajectory space

.

Trajectories

.

Equiv classes

.

of trajectories

.

Paths

.

✓

.

✓

.

✓

.

✓

.

1-1 correspondence

..........

..34

.Defining the ProbabilityMeasure..DMC.

Transition system

.

Markov chain

...

Path space (σ-algebragenerated by basic cylin-ders)

.

Probability measure forthe path space

..

Trajectory space (σ-algebra generated bybasic cylinders of equivclass of trajectories)

.

Probability measure forthe trajectory space

.

Trajectories

.

Equiv classes

.

of trajectories

.

Paths

.

✓

.

✓

.

✓

.

✓

.

✓

.

1-1 correspondence

..........

..35

THE MODELCHECKING

.PBLTL⊗: The Specification Logic

Local Bounded LTL (BLTLi)→ Time bounded LTL

→ Each agent i has a local set of atomic propositions APi

→ Formula of type i: ap ∈ APi | ¬φ | φ1 ∨ φ2 | φ1Uki φ2

→ φ1Uki φ2 — Until holds within k (local) moves of

agent i

Product Bounded LTL (BLTL⊗)→ Boolean combinations of {BLTLi} formulas

..37

.PBLTL⊗: The Specification Logic

Probabilistic Product Bounded LTL (PBLTL⊗)

→ Pr≥γ(φ), where φ is a BLTL⊗ formula

Example (coin toss):

Pr≥0.99

[(F7(L1) ∧ F7(W2)

)∨(F7(W1) ∧ F7(L2)

)]with probability at least 0.99, the coin toss gameterminates within 7 rounds(the local states serve as the atomic propositions)

..38

.Statistical Model Checking

→ Given a DMC and PBLTL⊗ formula Pr≥γ(φ)

→ Explicit computation is impractical for verylarge systems

→ Instead, estimate through sampling

→ Draw sample trajectories from the interleavedsemantics

→ Use statistical estimation: returns“γ in (predefined) interval (a,b)”with high confidence

..39

.Experimental Results

→ Modeled a number of PRISM benchmarksincluding:

(i) Distributed leader election protocol[Itai and Rodeh]

(ii) A randomized solution to the diningPhilosophers problem [Pnueli and Zuck]

→ Compared simulation time with a statisticalmodel checker — PLASMA

..40

.Distributed Leader Election

→ Verify: with probability 1, a leader is electedeventually

→ Since specification logic is BLTL⊗ and theverification procedure is SMC, we insteadverify:

In a ring of N nodes, with high probability(p), a leader will be elected within B rounds

(Type I and II error = 0.01, indifference region =0.01, for various choices of N and B)

..41

.Distributed Leader Election:Comparison with PLASMAp = 0.99, N up to 1000 (no parallelization in DMC)

..42

.Dining Philosophers Problem→ To start, a philosopher probabilistically

chooses the order in which it will try theforks

→ Forks between philosophers are also agentsin DMC formalism

→ We use deterministic round robin protocolto simulate shared variable

→ The property we verify:With high probability (p), every philosophereats within B rounds

..43

.Dining Philosophers Problem:Comparison with PLASMAp = 0.95, N up to 500 (no parallelization in DMC)

..44

FUTURE WORK

.FutureWork and Challenges

→ Other case studies from PRISM benchmark

→ Finding large systems with deterministicsynchronizations

→ Extend SMC procedure to a parallelimplementation

→ Generating independent samples in parallel

→ Redefining SMC parameters

..46

.FutureWork and Challenges

→ Use in other variations of probabilisticmodels

→ Currently building a probabilistic version ofnegotiation model

→ Reduction rules for new properties

→ Model systems with observable andnon-observable components

→ Model (non-)observable components ascommunicating agents

→ Predict behavior of non-observable agents

..47

.FutureWork and Challenges

→ Other application domains

→ Distributed decision making in robotics

→ Software model checking, Workflow systems

→ Exact probabilistic verification

→ PCTL and other probabilistic temporal logics

→ Devising new model checking algorithms

..48

THANK YOU!