Higher-Order Ghost State

Ralf Jung, Robbert Krebbers, Lars Birkedal, Derek DreyerICFP 2016 in Nara, JapanMax Planck Institute for Software Systems (MPI-SWS), Aarhus University

1

• This talk is about Iris, a logic wewant to apply to verify the safety ofRust.

• The key technical contribution is toshow how to extend Iris withhigher-order ghost state.

2

• This talk is about Iris, a logic wewant to apply to verify the safety ofRust.• The key technical contribution is toshow how to extend Iris withhigher-order ghost state.

2

• This talk is about Iris, a logic wewant to apply to verify the safety ofRust.• The key technical contribution is toshow how to extend Iris withhigher-order ghost state.

2

The Rust Programming Language

JavaGoHaskell. . .

CC++Assembly. . .

Focus on safety Focus on control

3

The Rust Programming Language

JavaGoHaskell. . .

CC++Assembly. . .

Focus on safety

Focus on control

3

The Rust Programming Language

JavaGoHaskell. . .

CC++Assembly. . .

Focus on safety Focus on control

3

The Rust Programming Language

• Higher-order functions• Polymorphism / Generics• Traits (typeclasses +associated types)

• Concurrency• Control over memoryallocation and data layout• Linear (ownership-based)type system with regions &region inference

Goal of RustBelt project:Prove safety of language and its

standard library.

4

The Rust Programming Language

• Higher-order functions• Polymorphism / Generics• Traits (typeclasses +associated types)

• Concurrency

• Control over memoryallocation and data layout

• Linear (ownership-based)type system with regions &region inference

Goal of RustBelt project:Prove safety of language and its

standard library.

4

The Rust Programming Language

• Higher-order functions• Polymorphism / Generics• Traits (typeclasses +associated types)

• Concurrency

• Control over memoryallocation and data layout• Linear (ownership-based)type system with regions &region inference

Goal of RustBelt project:Prove safety of language and its

standard library.

4

The Rust Programming Language

• Higher-order functions• Polymorphism / Generics• Traits (typeclasses +associated types)• Concurrency• Control over memoryallocation and data layout• Linear (ownership-based)type system with regions &region inference

Goal of RustBelt project:Prove safety of language and its

standard library.

4

The Rust Programming Language

• Higher-order functions• Polymorphism / Generics• Traits (typeclasses +associated types)• Concurrency• Control over memoryallocation and data layout• Linear (ownership-based)type system with regions &region inference

Goal of RustBelt project:Prove safety of language and its

standard library.

4

Picking the right tool

Wanted:

Higher-orderconcurrent

program logic

5

Picking the right tool

Wanted:

Higher-orderconcurrent

separation logic

5

Picking the right tool

Wanted:

Higher-order

concurrentseparation logic

5

Picking the right tool

Wanted:

Higher-orderconcurrent

separation logic

5

Picking the right tool

Wanted:

Higher-orderconcurrent

separation logic

5

Concurrency Logics

6

Concurrency Logics

6

Complex Foundations

In previous work:Let’s try to make it simple(r).

With Iris, we can derive the morecomplex reasoning principles from the

simple foundations.

7

Complex Foundations

In previous work:Let’s try to make it simple(r).

With Iris, we can derive the morecomplex reasoning principles from the

simple foundations.

7

Iris (POPL 2015) is built on twosimple mechanisms:

• Invariants• User-defined ghost state

8

Iris (POPL 2015) is built on twosimple mechanisms:

• Invariants• User-defined ghost state

8

Ghost State

Ghost state

PCM composition

Auxiliary program variables

Disjoint union

(“ghost heap”)Tokens / Capabilities

No composition

Monotone state

Maximum

(e.g., trace information)

User-defined ghost state:Pick your favorite!

Common structure of ghost state:Partial commutative monoid (PCM).

A PCM is a set M with an associative,commutative composition operation.

9

Ghost State

Ghost state

PCM composition

Auxiliary program variables

Disjoint union

(“ghost heap”)Tokens / Capabilities

No composition

Monotone state

Maximum

(e.g., trace information)

User-defined ghost state:Pick your favorite!

Common structure of ghost state:Partial commutative monoid (PCM).

A PCM is a set M with an associative,commutative composition operation.

9

Ghost State

Ghost state

PCM composition

Auxiliary program variables

Disjoint union

(“ghost heap”)Tokens / Capabilities

No composition

Monotone state

Maximum

(e.g., trace information)

User-defined ghost state:Pick your favorite!

Common structure of ghost state:Partial commutative monoid (PCM).

A PCM is a set M with an associative,commutative composition operation.

9

Ghost State

Ghost state

PCM composition

Auxiliary program variables

Disjoint union

(“ghost heap”)Tokens / Capabilities

No composition

Monotone state

Maximum

(e.g., trace information)

User-defined ghost state:Pick your favorite!

Common structure of ghost state:Partial commutative monoid (PCM).

A PCM is a set M with an associative,commutative composition operation.

9

Ghost State

Ghost state PCM compositionAuxiliary program variables Disjoint union(“ghost heap”)

Tokens / Capabilities No compositionMonotone state Maximum(e.g., trace information)

User-defined ghost state:Pick your favorite!

Common structure of ghost state:Partial commutative monoid (PCM).

A PCM is a set M with an associative,commutative composition operation.

9

Iris: Resting on Simple Foundations

Invariants

:

{.I ∗ P} e {.I ∗ Q}E atomic(e)I ι ` {P} e {v. Q}E]{ι}

Ghost state (any partial commutative monoid)

:

∀af . a # af ⇒ b # af

a V ba · b = ca ∗ b ⇔ c

a ⇒ V(a)

For specifying some synchronization primitives,these foundations are not enough!

Any PCM can be lifted to a CMRA, so allthe old reasoning remains valid.

10

Iris: Resting on Simple Foundations

Invariants:

{.I ∗ P} e {.I ∗ Q}E atomic(e)I ι ` {P} e {v. Q}E]{ι}

Ghost state (any partial commutative monoid):

∀af . a # af ⇒ b # af

a V ba · b = ca ∗ b ⇔ c

a ⇒ V(a)

For specifying some synchronization primitives,these foundations are not enough!

Any PCM can be lifted to a CMRA, so allthe old reasoning remains valid.

10

Complex Foundations

In previous work:Let’s try to make it simple(r).

With Iris, we can derive the morecomplex reasoning principles from the

simple foundations.

11

Iris: Resting on Simple Foundations

Invariants:

{.I ∗ P} e {.I ∗ Q}E atomic(e)I ι ` {P} e {v. Q}E]{ι}

Ghost state (any partial commutative monoid):

∀af . a # af ⇒ b # af

a V ba · b = ca ∗ b ⇔ c

a ⇒ V(a)

For specifying some synchronization primitives,these foundations are not enough!

Any PCM can be lifted to a CMRA, so allthe old reasoning remains valid.

12

First-Order Ghost State

User-defined ghost state: PCM M

LogicIris(M)

Iris 1.0 could not handlehigher-order ghost state.

13

First-Order Ghost State

User-defined ghost state: PCM M

LogicIris(M)

Iris 1.0 could not handlehigher-order ghost state.

13

Higher-Order Ghost State

User-defined ghost state: PCM Mreferring to Iris assertions

LogicIris(M)

Iris 1.0 could not handlehigher-order ghost state.

13

Higher-Order Ghost State

User-defined ghost state: PCM Mreferring to Iris assertions

LogicIris(M)

Iris 1.0 could not handlehigher-order ghost state.

13

Contributions

• Motivate why higher-order ghost state isuseful.• Demonstrate how to extend Iris to supporthigher-order ghost state.

14

Contributions

• Motivate why higher-order ghost state isuseful.• Demonstrate how to extend Iris to supporthigher-order ghost state.

14

Contributions

• Motivate why higher-order ghost state isuseful.• Demonstrate how to extend Iris to supporthigher-order ghost state.

14

Contributions

• Motivate why higher-order ghost state isuseful.• Demonstrate how to extend Iris to supporthigher-order ghost state.

14

Barrier

let b = newbarrier() in

[computation];

// Have: P ∗ Q

signal(b)wait(b);

{True}

wait(b);

// Have: P // Have: Q

[use result [use resultof computation] of computation]

Receive capability issplit.

{True}

let b = newbarrier()

{send(b,P) ∗ recv(b,P)}

{send(b,P) ∗ P} signal(b) {True}

{recv(b,P)} wait(b) {P}

recv(b,P ∗ Q) V recv(b,P) ∗ recv(b,Q)

15

Barrier

let b = newbarrier() in

[computation];

// Have: P ∗ Q

signal(b)wait(b);

{True}

wait(b);

// Have: P // Have: Q

[use result [use resultof computation] of computation]

Receive capability issplit.

{True}

let b = newbarrier()

{send(b,P) ∗ recv(b,P)}

{send(b,P) ∗ P} signal(b) {True}

{recv(b,P)} wait(b) {P}

recv(b,P ∗ Q) V recv(b,P) ∗ recv(b,Q)

15

Barrier (simple version)

let b = newbarrier() in

[computation];

// x 7→ ? is initialized

signal(b)

{True}

wait(b);

// x 7→ ? can be used

[use resultof computation]

{True}

let b = newbarrier()

{send(b,P) ∗ recv(b,P)}

{send(b,P) ∗ P} signal(b) {True}

{recv(b,P)} wait(b) {P}

Capability to send x 7→ ?.

Capability to receive x 7→ ?.

16

Barrier (simple version)

let b = newbarrier() in

[computation];// x 7→ ? is initializedsignal(b)

{True}

wait(b);// x 7→ ? can be used[use resultof computation]

x 7→ ?

{True}

let b = newbarrier()

{send(b,P) ∗ recv(b,P)}

{send(b,P) ∗ P} signal(b) {True}

{recv(b,P)} wait(b) {P}

Capability to send x 7→ ?.

Capability to receive x 7→ ?.

16

Barrier (simple version)

let b = newbarrier() in

[computation];// P is establishedsignal(b)

{True}

wait(b);// P can be used[use resultof computation]

P

{True}

let b = newbarrier()

{send(b,P) ∗ recv(b,P)}

{send(b,P) ∗ P} signal(b) {True}

{recv(b,P)} wait(b) {P}

Capability to send P.

Capability to receive P.

16

Barrier (simple version)

let b = newbarrier() in

[computation];// P is establishedsignal(b)

{True}

wait(b);// P can be used[use resultof computation]

P

{True}

let b = newbarrier()

{send(b,P) ∗ recv(b,P)}

{send(b,P) ∗ P} signal(b) {True}

{recv(b,P)} wait(b) {P}

Capability to send P.

Capability to receive P.

16

Barrier (simple version)

let b = newbarrier() in

[computation];// P is establishedsignal(b)

{True}

wait(b);// P can be used[use resultof computation]

P

{True}

let b = newbarrier()

{send(b,P) ∗ recv(b,P)}

{send(b,P) ∗ P} signal(b) {True}

{recv(b,P)} wait(b) {P}

Capability to send P.

Capability to receive P.

16

Barrier

let b = newbarrier() in

[computation];

// Have: P ∗ Q

signal(b)wait(b);

{True}

wait(b);

// Have: P // Have: Q

[use result [use resultof computation] of computation]

Receive capability issplit.

{True}

let b = newbarrier()

{send(b,P) ∗ recv(b,P)}

{send(b,P) ∗ P} signal(b) {True}

{recv(b,P)} wait(b) {P}

recv(b,P ∗ Q) V recv(b,P) ∗ recv(b,Q)

17

Barrier

let b = newbarrier() in

[computation];// Have: P ∗ Qsignal(b)

wait(b);

{True}

wait(b);// Have: P // Have: Q[use result [use resultof computation] of computation]

Receive capability issplit.

P Q

{True}

let b = newbarrier()

{send(b,P) ∗ recv(b,P)}

{send(b,P) ∗ P} signal(b) {True}

{recv(b,P)} wait(b) {P}

recv(b,P ∗ Q) V recv(b,P) ∗ recv(b,Q)

17

Barrier

let b = newbarrier() in

[computation];// Have: P ∗ Qsignal(b)

wait(b);

{True}

wait(b);// Have: P // Have: Q[use result [use resultof computation] of computation]

Receive capability issplit.

P Q

{True}

let b = newbarrier()

{send(b,P) ∗ recv(b,P)}

{send(b,P) ∗ P} signal(b) {True}

{recv(b,P)} wait(b) {P}

recv(b,P ∗ Q) V recv(b,P) ∗ recv(b,Q)

17

Barrier: A little history

• Spec first proposedby Mike Doddset al. (2011)

• First proof laterfound to be flawed• Fixed using namedpropositions

{True}

let b = newbarrier()

{send(b,P) ∗ recv(b,P)}

{send(b,P) ∗ P} signal(b) {True}

{recv(b,P)} wait(b) {P}

recv(b,P ∗ Q) Vrecv(b,P) ∗ recv(b,Q)

18

Barrier: A little history

• Spec first proposedby Mike Doddset al. (2011)• First proof laterfound to be flawed• Fixed using namedpropositions

{True}

let b = newbarrier()

{send(b,P) ∗ recv(b,P)}

{send(b,P) ∗ P} signal(b) {True}

{recv(b,P)} wait(b) {P}

recv(b,P ∗ Q) Vrecv(b,P) ∗ recv(b,Q)

18

Named Propositions

∀P. TrueV ∃γ. γ 7→ P

∀γ,P,Q. (γ 7→ P ∗ γ 7→ Q)⇒ (P⇔ Q)

Gives a fresh name γ to P.

Agreement about propositionnamed γ.

Derive named propositions fromlower-level principles:

Build named propositions onghost state.

19

Named Propositions

∀P. TrueV ∃γ. γ 7→ P

∀γ,P,Q. (γ 7→ P ∗ γ 7→ Q)⇒ (P⇔ Q)

Gives a fresh name γ to P.P does not have to hold!

Agreement about propositionnamed γ.

Derive named propositions fromlower-level principles:

Build named propositions onghost state.

19

Named Propositions

∀P. TrueV ∃γ. γ 7→ P

∀γ,P,Q. (γ 7→ P ∗ γ 7→ Q)⇒ (P⇔ Q)

Gives a fresh name γ to P.P does not have to hold!

Agreement about propositionnamed γ.

Derive named propositions fromlower-level principles:

Build named propositions onghost state.

19

Named Propositions

∀P. TrueV ∃γ. γ 7→ P

∀γ,P,Q. (γ 7→ P ∗ γ 7→ Q)⇒ (P⇔ Q)

Gives a fresh name γ to P.P does not have to hold!

Agreement about propositionnamed γ.

Derive named propositions fromlower-level principles:

Build named propositions onghost state.

19

Named Propositions

∀P. TrueV ∃γ. γ 7→ P

∀γ,P,Q. (γ 7→ P ∗ γ 7→ Q)⇒ (P⇔ Q)

Gives a fresh name γ to P.P does not have to hold!

Agreement about propositionnamed γ.

Derive named propositions fromlower-level principles:

Build named propositions onghost state.

19

Named Propositions

∀P. TrueV ∃γ. γ 7→ P

∀γ,P,Q. (γ 7→ P ∗ γ 7→ Q)⇒ (P⇔ Q)

Gives a fresh name γ to P.Allocates new slot in “table”.

Agreement about row γ of the“table”.

Derive named propositions fromlower-level principles:

Build named propositions onghost state.

19

Higher-Order Ghost State

User-defined ghost state: PCM Mreferring to Iris assertions

LogicIris(M)

Iris 1.0 could not handlehigher-order ghost state.

20

Contributions

• Motivate why higher-order ghost state isuseful.• Demonstrate how to extend Iris to supporthigher-order ghost state.

21

Higher-Order Ghost State: Technicalities

User-defined ghost state Mreferring to Iris assertions

LogicIris(M)

22

Higher-Order Ghost State: Technicalities

We got a problem with ourghost state.

Who we gonna call?

User-defined ghost state Mreferring to Iris assertions

LogicIris(M)

22

Higher-Order Ghost State: Technicalities

User-defined ghost state Mreferring to Iris assertions

LogicIris(M)

22

Higher-Order Ghost State: Technicalities

Step-Indexing

• Introduced 2001 by Appeland McAllester

• Used to solve circularitiesin models of higher-orderstate

User-defined ghost state Mreferring to Iris assertions

LogicIris(M)

22

Higher-Order Ghost State: Technicalities

Step-Indexing• Introduced 2001 by Appeland McAllester

• Used to solve circularitiesin models of higher-orderstate

User-defined ghost state Mreferring to Iris assertions

LogicIris(M)

22

Higher-Order Ghost State: Technicalities

• Equip PCMs with a“step-indexing structure”.

→ CMRA

• Let user define a functoryielding a CMRA.

• Tie the knot by taking afixed-point.

User-defined ghost state Mreferring to Iris assertions

LogicIris(M)

22

Higher-Order Ghost State: Technicalities

• Equip PCMs with a“step-indexing structure”.→ CMRA

• Let user define a functoryielding a CMRA.

• Tie the knot by taking afixed-point.

User-defined ghost state Mreferring to Iris assertions

LogicIris(M)

22

Higher-Order Ghost State: Technicalities

• Equip PCMs with a“step-indexing structure”.→ CMRA

• Let user define a functoryielding a CMRA.

• Tie the knot by taking afixed-point.

User-defined ghost state Mreferring to Iris assertions

LogicIris(M)

22

Higher-Order Ghost State: Technicalities

• Equip PCMs with a“step-indexing structure”.→ CMRA

• Let user define a functoryielding a CMRA.

• Tie the knot by taking afixed-point.

User-defined ghost state Mreferring to Iris assertions

LogicIris(M)

22

Higher-Order Ghost State: Technicalities

• Equip PCMs with a“step-indexing structure”.→ CMRA

• Let user define a functoryielding a CMRA.

• Tie the knot by taking afixed-point.

User-defined ghost state Mreferring to Iris assertions

LogicIris(M)

22

Named Propositions

∀P. TrueV ∃γ. γ 7→ P

∀γ,P,Q. (γ 7→ P ∗ γ 7→ Q)⇒ .(P⇔ Q)

Gives a fresh name γ to P.

Agreement about propositionnamed γ only holds at the nextstep-index.

Derive named propositions fromlower-level principles:

Build named propositions onghost state.

23

Barrier{True}

let b = newbarrier() in

{send(b,P ∗ Q) ∗ recv(b,P ∗ Q)}

[computation];

// Have: P ∗ Q

{P ∗ Q ∗send(b,P ∗ Q)

}{recv(b,P)} signal(b) {recv(b,Q)}wait(b); {True} wait(b);

// Have: P // Have: Q

{P} {Q}[use result [use resultof computation] of computation]

Receive capability issplit.

P Q

{True}

let b = newbarrier()

{send(b,P) ∗ recv(b,P)}

{send(b,P) ∗ P} signal(b) {True}

{recv(b,P)} wait(b) {P}

recv(b,P ∗ Q) V recv(b,P) ∗ recv(b,Q)

24

Iris: Resting on Simple Foundations

Invariants:

{.I ∗ P} e {.I ∗ Q}E atomic(e)I ι ` {P} e {v. Q}E]{ι}

Ghost state (any CMRA):

∀af ,n. a #n af ⇒ b #n af

a V ba · b = ca ∗ b ⇔ c

a ⇒ V(a)

For specifying some synchronization primitives,these foundations are not enough!

Any PCM can be lifted to a CMRA, so allthe old reasoning remains valid.

25

Iris: Resting on Simple Foundations

Invariants:

{.I ∗ P} e {.I ∗ Q}E atomic(e)I ι ` {P} e {v. Q}E]{ι}

Ghost state (any CMRA):

∀af ,n. a #n af ⇒ b #n af

a V ba · b = ca ∗ b ⇔ c

a ⇒ V(a)

For specifying some synchronization primitives,these foundations are not enough!

Any PCM can be lifted to a CMRA, so allthe old reasoning remains valid.

25

What else?

• Other examples of higher-order ghost state

• How to simplify the model with CMRAs• Coq formalization

Ongoing work:

• Encode invariants using higher-order ghoststate• Applying named propositions in the safetyproof of Rust

Thank you for your attention!

26

What else?

• Other examples of higher-order ghost state• How to simplify the model with CMRAs

• Coq formalization

Ongoing work:

• Encode invariants using higher-order ghoststate• Applying named propositions in the safetyproof of Rust

Thank you for your attention!

26

What else?

• Other examples of higher-order ghost state• How to simplify the model with CMRAs• Coq formalization

Ongoing work:

• Encode invariants using higher-order ghoststate• Applying named propositions in the safetyproof of Rust

Thank you for your attention!

26

What else?

• Other examples of higher-order ghost state• How to simplify the model with CMRAs• Coq formalization

Ongoing work:

• Encode invariants using higher-order ghoststate• Applying named propositions in the safetyproof of Rust

Thank you for your attention!

26

What else?

• Other examples of higher-order ghost state• How to simplify the model with CMRAs• Coq formalization

Ongoing work:

• Encode invariants using higher-order ghoststate• Applying named propositions in the safetyproof of Rust

Thank you for your attention!26