DYNAMIC ENFORCEMENT OF KNOWLEDGE-BASED SECURITY POLICIES Michael Hicks University of Maryland, College Park Joint work with Piotr Mardziel, Stephen Magill,

DYNAMIC ENFORCEMENT OF KNOWLEDGE-BASED SECURITY POLICIES

Michael HicksUniversity of Maryland, College Park

Joint work with Piotr Mardziel, Stephen Magill, and Mudhakar Srivatsa (IBM)

2

My research• Programming languages-based techniques for more

reliable, available, and secure software

3

My research• Programming languages-based techniques for more

reliable, available, and secure software

• Recent projects• Dynamic software updating (for C and Java)

• Blending PL and Crypto

• Adapton: Demand-driven Incremental Computation

• Druby: Static types for dynamic languages

4

Why am I here? http://www.pl-enthusiast.net/2014/09/08/probabilistic-programming/

21,858 pageviews,as of yesterday

5

Today’s talk

Using

probabilistic programming

to enforce

knowledge-based security policies

1.The setting, the problem, and our solution2.A technique for probabilistic inference that

is sound with respect to security• Based on abstract interpretation

6

“Free” sites

• Good: Useful service, no direct charge• Bad: Free use paid for by use of personal information• How to balance the two?

7

Status quo: they have your data

your data

you

query/filter

page

Photography

advertisers

+ ads

8

Take back control

Photography

advertisers

query/filterpage +query/filter

queryresponses

your data

you

9

Simpler model (for exposition)

response

querier

you

queryPhotography

10

Useful and non-revealing

Photography

out = 24 ≤ Age ≤ 30& Female?

& Engaged? *

true

* real query used by a facebook advertiser

querier

you

11

Reveals too much!out = (age, zip-code, birth-date) *

reject

* - age, zip-code, birth-date can be used to uniquely identify 63% of Americans

12

The problem: how to decide to answer based on the information revealed?

out := (age, zip-code, birth-date)

out := ageout := 24 ≤ age ≤ 30

& female? & engaged?

Q1 Q2 Q3

?

13

Approach• Maintain a representation of the querier’s belief about secret’s possible values• Each query result revises the belief; reject if actual secret becomes too likely

• Cannot let rejection defeat our protection.

time

Q1 Q3… …Q2

Reject

Belief ≜probabilitydistribution

Bayesian reasoningto revisebelief OK (answer) OK (answer)

14

Contributions

1. Knowledge-based security policies • Dynamic enforcement (CSF’11, JCS’13)

• Threshold to decide when an answer reveals too much• Query rejection does not reveal information

• Risk prediction (IEEE S&P’14, FCS’14)• Model series of interactions between adaptive adversary

and a defender able to change the secret• What is risk of certain change strategies?

All work collected in Piotr Mardizel’s dissertation

15

Contributions

2. Implementing checking and belief revision efficiently

• Treat queries as probabilistic programs, where inputs are distributions that represent beliefs

• Perform probabilistic inference via abstract interpretation, using a novel probabilistic polyhedra

• Domain associates polyhedron with overlapping descriptions of probability mass, for improved precision

• We track upper and lower bounds for sound conditioning and normalization

All work collected in Piotr Mardizel’s dissertation

16

Meet Bob

=0 ≤ bday ≤ 364

1956 ≤ byear ≤ 1992

Bob (born September 24, 1980)bday = 267byear = 1980 Secret

bday

byear

1956

1992

0 364

Assumption: this is accurateeach equally likely

17

bday-query1today := 260;if bday ≤ today && bday < (today + 7) then out := 1 else out := 0

|= (out = 0)

1956

1992

0 364259 267

1956

1992

0 364

18


|= (out = 0)

1956

1992

0 364259 267

=

Problem Policy: Is this acceptable?

19

Idea: policy as knowledge threshold

• Answer a query if, for querier’s revised belief, Pr[my secret] < t• Call t the knowledge threshold

• Choice of t depends on the risk of revelation

20

Bob’s policy

=0 ≤ bday ≤ 364

1956 ≤ byear ≤ 1992

Bob (born September 24, 1980)bday = 267byear = 1980 Secret

PolicyPr[bday] < 0.2Pr[bday,byear] < 0.05

bday

byear

1956

1992

0 364

CurrentlyPr[bday] = 1/365Pr[bday,byear] = 1/(365*37)

Pr[bday = 267] …

each equally likely

P

21

Bob’s policies

PolicyPr[bday] < 0.2

• bday specifically should never be certainly known

22

Bob’s policies

PolicyPr[bday,byear] < 0.05

• the pair of bday,byear should never be certainly known

• doesn’t commit to protecting bday or byear alone• allows queries that need high precision in one, or low

precision in both

23


|= (out = 0)

1956

1992

0 364259 267

1956

1992

0 364

24


|= (out = 0)

1956

1992

0 364259 267

PotentiallyPr[bday] = 1/358 < 0.2Pr[bday,byear] = 1/(358*37) < 0.05

=

25


Next day …

1956

1992

0 364259 267

|= (out = 1)

1956

1992

0 267 So reject?Pr[bday] = 1

P

26

1956

1992

0 267

if bday ≠ 267 if bday = 267

1956

1992

0 364259 268

Querier’s perspective

will get answer will get reject

Assume querier knows policy

27

Rejection problem

| =reject

• Policy: Pr[bday = 267 | out = o] < t• Rejection, intended to protect secret, reveals secret!

28

Rejection revised

• Policy: Pr[bday = 267 | out = o] < t

• Solution?• Decide policy independently of secret

• Revised procedure

• for every possible output o,• for every possible bday b,

• Pr[bday = b | out = o] < t

• So the real bday in particular

29


accept

initial belief

30


reject

(regardless of what bday actually is)

(after bday-query1)

31


(after bday-query1)

accept

32

Implementing belief revision

• Idea: Use probabilistic programming!• Query is a probabilistic program• Inputs are distributions representing adversary beliefs• Observations are actual/possible outputs

• Posterior is conditioned on these

33

• BUT: Enumeration/sampling-based probabilistic interpretation is problematic• In the limit, time taken is proportional to the running time of the

query and the size of the state space• Approximations used may not be sound (may underestimate risk)

• Can address the latter problem using abstract interpretation• Sound approximation of enumeration-based semantics

Implementing belief revision

34

Three slides on abstract interpretation• Given a programming language

• e ::= x | n | e * e

• The concrete interpretation is its actual semantics• Concrete domain is natural numbers• Concrete interpretation Eval(e) is multiplication

• 1 * (2 * 3) = 6, etc.

• An abstract interpretation approximates the semantics• Abstract domain may be intervals of natural numbers [n1,n2]

• Abstract interpretation Aeval(e): [n1,n2] * [n3,n4] = [n1*n3, n2*n4]• [1,4] * ([2,2] * [3,6]) • = [1,4] * [6,12] • = [6,48]

35

Relating abstract and concrete• Concretization function γ relates the abstract domain A

to the concrete one C• γ : A C⟶• γ ([n1,n2]) = { n | n1 ≤ n ≤ n2 } – here, C is the power-set of naturals

• Abstraction function α relates the concrete domain C to the abstract one A• α : C A⟶• α(S) = [n1,n2] where n S implies n∈ 1 ≤ n and n ≤ n2

• Galois insertion: S γ(α(S)) and α(γ(A)) = A⊆

36

Soundness• Aeval(e) (where x appears in e) is sound iff

• Aeval(e)[x=A] = A’• forall n γ(A), Eval(e)∈ [x=n] = n’ implies n’ γ(A’)∈• (With obvious generalization to multiple variables)

• Suppose e is x * 5• Abstract interpretation, assuming x=[1,3]

• Aeval(e)[x=[1,3]] = [5,15]

• Concrete interpretation, assuming x=2 γ([1,3])∈• Eval(e)[x=2] = 10

• Notice that 10 γ([5,15])∈

37

Concrete domain: probability distributions

• Distributions δ : States Real numbers⟶• A state σ is a map from variables to integers

• Semantics of a statement S under distribution δ is written ⟦S⟧δ. Informally, concrete interpretation is:• For each state σ in the support of δ, let p be its probability• Execute S in that state, producing σ’ with probability mass p

• Add to it the probability mass of all other input states that produce σ’

• Note that distributions’ probabilities may not sum to 1• Frequency distributions are more efficient, simplify development

Back to our setting of probabilistic inference:

38

1956

1992

0 364


Implementation by enumeration

1956

1992

364

259267

out = 1

out = 0

| (out = 0) *

* δ | (out = 0) = normalize(δ | (out = 0))

input state

output state

=

41

Abstract domain: probabilistic polyhedra

P1: 0 ≤ bday ≤ 259, 1956 ≤ byear ≤ 1992, out = 0p = 0.000074

P2: 267 ≤ bday ≤ 364, 1956 ≤ byear ≤ 1992, out = 1p = 0.000074

…

1956

1992

0 364

P1

P1: 0 ≤ bday ≤ 364, 1956 ≤ byear ≤ 1992p = 0.000074

1956

1992

0

364

259267

P 1

P 2out = 0

P 3

out = 1

Key elements of Prob. Poly. P: Constraints describing polygons Ci

Probability p per point within the polygon

42

Performance / precision tradeoff

• (Sets of) polyhedra are a standard abstract domain, so we can reuse (parts of) existing tools to operate on them directly

• Can end up with many small regions• May be expensive to manipulate

• Approach• Collapse regions: specify the “boundary”

of a region and bounds on the number and probability of the points within it

• Nonuniform regions hurt precision but fewer regions improve performance

43

nasty-query1… many disjuncts …

age := 2011 – byear;if age = 20 || … || age = 60 then out := true else out := false;

1991

1956

0 364259 267

1961

1971

1981

1992

Too many regions

Let = belief after bday-query1

| (out = false)

44

Approximation

P1: 0 ≤ bday ≤ 259, 1956 ≤ byear ≤ 1992p = 0.000067s = 8580 ( < true size = 9620)

P2: 267 ≤ bday ≤ 364, 1956 ≤ byear ≤ 1992p = 0.000067s = 3234 ( < true size = 3626)

1956

0 364259 267

1961

1971

1981

1992P1

P2

P1: 0 ≤ bday ≤ 259, 1992 ≤ byear ≤ 1992p = 0.000067

P2: 0 ≤ bday ≤ 259, 1982 ≤ byear ≤ 1990p = 0.000067

… (ten total)P10

p,s refer to possible (non-zero probability) points in region

1956

0 364259 267

1961

1971

1981

19921991

approximate

P1 P2

exact

45

Abstraction imprecision

1956

0 364259 267

1961

1971

1981

19921991

abstract

P1 P2

1956

0 364259 267

1961

1971

1981

1992

1956

0 364259 267

1961

1971

1981

1992

1956

0 364259 267

1961

1971

1981

1992

exact

46

nasty-query-2… many disjuncts …

age := 2011 – byear;if age = 20 || … || age = 60 then out := true else out := false;

1956

0 364259 267

1961

1971

1981

1992

Non-uniform regions

Takes the true branch one time out of ten Thus, out = true implies age is a decade or the coin flipped in our favor; former more likely

| out = true

pif 0.1 then out := true

… probabilistic choice:

47

Approximation

1956

0 364259 267

1961

1971

1981

19921991

approximate

P1 P2

P1: 0 ≤ bday ≤ 259, 1956 ≤ byear ≤ 1992p ≤ 0.000074 (less-equal)s = 9620

P2: 267 ≤ bday ≤ 364, 1956 ≤ byear ≤ 1992p ≤ 0.000074s = 3626

1956

0 364259 267

1961

1971

1981

1992P1

P2

P1: 0 ≤ bday ≤ 259, 1992 ≤ byear ≤ 1992p = 0.0000074

P2: 0 ≤ bday ≤ 259, 1991 ≤ byear ≤ 1991p = 0.000074

… (18 total)P18

for policy checkPr[bday = b | out = o] < t

48

Final abstraction

1956

0 364259 267

1961

1971

1981

19921991

P1 P2

For each Pi, storeregion (polyhedron)p: upper bound on probability of each possible points: upper bound on the number of (possible) pointsm: upper bound on the total probability mass (useful)

Also storelower bounds on the above

Pr[A | B] = Pr[A ∩ B] / Pr[B]

ProbabilisticPolyhedron

Relating concrete and abstract

Ci

δ ∈ γ(P) iffsupport(δ) ⊆ γ(C)pmin ≤ δ(σ) ≤ pmax for every σ support(∈ δ)smin ≤ |support(δ)| ≤ smax

mmin ≤ mass(δ) ≤ mmax

support(δ) = {σ | δ(σ) > 0}

pimin, pi

max si

min, simax

mimin, mi

max ∈ γ( )…

δ1 δ2

δ3

δ ∈ γ({P1,P2}) iff δ = δ1 + δ2 with δ1 ∈ γ(P1) and δ2 ∈γ(P2)

similarly for more than two

Abstract Semantics• Define

• ⟦S⟧P• Soundness: if δ γ(P) then ∈ ⟦S⟧δ γ (∈ ⟦S⟧P)

• Need Abstract operations• P1 + P2

• if δi γ(P∈ i) for i = 1,2 then δ1 + δ2 γ(P∈ 1 + P2)

• P | B• if δ γ(P) then ∈ δ | B γ(P ∈ | B)

• p*P• if δ γ(P) then p * ∈ δ γ(p*P)∈

• …• (and likewise for sets of P)

Abstract operation exampleδ1 + δ2 – combine mass from both

δ1 + δ2 = λσ. δ1(σ) + δ2(σ)

δ1 δ2 δ1+δ2

s1max = 100 s2

max = 30

P1

C1

P2

C2

P3 = P1 + P2

s3max ≤ s1

max + s2max = 130

| C1 ⋂ C2 | = 20

20

20

80

| C1 | = 100

| C2 | = 40

s3max = 120

What is the maximum number of possible points in the sum?• determine minimum

overlap (20)

Operations• P3 = P1 + P2

• C3 – convex hull of C1, C2

• s3max – what is the smallest overlap?

• s3min – what is the largest overlap?

• p3max – is overlap possible?

• p3min – is overlap impossible?

• m3max – simple sum m1

max + m2max

• m3min – simple sum m1

min + m2min

• Other operations, similar, complicated formulas abound• Need to

• count number of integer points in a convex polyhedra• Latte

• maximize a linear function over integer points in a polyhedron

• Latte

• convex hull, intersection, affine transform• Parma

± P

PPL

53

Implementation and experiments• Interpreter written in Objective Caml for a simple

imperative language• Calling out to LattE and Parma, as mentioned

• In addition to full-precision polyhedra, implemented probabilistic versions of two other domains• Intervals – constraints have form c1 ≤ x ≤ c2

• Can implement counting without LattE

• Octagons – constraints have form ax + by ≤ c with a,b {-1,0,1}∈

• Experiments on several queries• Results for a Mac Pro with two 2.26 GHz quad-core Xeon

processors using 16 GB of RAM

54

Queries• Birthday query 1• Birthday query 1+2• Birthday query 1+2+special• Pizza

• User in particular age range, in college, lives within 2 mile square

• Photo• User in particular age range, female, engaged

• Travel• Lives in particular country, speaks English, over 21, completed high

school

55

Scales better than enumeration

=0 ≤ bday ≤ 364

1956 ≤ byear ≤ 1992

=0 ≤ bday ≤ 364

1910 ≤ byear ≤ 2010

1 pp

> 1 pp

each equally likely

each equally likely

bday1 small

bday 1 large

56

Performance/precision tradeoffBirthday query 1+2+special

57

Intervals very fast generallyQuery Intervals Octagons Polyhedra

Bday1 (small) 0.01 1.87 2.81

Bday1+2 (small) 0.01 2.9 5.25

Bday1+2+spec 0.47 17.8 23.0

Bday1 (large) 0.01 2.1 2.48

Bday1+2 (large) 0.02 3.02 4.52

Bday1+2+spec 0.58 33.6 46.5

Pizza 0.26 92.7 125.5

Photo 0.02 5.47 7.98

Travel 0.48 126.9 154.5

Times in secondsAll achieve maximum precision when given unlimited polyhedra

58

LattE is the bottleneck

59

Limitations: Prototype impoverished• No real or rational numbers• No probabilistic choice where weight is unknown• No widening/narrowing operator for better handling loops• Little support for structured data

• These things should all be possible, starting from standard domains for abstract interpretation

60

Ongoing work, other applications• Knowledge-based security policies in Symmetric setting

• A and B are private inputs to Alice and Bob, and Charlie computes f(A,B) and returns the result to both

• What can Alice and/or Bob learn about the other’s input?• See our paper in PLAS’12

• Location privacy• Apply knowledge policies to locations as secrets• Must consider time-varying nature of current location• Ongoing work with Neil Toronto and David Van Horn

• More … !

61

Summary of contributions• Knowledge-based security policies• Implementation of belief tracking using probabilistic

abstract interpretation• Performance results using intervals are promising

62

BACKUP

63

Creating dependency• Dependencies can be created.

1956

1992

0 364

P1

1956

1992

0 364

P1

strange-queryif bday – 1956 ¸ byear - 5 Æ bday – 1956 · byear + 5 then out := 1 else out := 0

* not to scale and/or shape

64

Differential privacy• Encourage participation in a database by protecting inference of a record.• Add noise to query result.• Requires trusted curator.• Suitable for aggregate queries where exact results are not required.• Not suitable to a single user protecting him(her)self.

• Using differential privacy on a query with 2 output values results in the incorrect query output almost half the time (for reasonable ²)

65

Composability / Collusion• If collusion is suspected among queries, one can make them share belief

• query outputs are assumed to have been learned by all of them• assumes deterministic queries only

• Confusion problem: non-deterministic queries can result in a revised belief which is less sure of the true (or some) secret value than the initial belief

• If collusion is suspected but not present, this can result in incorrect belief• Abstract solution: abstract union of the revised belief (colluded) and the

prebelief (not colluded)P1 [ P2

if ±i 2 °(Pi) for i = 1,2 then ±1, ±2 2 °(P1 [ P2)

66

Setting, Known Secret• Measures of badness that weigh their values by the distribution of the secret

(or the distribution of the outputs of a query) can be problematic.• An unlikely user could theorize a query is safe, but know it is not safe for

them• Consider conditional min-entropy

• (see Information-theoretic Bounds for Differentially Private Mechanisms)

H1(X | Y) = -log2 y PY(y) maxx PX|Y(x,y) > - log2 t

probability of certain query output

• Our policy:• -log2 maxy maxx PX|Y(x,y) > - log2 t• so that -log2 maxx PX|Y(x,yreal) > - log2 t

67

Abstract Conditioning

1956

0 364259 267

1961

1971

1981

1992

1956

0 364259 267

1961

1971

1981

1992

68

Abstract Conditioning

1956

0 364259 267

1961

1971

1981

19921991

approximate

P1 P2

1956

0 364259 267

1961

1971

1981

1992

1956

0 364259 267

1961

1971

1981

1992

1956

0 364259 267

1961

1971

1981

1992

69

Initial distribution

bday

byear

1956

1992

0 364

P1

1/(37*365) - ² · probability · 1/(37*365) + ²37*365 · # of points · 37*365

• Need attacker’s actual belief to be represented by this P1

• Much easier task than knowing the exact querier belief.

Documents

DYNAMIC ENFORCEMENT OF KNOWLEDGE-BASED SECURITY POLICIES Michael Hicks University of Maryland, College Park Joint work with Piotr Mardziel, Stephen Magill,