Obliviate: A Data Oblivious File System for Intel SGX

Adil AhmadKyungtae Kim

Muhammad Ihsanulhaq SarfarazByoungyoung Lee

1

Clouds? The Ultimate Dream?

CloudsUser

2

Clouds? The Ultimate Dream?

CloudsUser

2

Clouds? The Ultimate Dream?

CloudsUser

Hmm, SGX??

2

Clouds? The Ultimate Dream?

CloudsUser

Hmm, SGX??

2

Clouds? The Ultimate Dream?

CloudsUser

Hmm, SGX??

2

Clouds? The Ultimate Dream?

CloudsUser

Hmm, SGX?? Thanks,

SGX?! ☺

2

Clouds? The Ultimate Dream?

CloudsUser

Hmm, SGX?? Thanks,

SGX?! ☺

2

The real world is a bit more complicated!

The sorcery behind SGX

Program’s Address Space

3

The sorcery behind SGX

Program’s Address Space

Non-Enclave

Enclave

3

Confidentiality and integrity-protected

Trusted execution region

The sorcery behind SGX

Program’s Address Space

Non-Enclave

Enclave

SystemComponents

Restricted by the processor

3

Confidentiality and integrity-protected

Trusted execution region

Possible SGX File Systems

Disk 4

Possible SGX File Systems

Disk

Enclaves are ring-3

4

Possible SGX File Systems

Disk

Enclaves are ring-3

Rely on OS for ring-0 ops

Operating System

4

Possible SGX File Systems

Disk

Enclaves are ring-3

1. open(“a.txt”);2. read(2, 0x1000, 4096);3. ….

Rely on OS for ring-0 ops

Operating System

4

Possible SGX File Systems

Disk

Enclaves are ring-3

1. open(“a.txt”);2. read(2, 0x1000, 4096);3. ….

Rely on OS for ring-0 ops

Operating System

Allow OS to handle file buffer

(native)

4

Possible SGX File Systems

Disk

Enclaves are ring-3

1. open(“a.txt”);2. read(2, 0x1000, 4096);3. ….

Rely on OS for ring-0 ops

Operating System

Allow OS to handle file buffer

(native)

Buffer the file within the enclave

(in-memory)

4

Side-channel attacks against in-memory FS

OperatingSystem

Enclave

Page table attacks against SGX[S&P14, SEC17]

Cache attacks against SGX[DIMVA17, WOOT17, EuroSec17]

5

Data.txt

Side-channel attacks against in-memory FS

Access Frame #

0 0x1000

0 0x1001

0 0x1002

0 0x1003

0 0x1004

Page Table

OperatingSystem

Enclave

Accessed by the enclave

Page table attacks against SGX[S&P14, SEC17]

Cache attacks against SGX[DIMVA17, WOOT17, EuroSec17]

5

Data.txt

Side-channel attacks against in-memory FS

Access Frame #

0 0x1000

0 0x1001

0 0x1002

0 0x1003

0 0x1004

Page Table

OperatingSystem

Enclave

Accessed by the enclave

1 0x1000

1 0x1003

Page table attacks against SGX[S&P14, SEC17]

Cache attacks against SGX[DIMVA17, WOOT17, EuroSec17]

5

Data.txt

Side-channel attacks against in-memory FS

Access Frame #

0 0x1000

0 0x1001

0 0x1002

0 0x1003

0 0x1004

Page Table

OperatingSystem

Enclave

Accessed by the enclave

1 0x1000

1 0x1003

Page table attacks against SGX[S&P14, SEC17]

cache-set 0

cache-set 1

cache-set 2

cache-set 3

Cache

Cache attacks against SGX[DIMVA17, WOOT17, EuroSec17]

5

Data.txt

Side-channel attacks against in-memory FS

Access Frame #

0 0x1000

0 0x1001

0 0x1002

0 0x1003

0 0x1004

Page Table

OperatingSystem

Enclave

Accessed by the enclave

1 0x1000

1 0x1003

Page table attacks against SGX[S&P14, SEC17]

cache-set 0

cache-set 1

cache-set 2

cache-set 3

Cache

cache-set 0

cache-set 3Cache attacks against SGX

[DIMVA17, WOOT17, EuroSec17]

5

Data.txt

Case Study: Attacking SQlite

Doctor Cloud

6

Case Study: Attacking SQlite

Doctor

Doctor attempts to access a patient’s history

Cloud

6

Case Study: Attacking SQlite

Query1: Bob’s heart history

Doctor

Doctor attempts to access a patient’s history

Cloud

6

Query2: Alice’s heart history

Case Study: Attacking SQlite

Query1: Bob’s heart history

Doctor

SGX-protected SQLite

Doctor attempts to access a patient’s history

Cloud

6

Query2: Alice’s heart history

Case Study: Attacking SQliteName

LungsCondition

Heartcondition

Query1: Bob’s heart history

Doctor

SGX-protected SQLite

Doctor attempts to access a patient’s history

Cloud

6

Query2: Alice’s heart history

What the attacker sees?

med.db

Name

Bob …

LungCondition

…

Heartcondition

… … …

Alice …

Eve …

…

…

What the attacker sees?

med.db

Query1:Bob’s heart

historyName

Bob …

LungCondition

…

Heartcondition

… … …

Bob …

Alice …

Eve …

…

…

What the attacker sees?

med.db

Query1:Bob’s heart

history 1. open(”med.db”, . .);2. pread64(…,4096,0);3. pread64(…,4096,4096);4. pread64(…,4096,32768);

Syscall Snooping Attack

Name

Bob …

LungCondition

…

Heartcondition

… … …

Bob …

Alice …

Eve …

…

…

What the attacker sees?

med.db

Query1:Bob’s heart

history 1. open(”med.db”, . .);2. pread64(…,4096,0);3. pread64(…,4096,4096);4. pread64(…,4096,32768);

Syscall Snooping Attack

Name

Bob …

LungCondition

…

Heartcondition

… … …

Bob …

Alice …

Eve …

…

…

Page Table Attack

Time

Ad

dre

ss

What the attacker sees?

med.db

Query1:Bob’s heart

history 1. open(”med.db”, . .);2. pread64(…,4096,0);3. pread64(…,4096,4096);4. pread64(…,4096,32768);

Syscall Snooping Attack

Name

Bob …

LungCondition

…

Heartcondition

… … …

Bob …

Alice …

Eve …

…

…

Page Table Attack

Query1

Time

Ad

dre

ss

What the attacker sees?

med.db

Query1:Bob’s heart

history 1. open(”med.db”, . .);2. pread64(…,4096,0);3. pread64(…,4096,4096);4. pread64(…,4096,32768);

Syscall Snooping Attack

Name

Bob …

LungCondition

…

Heartcondition

… … …

Bob …

Alice …

Eve …

…

…

Alice …

Page Table Attack

Time

Ad

dre

ss

Query2:Alice’s heart

history

What the attacker sees?

med.db

Query1:Bob’s heart

history 1. open(”med.db”, . .);2. pread64(…,4096,0);3. pread64(…,4096,4096);4. pread64(…,4096,32768);

1. open(”med.db”, . .);2. pread64(…,4096,0);3. pread64(…,4096,4096);4. pread64(…,4096,40960);

Syscall Snooping Attack

Name

Bob …

LungCondition

…

Heartcondition

… … …

Bob …

Alice …

Eve …

…

…

Alice …

Page Table Attack

Time

Ad

dre

ss

Query2:Alice’s heart

history

What the attacker sees?

med.db

Query1:Bob’s heart

history 1. open(”med.db”, . .);2. pread64(…,4096,0);3. pread64(…,4096,4096);4. pread64(…,4096,32768);

1. open(”med.db”, . .);2. pread64(…,4096,0);3. pread64(…,4096,4096);4. pread64(…,4096,40960);

Syscall Snooping Attack

Name

Bob …

LungCondition

…

Heartcondition

… … …

Bob …

Alice …

Eve …

…

…

Alice …

Page Table Attack

Time

Ad

dre

ss

Query2

Query2:Alice’s heart

history

What the attacker sees?

med.db

Query1:Bob’s heart

history 1. open(”med.db”, . .);2. pread64(…,4096,0);3. pread64(…,4096,4096);4. pread64(…,4096,32768);

1. open(”med.db”, . .);2. pread64(…,4096,0);3. pread64(…,4096,4096);4. pread64(…,4096,40960);

Syscall Snooping Attack

Name

Bob …

LungCondition

…

Heartcondition

… … …

Bob …

Alice …

Eve …

…

…

Alice …

Page Table Attack

Time

Ad

dre

ss

Query2

Query2:Alice’s heart

history

Predictable access patterns in file operations leak sensitive information!

What should we do?

8

What should we do?

Masking individual memory side-channels is risky

8

What should we do?

Masking individual memory side-channels is risky

Memory side-channels rely on predictable access patterns

8

What should we do?

Masking individual memory side-channels is risky

Memory side-channels rely on predictable access patterns

How to provide strong protection despite memory traces?

8

What should we do?

Masking individual memory side-channels is risky

Memory side-channels rely on predictable access patterns

How to provide strong protection despite memory traces?

8

Oblivious RAM is one possible solution to this problem

Oblivious RAM

User Clouds

B

D E

C

F

A

User’s goal: Securely access data stored in the cloud

Attacker’s goal: Figure out what data-block is being accessed

9

Path ORAM

Improved variant of Oblivious RAM [Stephanov et. al, CCS12]

d

d C

A B d D

Server

A 00

B 01

C 10

D 11

Position Map

Client

Stash

stores position of block

stores acquired blocks

holds encrypted real blocks and dummy blocks

ORAM Tree

0 1

0 01 1 d dummy

A real

Legend

10

11

11

OBLIVIATE!

11

OBLIVIATE!

11

Obliviate: memory charm against the OS ☺

Obliviate

Filesystem EnclaveApplication Enclave

Disk

12

Obliviate: memory charm against the OS ☺

Obliviate

Filesystem EnclaveApplication Enclave

Disk

12

(Init) load all files into

ORAM Tree(s)

ORAM Trees

C

A B D

Obliviate: memory charm against the OS ☺

Trusted Proxy

1. FS Syscall Interceptor

Obliviate

Filesystem EnclaveApplication Enclave

Disk

12

ORAM Trees

C

A B D

Obliviate: memory charm against the OS ☺

Trusted Proxy

Obliviate

2. Encrypted Channel

Filesystem EnclaveApplication Enclave

Disk

12

ORAM Trees

C

A B D

Obliviate: memory charm against the OS ☺

Trusted Proxy

Obliviate

Filesystem EnclaveApplication Enclave

Disk

12

ORAM Trees

C

A B D

3. Data Oblivious Metadata Handling

Obliviate: memory charm against the OS ☺

Trusted Proxy

Obliviate

Filesystem EnclaveApplication Enclave

Disk

12

ORAM Trees

C

A B D

4. Asynchronous ORAM Operation

Obliviate: memory charm against the OS ☺

Trusted Proxy

Obliviate

Filesystem EnclaveApplication Enclave

Disk5. Extended

Secure Region12

ORAM Trees

C

A B D

Decoupling file system support

Disk

Application Enclaves Obliviate

13

Decoupling file system support

Disk

Application Enclaves Obliviate

Pass all FS syscalls using encrypted

channel

13

Decoupling file system support

Allow Obliviate to worry about securing

file access

Disk

Application Enclaves Obliviate

Pass all FS syscalls using encrypted

channel

13

Decoupling file system support

Allow Obliviate to worry about securing

file access

Disk

Application Enclaves Obliviate

Pass all FS syscalls using encrypted

channel

13

Separation of functions facilitates development!

Legacy application supportApplication

14

Legacy application support

Intercept FS syscalls and encrypt

Trusted Proxy

Application

14

Legacy application support

Intercept FS syscalls and encrypt

Trusted Proxy

Exit-less message queue(SCONE [OSDI16], ELEOS

[EuroSys17])

Application

Oblivate

Disk 14

Legacy application support

Intercept FS syscalls and encrypt

Trusted Proxy

Exit-less message queue(SCONE [OSDI16], ELEOS

[EuroSys17])

Application

Oblivate

Disk 14

No changes from the app developer!

Securing ORAM

Obliviate

Disk15

Application

Securing ORAM

Obliviate

Disk

Position Map

Stash

ORAM client

15

Need to store metadata in enclaveApplication

Securing ORAM

Obliviate

Disk

Position Map

Stash

ORAM client

Obliviate’s enclave is not side-channel

free

15

Application

Securing ORAM

Position Map

Obliviate

Disk

Position Map

Stash

ORAM client

15

Application

Securing ORAM

Position Map

Load from index

Obliviate

Disk

Position Map

Stash

ORAM client

15

Application

Securing ORAM

Position Map

Access Frame #

0 0x1000

0 0x1001

0 0x1002

0 0x1003

Page Table

cache-set 0

cache-set 1

cache-set 2

cache-set 3

Last-Level Cache

1 0x1003

Obliviate

Disk

Position Map

Stash

ORAM client

15

Application

Securing ORAM

Position Map

Use Conditional Move(CMOV)

Access Frame #

0 0x1000

0 0x1001

0 0x1002

0 0x1003

Page Table

cache-set 0

cache-set 1

cache-set 2

cache-set 3

Last-Level Cache

1 0x1003

Obliviate

Disk

Position Map

Stash

ORAM client

15

Application

Securing ORAM

Position Map

Use Conditional Move(CMOV)

Access Frame #

0 0x1000

0 0x1001

0 0x1002

0 0x1003

Page Table

cache-set 0

cache-set 1

cache-set 2

cache-set 3

Last-Level Cachecache-set 0

cache-set 1

cache-set 2

1 0x1003

1 0x1000

1 0x1001

1 0x1002

Obliviate

Disk

Position Map

Stash

ORAM client

15

Application

Securing ORAM

Position Map

Use Conditional Move(CMOV)

Access Frame #

0 0x1000

0 0x1001

0 0x1002

0 0x1003

Page Table

cache-set 0

cache-set 1

cache-set 2

cache-set 3

Last-Level Cachecache-set 0

cache-set 1

cache-set 2

1 0x1003

1 0x1000

1 0x1001

1 0x1002

Obliviate

Disk

Position Map

Stash

ORAM client

15

The attacker cannot distinguish CMOV from MOV

Application

Securing ORAM

Position Map

Use Conditional Move(CMOV)

Access Frame #

0 0x1000

0 0x1001

0 0x1002

0 0x1003

Page Table

cache-set 0

cache-set 1

cache-set 2

cache-set 3

Last-Level Cachecache-set 0

cache-set 1

cache-set 2

1 0x1003

1 0x1000

1 0x1001

1 0x1002

Obliviate

Disk

Position Map

Stash

ORAM client

15

The attacker cannot distinguish CMOV from MOV

Application

Side-channel resistant ORAM implementation!

Extending Enclave Memory

16

Obliviate

Disk

Extending Enclave Memory

16

EPC

Physical Memory

Program

Obliviate

Disk

Large enclaves degrade performance

Extending Enclave Memory

Metadata (small) inside enclave

ORAM Trees (large) outside enclave

16

EPC

Physical Memory

Program

Obliviate

Disk

Encrypted ORAM Trees

C

A B D

Position Map

Stash

ORAM Client

Large enclaves degrade performance

Extending Enclave Memory

Metadata (small) inside enclave

ORAM Trees (large) outside enclave

16

EPC

Physical Memory

Program

Obliviate

Disk

Encrypted ORAM Trees

C

A B D

Position Map

Stash

ORAM Client

Large enclaves degrade performance

Encrypted ORAM trees outside enclave!

Leveraging asynchronicity

CommunicationThread

OperationThread

Obliviate

17

Disk

Encrypted ORAM Trees

C

A B D

Application

Leveraging asynchronicity

CommunicationThread

OperationThread

Obliviate

17

Disk

Encrypted ORAM Trees

C

A B D

Application

(a) read(1, 0x18289, 4096)

Leveraging asynchronicity

CommunicationThread

OperationThread

Obliviate

17

Disk

Encrypted ORAM Trees

C

A B D

(b) Read(A)

Application

(a) read(1, 0x18289, 4096)

Leveraging asynchronicity

CommunicationThread

OperationThread

Obliviate

17

(c) Reply to the request

Disk

Encrypted ORAM Trees

C

A B D

(c) Write-back(A)(b) Read(A)

Application

(a) read(1, 0x18289, 4096)

Leveraging asynchronicity

CommunicationThread

OperationThread

Obliviate

17

(c) Reply to the request

Disk

Encrypted ORAM Trees

C

A B D

(c) Write-back(A)(b) Read(A)

Application

(a) read(1, 0x18289, 4096)

Perform Asynchronous ORAM write-back!

Implementation

1. Obliviate runs using Intel SGX SDK Library

2. Graphene-SGX integration to run “heavyweight”applications, e.g. , SQLite and Lighttpd

18

Performance Evaluation

Evaluated filesystems:

1. Native Filesystem (Non-SGX)

2. In-memory Filesystem (SGX, based on Graphene-SGX)

3. Obliviate (SGX, based on Intel SGX SDK)

19

Iozone Benchmarks

a) Sequential Reads (Bytes/sec) b) Sequential Writes (Bytes/sec)

1

10

100

1000

10000

100000

1000000

10000000

2M 128M 512M 1G

Native FS In-memory FS Obliviate

1

10

100

1000

10000

100000

1000000

10000000

2M 128M 512M 1G

Native FS In-memory FS Obliviate

20

Iozone Benchmarks

a) Sequential Reads (Bytes/sec) b) Sequential Writes (Bytes/sec)

1

10

100

1000

10000

100000

1000000

10000000

2M 128M 512M 1G

Native FS In-memory FS Obliviate

1

10

100

1000

10000

100000

1000000

10000000

2M 128M 512M 1G

Native FS In-memory FS Obliviate

2-3x overhead over the in-memory FS

20

Iozone Benchmarks

a) Sequential Reads (Bytes/sec) b) Sequential Writes (Bytes/sec)

In-memory FS exerts a lot of pressure on EPC

1

10

100

1000

10000

100000

1000000

10000000

2M 128M 512M 1G

Native FS In-memory FS Obliviate

1

10

100

1000

10000

100000

1000000

10000000

2M 128M 512M 1G

Native FS In-memory FS Obliviate

2-3x overhead over the in-memory FS

20

Iozone Benchmarks

a) Sequential Reads (Bytes/sec) b) Sequential Writes (Bytes/sec)

Comparable performance for smaller file sizes

In-memory FS exerts a lot of pressure on EPC

1

10

100

1000

10000

100000

1000000

10000000

2M 128M 512M 1G

Native FS In-memory FS Obliviate

1

10

100

1000

10000

100000

1000000

10000000

2M 128M 512M 1G

Native FS In-memory FS Obliviate

2-3x overhead over the in-memory FS

20

Macro-Benchmarks

a) SQLite Response Times (milli-sec) b) Lighttpd Throughput (Req/s)

100

1000

10000

1K 16K 128K 1M

In-memory FS Obliviate

0

500

1000

1500

2000

2500

INSERT SELECT

In-memory FS Obliviate

21

Macro-Benchmarks

a) SQLite Response Times (milli-sec) b) Lighttpd Throughput (Req/s)

~2x overhead over in-memory FS

100

1000

10000

1K 16K 128K 1M

In-memory FS Obliviate

0

500

1000

1500

2000

2500

INSERT SELECT

In-memory FS Obliviate

21

Conclusion

22

Conclusion

1. All existing SGX filesystems are vulnerable to side-channels

22

Conclusion

1. All existing SGX filesystems are vulnerable to side-channels

2. File system operations can leak sensitive information about program execution.

22

Conclusion

1. All existing SGX filesystems are vulnerable to side-channels

2. File system operations can leak sensitive information about program execution.

3. Obliviate provides theoretically-strong defense against side-channels.

22

Conclusion

1. All existing SGX filesystems are vulnerable to side-channels

2. File system operations can leak sensitive information about program execution.

3. Obliviate provides theoretically-strong defense against side-channels.

22

Opensource: https://github.com/adilahmad17/ObliviateContact: ahmad37@purdue.edu

Thanks! Merci! Shukriya!

23

Extra Slides

24

Securing file system

25

Securing file system

c

a b d

25

Securing file system

c

a b d

Single ORAM Tree protects

file offset

25

Securing file system

c

a b d

c

a b d

c

a b d

c

a b d

c

a b d

c

a b d

c

a b d

Hierarchical ORAM Trees

can protect files

Single ORAM Tree protects

file offset

25

Securing file system

c

a b d

c

a b d

c

a b d

c

a b d

c

a b d

c

a b d

c

a b d

Hierarchical ORAM Trees

can protect files

Single ORAM Tree protects

file offset

Protect both file and file offset!

25