Environment Selection• Application

- Firefox 1.0 or 2.0

- Apache 2.0.36

• Operating System

- Linux

- Windows XP

• Instrumentation Package

- JIT (DynamoRio, Pin)

- Trampoline (Dyninst, Detours, Pin, etc)

Application• Firefox 1.0 (Phase 1)

- Complex app with embedded interpreter

- 39 to 46 applicable vulnerabilities

• Firefox 2.0

- Similar vulnerabilities as 1.0

• Apache 2.0.36

- Less complex application

- 6-8 applicable vulnerabilities

• Proposal: Firefox 1.0

- Many interesting vulnerabilities

- Leverages Phase 1 experience

Operating System• Linux

- Open source

- Open source tools (gcc, Xnee, etc)

- Instrumentation tools are supported better

• Windows XP

- Closed source

- More marketable results

• Proposal: Windows XP

- No show stoppers for Windows

- Shows program is more generally applicable

Instrumentation Tools

• Instrumentation tool approaches

- JIT

- Probe based

• Call interception

- System call

- Library call

JIT Binary Translation

• PIN & DynamoRIO

• Allows us, at runtime, to manipulate every instruction, with:

- Minimal performance overhead

- Full transparency

• Exports interface for building custom tools

• No modifications to hardware, operating system, or application

How does it work? (conceptually)

fetchfetch decodedecode executeexecute

Start

In more detail

120% to 200%

JIT-mode Summary

• Powerful instruction-level instrumentation

- Supports shadow stack

- Supports arbitrary repairs

- Stack-walk

• Direct access to system call gateway

Probe based instrumentation:

PIN probe, Dyninst, Detours

Probe-based Repair

Probe-mode Summary

• Considerably faster than JIT-mode

- No constant performance overhead

• Potential issues- x86: need at least 5 bytes for trampolines

- Can be expensive for fine-grained instrumentation

- Limited to function-level instrumentation

- Does not support shadow stack

Direct System Call Interception

ApplicationApplication

System call System call gatewaygateway

Operating Operating SystemSystem

Interception

Library System Call Interception

ApplicationApplication

System call System call gatewaygateway

Operating Operating SystemSystem

Win32 APIWin32 API

Win32 DLLsWin32 DLLs

Interception

Library Interception

• Can only catch system calls made through API (libc, win32API)

• Malicious attacker could inject a different version of the library we are intercepting

- But that would require code-injection

• Stable, coherent interface

Monitor/Repair Matrix

Tool Type OS StackReplac

eArgs

Changeor drop syscall

Syscall

return value

Performance

PIN JIT Win, Linux

SSSW Y Y Y 500%

240%

PIN Probe Win, Linux SW L L L 180%

DR JIT Win, Linux

SS SW Y L L 400%

220%

Detours Probe Win SW L L L ~180%

Dyninst Probe Linux SW L L L ~180%

PIN• Automatically in-lines

instrumentation code

- Uses callouts‣ More expensive but easy to write

- No restrictions on library usage

• Simple, easy-to-use API

• Works on Linux and Windows

• Two modes of operation: JIT and Probe

- Cover both models we want to use

• Only slightly slower than DynamoRIO

DynamoRIO• Lower level interface

• Library calls are constrained

- Must use DR version of calls (e.g., malloc)

- Some calls (e.g., sockets) not supported

• Does not allow direct manipulation of system calls

• Just released as open source

• Phase 1 code (shadow stack, HeapGuard) now available

Plan• Use Pin to develop prototype

- Supports both JIT and Probe

- Easy to use

• Implement final approach later

- Evaluate numerous exploits

- Understand what our needs are

• Options

- Use probe mode if possible

- Consider DynamoRio if necessary for speed and/or flexibility

Conclusion

• Application: Firefox 1.0

• Operating System: Windows XP

• Instrumentation: Pin for now