Learning to Fuzz from Symbolic Execution with Application

Learning to Fuzz from Symbolic Executionwith Application to Smart Contracts

MislavBalunovic

NodarAmbroladze

PetarTsankov

MartinVechev

JingxuanHe

Random Fuzzing vs. Symbolic Execution

Random Fuzzing Symbolic Execution

Speed

Inputs

Coverage

Fast

Ineffective Effective

Slow

Low Low

1

2

Initial

. . .

. . .. . .

. . .

. . . . . .

, deposit(), P ether

, setOwner( )

, withdraw(P)

steals P ether from

Smart Contract Testing: Challenge

3

Smart Contract Testing: Challenge

Wanted: Transaction sequences that thoroughly explore the state space

4

Random Fuzzing vs. Symbolic Execution

Imitation Learning based Fuzzer

ILF (this work)

Fast

Effective

High

Random Fuzzing Symbolic Execution

Speed

Inputs

Coverage

Fast

Ineffective Effective

Slow

Low Low

5

- Analyzing Ethereum’s Contract Topology. Kiffer et al.. IMC ’18

~120K contracts ~16K clusters

Imitation Learning

6

Demonstration

Symbolic execution

Human expert

Demonstration

Robot

Fuzzer

Learning to Fuzz from Symbolic Execution

Training Transaction sequences

Symbolic execution expert

Smart contracts

≈ 15K contracts

Fuzzing

New contract Fuzzing policy(neural networks)

7

Coverage

VulnerabilityReport

Learning to Fuzz from Symbolic Execution

Training Transaction sequencesSmart contracts

Symbolic execution expert

Fuzzing

New contract

Coverage

VulnerabilityReportFuzzing policy

(neural networks)8

Smart Contract Fuzzing Policy

TestedContract

Transaction

Feedback

! = ($ &̅ , ()*+),, -./0*!)

= + + +$ &̅ ()*+), -./0*!

Example: a Uniformly Random Policy

2*3$/,.(4):

2*3$/,.(536*-!0,)($)):

72*3$/,. 0,9: $ is payableC(0) = 1 otherwise:

2*3$/,.(5JKLJM5):

may modify blockchain state

9

FuzzingPolicy

Neural Network Fuzzing Policy

GRUfuzzat step 3

FCNfunc

GRUint + FCNint

FCNsender

FCNamount

GRUfuzzat step 3 − 1

GRUfuzzat step 3 + 1

ℎQRS

ℎQ

10

Featureof $QRS ℎQ

hiddenstate

hidden state

$Q

&̅Q

()*+),Q

-./0*!Q

Features of 4

Neural Network Fuzzing Policy – Fuzzing State

GRUfuzzat step 3 − 1

GRUfuzzat step 3

Feature of $QRS Current hidden state

11

e.g., Coverage, opcodes, function name. (can be

dynamic)Last

hiddenstate

[3.5, 0.3, 4.0, …]

[1, 6.2, 5, …] [1.2, 8.7, 2.5, …]

Neural Network Fuzzing Policy

GRUfuzzat step 3

FCNfunc

GRUint + FCNint

FCNsender

FCNamount

GRUfuzzat step 3 − 1

GRUfuzzat step 3 + 1

ℎQRS

ℎQ

12

Featureof $QRS ℎQ

hiddenstate

hidden state

$Q

&̅Q

()*+),Q

-./0*!Q

Features of 4

Neural Network Fuzzing Policy – Function

FCNfunc+

Softmax

13

Current hidden state [1.2, 8.7, 2.5, …]

[[1, 6.2, 5, …],[4, 3.7, 6, …],…[2, 9.2, 7, …]]

Feature of 4 SetOwner

Deposit

Withdraw

Withdraw

Neural Network Fuzzing Policy

GRUfuzzat step 3

FCNfunc

GRUint + FCNint

FCNsender

FCNamount

GRUfuzzat step 3 − 1

GRUfuzzat step 3 + 1

ℎQRS

ℎQ

14

Featureof $QRS ℎQ

hiddenstate

hidden state

$Q

&̅Q

()*+),Q

-./0*!Q

Features of 4

One-hot

Neural Network Fuzzing Policy – Arguments

GRUintat step 0

GRUintat step 1

15

. . .

FCNint FCNint

[0, 0, 1, 0, …]

Current hidden state[1.2, 8.7, 2.5, …]

1

0x100x800

0x200. . .

11

0x100x800

0x200. . .

0x200

Distribution over 50 seed integer values from expert

Neural Network Fuzzing Policy

GRUfuzzat step 3

FCNfunc

GRUint + FCNint

FCNsender

FCNamount

GRUfuzzat step 3 − 1

GRUfuzzat step 3 + 1

ℎQRS

ℎQ

16

Featureof $QRS ℎQ

hiddenstate

hidden state

$Q

&̅Q

()*+),Q

-./0*!Q

Features of 4

Learning to Fuzz from Symbolic Execution

Training Transaction sequencesSmart contracts

Symbolic execution expert

Fuzzing

New contract

Coverage

VulnerabilityReportFuzzing policy

(neural networks)17

Symbolic Execution Expert

. . .ExecuteExecute

TQUQV TS TW

18

Symbolic: VerXS&P 2020

!S

!W

RevisitRevisit Revisit

Learning to Fuzz from Symbolic Execution

Training Transaction sequencesSmart contracts

Symbolic execution expert

Fuzzing

New contract

Coverage

VulnerabilityReportFuzzing policy

(neural networks)19

Training Neural Network Fuzzing Policy

20

$Q

&̅Q

()*+),Q

-./0*!Q

!Q by expert

Cross-Entropy loss

Back Prop.

NN Policyat step 3 − 1

InferenceNN Policyat step 3

Features

!Q by expert

HiddenState

Learning to Fuzz from Symbolic Execution

Training Transaction sequencesSmart contracts

Symbolic execution expert

Fuzzing

New contract

Coverage

VulnerabilityReportFuzzing policy

(neural networks)21

ILF System: Coverage & Vulnerability Detection

• Instruction coverage.• Basic block coverage.

22

• Locking: The contract cannot send out but can receive ether.• Leaking: An attacker can steal ether from the contract.• Suicidal: An attacker can deconstruct the contract.• Block Dependency: Ether transfer depends on block state variables.• Unhandled Exception: Root call does not catch exceptions from child calls.• Controlled Delegatecall: Transaction parameters explicitly flow into

arguments of a delegatecall instruction.

Evaluation

23

• 18,496 Contracts (5,013 Large & 13,483 Small)• 5-fold Cross Validation

• Echidna• UNIF• EXPERT • MAIAN

• ContractFuzzer

• Vulnerability Detection• Coverage & Speed• Fuzzing Components • Case Study

Coverage: ILF vs. Fuzzers

0%

20%

40%

60%

80%

100%

0 200 400 600 800 1000Number of Transactions

Instr. Coverage

ILF UNIF Echidna

24

0%

20%

40%

60%

80%

100%

0 200 400 600 800 1000Number of Transactions

Instr. Coverage

ILF UNIF Echidna

Large contractsSmall contracts

Coverage: ILF vs. Symbolic Expert

25

50%

60%

70%

80%

90%

100%

Small Contracts Large Contracts

Instr. Coverage

EXPERT

ILF (#tx same as EXPERT)

ILF (2k txs)

Small: 30 txs, 547sLarge: 49 txs, 2,580s

Small: 13sLarge: 17s148 txs/s

Vulnerability Detection

0%

20%

40%

60%

80%

100%

Leaking Suicidal Locking

% of True Vulnerabilities

ILF UNIF MAIAN

0%

20%

40%

60%

80%

100%

BlockDependency

UnhandledException

ControlledDelegatecall

% of True Vulnerabilities

ILF UNIF ContractFuzzer

26

∪ ∪

13 FPs

6 FPs

ILF: 0 FPs

Importance of Policy Components

27

50%

60%

70%

80%

90%

100%

Coverage Leaking

ILF

ILF-func

ILF-args

ILF-sender

ILF-amount

Mostimportant

All components are necessary

Summary

28

?Q & A