70
Inhyuk Seo(inhack), Jisoo Park(J.Sus), Seungjoo Kim SANE(Security Analysis aNd Evaluation) Lab Korea University( 高高高高高 ) Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness

[CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Embed Size (px)

Citation preview

Page 1: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Inhyuk Seo(inhack), Jisoo Park(J.Sus), Seungjoo Kim

SANE(Security Analysis aNd Evaluation) Lab

Korea University(高麗大學校 )

Using the CGC’s fully automated vulnerabil-ity detection tools in security evaluation and its effectiveness

Page 2: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Contents• Who are we?• Introduction• Security Engineering, the Way to Information Assurance• High-Assurance, the Key of CPS• Tools for Security Testing & Evaluation

- Tools for Design Assurance / Tools for Code Assurance

• Demo (Design / Code)• Conclusion• Acknowledgement• Q&A• Reference

Page 3: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Who are we?Inhyuk Seo (徐寅赫 )

E-mail : [email protected]

Jisoo Park received his B.S (2015) in Computer Science Engineering from Dongguk University in Ko-rea. He worked at antivirus company Ahnlab as S/W QA trainee for 6 month. Also he completed high-quality information security education course “Best of the Best” hosted by KITRI(Korea Information Technology Research Institute). Now, He is a M.S course student at CIST SANE Lab, Korea Univer-sity and interested in Common Criteria, Security Engineering(Especially Threat modeling).

Jisoo Park (朴志洙 )

E-mail : [email protected]

My name is Inhyuk Seo(Nick: inhack). I graduated B.S. in Computer Science and Engineering at Hanyang University(ERICA) in 2015. Now I’m a researcher and M.S. of SANE(Security Analaysis aNd Evaluation) Lab at Korea University. In 2012, I completed high-quality information security education course “the Best of the Best(BoB)” hosted by KITRI(Korea Information Technology Research Institute) and participated in many projects related with vulnerability analysis. I’m interested in Programming Language, Software Testing, Machine Learning, Artificial Intelligence.

Page 4: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Seungjoo Gabriel Kim (金昇柱 )E-mail: [email protected] : www.kimlab.netFacebook, Twitter : @skim71

Prof. Seungjoo Gabriel Kim received his B.S, M.S and Ph.D. from Sungkyunkwan University(SKKU) of Korea, in 1994, 1996, and 1999, respectively. Prior to joining the faculty at Korea University (KU) in 2011, he served as Assistant & Associate Professor at SKKU for 7 years. Before that, he served as Director of the Cryptographic Technology Team and the (CC-based) IT Security Evaluation Team of the Korea Internet & Security Agency(KISA) for 5 years. He is currently a Professor in the Graduate School of Information Security Technologies(CIST). Also, He is a Founder and Advisory director of hacker group, HARU and an international security & hacking conference, SECUINSIDE. Prof. Se-ungjoo Gabriel Kim’s research interests are mainly on cryptography, Cyber Physical Security, IoT Se-curity, and HCI Security. He is a corresponding author.

Who are we?

Page 5: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Intro

Level of trust that it really does!

Assurance

The User’s degree of trust in that information

InformationAssurance

Page 6: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Intro

Rise of the Information Assurance

Gulf War has often been called the first in-formation war. “The harbinger of IA”

1991U.S. DoD Directive 5-3600.1 :The first standardized definition of IA

1996

Information Security (INFOSEC) Era

1980 ~

“The communication network that supported Operation Desert Storm was the largest joint theater system ever established. It was built in record time and maintained a phenomenal 98 percent availability rate. At the height of the operation, the system supported 700,000 telephone calls and 152,000 messages per day. More than 30,000 radio frequencies were managed to pro-vide the necessary connectivity and to ensure minimum interference.”

Debra S. Herrmann, “Security Engineering and Information Assurance”

Page 7: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Intro

Information Assurance

“Measures that protect and defend information and information systems by ensuring their availability, integrity, authentication, confidentiality, and non- repudiation. This includes providing for restoration of information systems by incorporating protection, detection, and reaction capabilities.”

DoD Directive 8500.01E

Rise of the Information Assurance

Gulf War has often been called the first in-formation war. “The harbinger of IA”

1991U.S. DoD Directive 5-3600.1 :The first standardized definition of IA

1996

Information Security (INFOSEC) Era

1980 ~

Page 8: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

What are the differences between

Information Security and Information Assurance?

Intro

Page 9: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Intro

Information Security (情報保護 ) Information Assurance (情報保證 )

Dates Since 1980s Since 1998

Subject of protection Information and Information system Business as a whole

Goal Confidentiality, Integrity, Availability

Confidentiality, Integrity, Availability, Non-repudiation, Accountability, Auditability, Transparency, Cost-effectiveness, Effi-ciency

Type of information Primarily electronic All types

Approach Domination of the technical approach, initial attempts to consider soft aspects

All-encompassing multi-disciplinary systematic approach

Security MechanismPrimary focus is on technical security mecha-nism; initial consideration of organizational and human-oriented mechanism

All available (technical, organizational, human-oriented, le-gal)

Role within a business Supporting system, often inducing some re-strictions on business

An integral aspect of business, business en-abler

Flow of security deci-sion Bottom-Top Top-Bottom

Page 10: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Intro

Information Security (情報保護 ) Information Assurance (情報保證 )

Dates Since 1980s Since 1998

Subject of protection Information and Information system Business as a whole

Goal Confidentiality, Integrity, Availability

Confidentiality, Integrity, Availability, Non-repudiation, Accountability, Auditability, Transparency, Cost-effectiveness, Effi-ciency

Type of information Primarily electronic All types

Approach Domination of the technical approach, initial attempts to consider soft aspects

All-encompassing multi-disciplinary systematic approach

Security MechanismPrimary focus is on technical security mecha-nism; initial consideration of organizational and human-oriented mechanism

All available (technical, organizational, human-oriented, le-gal)

Role within a business Supporting system, often inducing some re-strictions on business

An integral aspect of business, business en-abler

Flow of security deci-sion Bottom-Top Top-Bottom

Protecting information and information systems from unauthorized access, use, disclosure, disruption, modification, or

destruction

Validating that the information is au-thentic, trustworthy, and accessible

Page 11: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Security Engineering, the Way to Information Assurance

Page 12: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

What is Information Assurance’s Goal?

Security Engineering

Page 13: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Security Engineering

Goal of Information Assurance

Dependability

The ability of the system to deliver services when re-

quested

Availability

The ability of the system to deliver services as speci-

fied

Reliabil-ity

The ability of the system to protect itself against acci-dental or deliberate intru-

sion

The ability of the system to operate without cata-

strophic failure

Safety Security

Reflect the extent of the user’s confidence that it will operate as users expects that it will not ‘fail’ in normal use

Page 14: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Domain Reliability Security Safety

Financial System Medium High No

DB of Medical Records Medium Medium Medium

Air Traffic Control System Medium High High

Automobile High Medium High

Defcon 23 – Charlie Miller & Chris Valasek “Remote Ex-ploitation of an Unaltered Passenger Vehicle”

It was ‘Low’ at first,

Security Engineering

Goal of Information Assurance

Page 15: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

How can we achieve Information Assurance?

Security Engineering

Page 16: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

How can we achieve Information Assurance?

Security Engineering

Security Engineering

Page 17: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Security Engineering is about building systems to remain dependable in the face of malice, error and mischance. As a discipline, it focuses on the tools, needed to design, implement and test complete systems and to adapt existing systems as their environment evolves.

– Ross Anderson, Computer Laboratory in University of Cambridge -

What is Security Engineering?

Security Engineering

Page 18: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Policy

Assurance

Mechanisms

Policy Assurance

Design Assurance

Implementation Assurance

Operational Assurance

Assurance needed at all stage of System life cycle

Ultimate Goal of Security Engineering

Security Engineering

What is Security Engineering?

Page 19: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Requirements Design Implementation Release Maintenance

System Engineering Life Cycle Process (ISO/IEC/IEEE 15288 : 2015)

• Business or Mis-sion Analysis

• Stakeholder Needs and Re-quirements Definitions

• System Requirements Definition

• Architecture Defini-tion

• Design Definition • System Analysis • Implementation • Integration

• Verification • Transition • Validation • Operation

• Maintenance • Disposal

Security Engineering

What is Security Engineering?

Provide Security Engineering throughout the Life Cycle

Page 20: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Case Study : Microsoft Security Development Life Cycle

Security Engineering

Page 21: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Case Study : Microsoft Security Development Life Cycle

Does it really work?

SQL Server 2000 SQL Server 2005 Competing commercial DB

343

187

Total Vulnerabilities Disclosed 36 Month after Release

46% re-duction

Windows XP

Windows Vista

OS A OS B OS C

11966

400

242157

Total Vulnerabilities Disclosed On year after Release

46% re-duction

After SDLBefore SDL After SDLBefore SDL

91% re-duction

Analysis by Jeff Jones(Microsoft technet security blogWindows Vista One year Vulnerability Report, Microsoft Security Blog 23 Jan 2008

Security Engineering

Page 22: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

High-Assurance, the Key of CPS

Page 23: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

High Assurance, the Key of CPS

What is “High-Assurance”?

High-Assurance means that it can be mathematically proven that the system works precisely as intended and designed.

and High-Assurance development means that there are clear and compelling evidences in each development phase.

Page 24: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

What is “CPS”?

Cyber Physical Systems(CPS) are co-

engineered interacting network of physi-

cal and computational components.

CPS will provide the foundation of our

critical infrastructure, form the basis of

emerging and future smart services, and

improve our quality of life in many areas. Internet of Things Cyber Physical Sys-tem

AssuranceSecurity VS

High Assurance, the Key of CPS

Page 25: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Where “High-Assurance” needed

InformationAssurance

SecurityEngineering

Critical InfrastructureFinanceAviation

GovernmentMedical

AutomotiveRailwayEnergy

.

.

High-As-surance

Apply & Guarantee

High Assurance, the Key of CPS

Page 26: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Some standards or regulations for critical infrastructure are not enough for achieving dependability.

• Most of them don’t have Security feature.Domain Standard / Regulation

Road Vehicles ISO 26262Aviation DO-178B, 178C, 254, 278A ….Medical IEC 62304Railways EN 50128

High Assurance, the Key of CPS

Page 27: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

ISO/IEC 29128 and ISO/IEC 15408 have “Reliability” and “Security”

ISO 26262, DO-254 : Mainly focusing on “Safety” and “Reliability”

Standard / Regulation Assurance LevelISO 26262 ASIL A ASIL B ASIL C ASIL D

DO-254 DAL E DAL D DAL C DAL B DAL AISO/IEC 29128 PAL 1 PAL 2 PAL 3 PAL 4ISO/IEC 15408 EAL1 EAL 2 EAL 3 EAL 4 EAL 5 EAL 6 EAL 7

HighLow

High Assurance, the Key of CPS

Page 28: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

ISO/IEC 29128 and ISO/IEC 15408 have “Reliability” and “Security”

ISO 26262, DO-254 : Mainly focusing on “Safety” and “Reliability”

Standard / Regulation Assurance LevelISO/IEC 29128 PAL 1 PAL 2 PAL 3 PAL 4ISO/IEC 15408 EAL1 EAL 2 EAL 3 EAL 4 EAL 5 EAL 6 EAL 7

HighLow

High Assurance, the Key of CPS

Page 29: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Example : ISO/IEC 29128 Verification of Cryptographic Protocol

Protocol Assur-ance Level

PAL1 PAL2 PAL3 PAL4

Protocol Specifica-tion Semiformal descrip-

tion of protocol speci-fication

Formal description of protocol specifica-tion

Formal description of protocol specification in a tool-specific specification language, whose semantics is mathematically defined

Adversarial Model

Security Property

Self-assessment ev-idence

Informal argument or mathematically formal paper-and-pencil proof that the cryptographic protocol satisfies the given objectives and properties with respect to the adversarial model

Tool-aided bounded verification that the specification of the cryptographic protocol satisfies the given ob-jectives and proper-ties with respect to the adversarial model

Tool-aided un-bounded verification that the specification of the cryptographic protocol satisfies the given objectives and properties with re-spect to the adversar-ial model

Tool-aided un-bounded verification that the specification of the cryptographic protocol in its adver-sarial model achieves and satisfies its objec-tives and properties.

High Assurance, the Key of CPS

Page 30: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Example : Common Criteria ISO/IEC 15408 Evaluation criteria for IT security

Evaluation As-surance Level Description

EAL 7 Formally verified design and tested

EAL 6 Semiformally verified design and tested

EAL 5 Semiformally designed and tested

EAL 4 Methodically designed, tested, and reviewed

EAL 3 Methodically tested and checked

EAL 2 Structurally tested

EAL 1 Functionally tested

Gerwin Klein, Operating System Verification – An Overview

High Assurance, the Key of CPS

Page 31: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Example : Common Criteria ISO/IEC 15408 Corresponding assurance levels in ISO/IEC 29128

High Assurance, the Key of CPS

Page 32: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

How to Get it?

• Measurable & Mathematically provable

Formal Verification

• By using Tools

High Assurance, the Key of CPS

Page 33: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

How to Get it?

Established in March 2012, as a Research Association, which headquarters is located in Tagajo City of Miyagi

Prefecture. CSSC’s testbed is composed of 9-types of simulated plants and it is capable to organize cybersecu-

rity hands-on exercises which simulate cyber attack

Control System Security Center (C-SSC)

Major operation plans – System security verification

High Assurance, the Key of CPS

Page 34: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

How to Get it?

“The goal of the HACMS program is to create technology for the construction of high-assurance cyber-

physical systems, where high assurance is defined to mean functionally correct and satisfying

appropriate safety and security properties.”

Dr. Raymond Richards, Information Innovation Office

Program Manager of HACMS

High-Assurance Cyber Military System (HACMS)

High Assurance, the Key of CPS

Page 35: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Tools for Security Testing & Evaluation

Page 36: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Tools for Security Testing & Evaluation

Automation Tools for Hacker & Bug Hunters• Automation Vulnerability Detection Tools developed by hacker/bug

hunter are only for the purpose of finding 0-day (Unknown Vulnerability) easily.

Automation Tools for EvaluationHigh-AssuranceNo Mistake&

Trusted Result

Evaluator-Independent

Ultimate goal of Security testing & evaluation

There are no mistakes in security testing process and Guarantee objective analysis reports or evaluation results

Independent from evaluator’s capability or expertise. So anyone who uses the same tools should be able to make same results.

Page 37: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

What should we consider when we choose

Automated security testing tools in evaluation?

Tools for Security Testing & Evaluation

Page 38: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Assessment Features for Automated Tools

User-Friendly Effectiveness Scalability

Tools for Security Testing & Evaluation

Page 39: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Tools for Design Assurance

Page 40: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Tools for Design Assurance

Assessment items to choose Automated Tools for Design Assur-ance

(1) User-Friendly

• Usability

• Analysis Report

• Requirement to Evaluator (Expertise, Background Knowledge)

(2) Effectiveness

• Automation Level

• Model Description Method

• Licensing & Cost

(3) Scalability

• Supported Platforms

Page 41: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Cryptographic Protocol

Model Checking

Theorem Proving Based

• NRL

• FDR

• SCYTHER

• ProVerif • AVISPA(TA4SP)

• CryptoVerif • EBMC

…….

• Isabelle/HOL

• BPW

• Game-based Security Proof

• VAMPIRE • …….

Tools for Design Assurance

Page 42: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Tools for Design Assurance

Cryptographic Protocol (Model Checking)

• The Maude NRL Protocol Analyzer (Maude-NPA)

Assessment Items Description

Usability GUI(Graphic User Interface)

Analysis Report O

Requirement to Evaluator Protocol Design & Modeling Ability

Automation Level Interactive

Model Description Method Maude-PSL (Maude Protocol Specification Language)

Licensing & Cost Non-Commercial (University of Illinois)

Supported Platform Mac OS X

Page 43: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Cryptographic Protocol (Model Checking)• FDR(Failure-Divergence-Refinement)

Assessment Items Description

Usability GUI

Analysis Report O

Requirement to Evaluator Protocol Design & Modeling Ability

Automation Level Interactive

Model Description Method Formal Language (CSP)

Licensing & Cost Non-Commercial (University of Oxford)

Supported Platform Linux / Mac OS X

Tools for Design Assurance

Page 44: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Cryptographic Protocol (Model Checking)• Syther

Assessment Items Description

Usability GUI

Analysis Report O

Requirement to Evaluator Protocol Design & Modeling Ability

Automation Level Interactive

Model Description Method SPDL (Standard Page Description Language)

Licensing & Cost Non-Commercial (University of Oxford)

Supported Platform Linux / Windows / Mac OS X

Tools for Design Assurance

Page 45: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Cryptographic Protocol (Model Checking)• ProVerif

Assessment Items Description

Usability CLI (but Easy to Use)

Analysis Report O

Requirement to Evaluator Protocol Design & Modeling Ability

Automation Level Interactive

Model Description Method PV Script (ProVerif Script)

Licensing & Cost Non-Commercial (PROSECCO)

Supported Platform Linux / Windows / Mac OS X

Tools for Design Assurance

Page 46: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Cryptographic Protocol (Theorem Proving)• Isabelle/HOL(Higher-Order Logic)

Assessment Items Description

Usability GUI, IDE(Integrated Development Environment)

Analysis Report O

Requirement to Evaluator Protocol Design & Modeling Ability

Automation Level Interactive

Model Description Method Functional & Logic Language (HOL)

Licensing & Cost Non-Commercial (University of Cambridge)

Supported Platform Linux / Windows / Mac OS X

Tools for Design Assurance

Page 47: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Tools for Code Assurance

Page 48: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Tools for Code Assurance

Assessment Items to choose Automated Tools for Code Assurance (1) User-Friendly

• Usability• Analysis Report• Requirement to Evaluator (Expertise, Background Knowledge)

(2) Effectiveness• Automation Level• Analysis Method• Detectable Vulnerability Type• Code Coverage• Licensing & Cost

(3) Scalability• Supported Languages• Supported Platforms

Page 49: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

CGC(Cyber Grand Challenge) Finalist• Mayhem CRS (ForAllSecure)

• Xandra (TECHx)

• Mechanical Phish (Shellphish)

• Rebeus (Deep Red)

• Crspy (Disekt)

• Galactic (Codejitsu)

• Jima (CSDS)

Tools for Code Assurance

Page 50: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

CGC (Cyber Grand Challenge)

• CRS (Cyber Reasoning System)• Fully Automated Security Testing for Software

(no human intervention!)

GenerateInput

(Random, Mutation, Model-Based, … )

InputGeneration

SoftwareAnalysis

&Excavate

Vulnerability

VulnerabilityScanning

Crash is Ex-ploitable?

Crash Anayl-sis

GenerateExploit CodeAutomatically

ExploitGeneration

PatchedBinary

AutomaticPatching

Tools for Code Assurance

Page 51: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Fortify SCAAssessment Items Description

Usability GUI(Graphic User Interface), Easy to Use

Analysis Report XML Report

Requirement to Evaluator X

Automation Level Fully Automated

Analysis Method Static / Source Code Analyzer

Detectable Vulnerability Type Hundreds of Vulnerability

Code Coverage High Code Coverage

Licensing & Cost Commercial (HP Enterprise)

Supported Languages Java, .NET, C/C++, JSP, PL/SQL, TSQL, Javascript/Ajax, PHP, ASP, VB6, COBOL

Supported Platforms Windows, Linux, Solaris, Mac OS X

Tools for Code Assurance

Page 52: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

CodeSonarAssessment Items Description

Usability GUI, Easy to use

Analysis Report HTML, XML, CSV Report

Requirement to Evaluator X

Automation Level Fully Automated

Analysis Method Static / Source Code Analyzer / Binary Anaylzer

Detectable Vulnerability Type Hundreds of Vulnerability

Code Coverage High Code Coverage

Licensing & Cost Commercial (Grammatech)

Supported Languages C, C++, Java

Supported Platforms Windows, Linux, Solaris

Tools for Code Assurance

Page 53: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

CheckMarx SASTAssessment Items Description

Usability GUI, Easy to Use (Just throw the source code!)Analysis Report Dashboard Report (PDF, RTF, CSV, XML)

Requirement to Evaluator XAutomation Level Fully AutomatedAnalysis Method Static / Source Code Analyzer

Detectable Vulnerability Type Hundreds of VulnerabilityCode Coverage High Code Coverage

Licensing & Cost Commercial (CheckMarx)

Supported LanguagesJava , Javascript , PHP , C# , VB.NET , VB6 , ASP.NET , C/C++ , Apex , Ruby , Perl , Objective-C , Python , Groovy , HTML5 , Swift , APEX , J2SE , J2EE

Supported Platforms Android , iOS , Windows

Tools for Code Assurance

Page 54: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

KLEEAssessment Items. Description

Usability CLI

Analysis Report X

Requirement to Evaluator O

Automation Level Interactive

Analysis Method Dynamic / Concolic Execution

Detectable Vulnerability Type Memory Corruption

Code Coverage High Code Coverage

Licensing & Cost Non-Commercial (Researched by Stanford University)

Supported Languages C, C++, Objective C

Supported Platforms Linux

Tools for Code Assurance

Page 55: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Mayhem (Research Paper Ver.)Assessment Items Description

Usability CLI, Write Input Specification

Analysis ReportO (Exploit Type, Input Source, Symbolic Input Size, Precon-dition, Adivsory ,Exploit Generation Time)

Requirement to Evaluator OAutomation Level InteractiveAnalysis Method Dynamic / Concolic Execution

Detectable Vulnerability Type Memory CorruptionCode Coverage High Code Coverage

Licensing & Cost Non-Commercial (Carnegie Mellon University)Supported Languages Raw Binary CodeSupported Platforms Linux, Windows

Tools for Code Assurance

Page 56: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

SAGEAssessment Items Description

Usability UnknownAnalysis Report Unknown

Requirement to Evaluator OAutomation Level InteractiveAnalysis Method Dynamic / Whitebox Fuzz Testing

Detectable Vulnerability Type Hundreds of VulnerabilityCode Coverage Limited Code Coverage

Licensing & Cost Restriced-Commercial (Microsoft)Supported Languages Raw Binary CodeSupported Platforms Windows

Tools for Code Assurance

Page 57: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

TritonAssessment Items Description

Usability CLI, Write Program based TritonAnalysis Report X

Requirement to Evaluator OAutomation Level InteractiveAnalysis Method Dynamic / Concolic Execution / Framework

Detectable Vulnerability Type Memory CorruptionCode Coverage High Code Coverage

Licensing & Cost Non-Commercial (Carnegie Mellon University)Supported Languages Raw Binary Code (Bordeaux University, Qarkslab)Supported Platforms Linux, Windows, Mac OS X

Tools for Code Assurance

Page 58: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

AFL (American Fuzzy Lop)Assessment Items Description

UsabilityCLI(Command Line Interface)Install & Setup process is a little complexed.But provide colorful user interface and statistics.

Analysis Report Crash/Vulnerability Type by Address SanitizerRequirement to Evaluator O (Crash Analysis, Exploit Generation, Patching)

Automation Level InteractiveAnalysis Method Dynamic / Guided Fuzz Testing

Detectable Vulnerability Type Memory CorruptionCode Coverage High Code Coverage (More time, More Coverage)

Licensing & Cost Open Source (Michael Zalewski)Supported Languages C, C++, Objective C

Supported Platforms Linux, *BSD, Solaris, Mac OS XOn Linux, Only Binary(Blackbox) Testing Possible

Tools for Code Assurance

Page 59: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

IoTcubeAssessment Items Description

Usability Easy to Use (Web Interface, Drag & Drop)Analysis Report O

Requirement to Evaluator XAutomation Level Fully Automated

Analysis MethodSource Code Analysis (Code Clone Detection)Binary Fuzz TestingNetwork Vulnrability Testing (TLS)

Detectable Vulnerability Type Hundreds of VulnerabilityCode Coverage High Code Coverage

Licensing & Cost Non-Commercial (CSSA, cssa.korea.ac.kr, iotcube.net)

Supported Languages C/C++, Raw Binary CodeSupported Platforms Linux, Windows, Mac OS X

Tools for Code Assurance

Page 60: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Mechanical Phish (Shellphish CRS)Assessment Items Description

Usability CLI, Install & Setup process is a little complexed but Easy to Use

Analysis Report -

Requirement to Evaluator X (Vulnerability Excavation, Crash Analysis, Exploit Generation, Patch)

Automation Level Fully Automated

Analysis Method Dynamic, Concolic Execution, Guided Fuzz Testing, Au-tomatic Exploit Generation, Automatic Patching

Detectable Vulnerability Type Memory CorruptionCode Coverage High Code Coverage

Licensing & Cost Non-Commercial (Shellphish)Supported Languages Raw Binary CodeSupported Platforms Linux-Like Platforms(Custom by CGC), Intel x86

Tools for Code Assurance

Page 61: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Demo (Design / Code)

Page 62: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Conclusion

Page 63: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Conclusion

There are many kind of Vulnerability Detection Tools developed by hackers, researchers.

In present, we use these tools for security testing and evaluation. But there are some limits.

• Objectivity• Coverage

Recently, many of hackers research and develop automation tools that can find unknown vulnerability easily.

We can’t apply these tools to security evaluation immediately.

But if fully automated security testing techniques are developed and we make an effort to apply it for evaluation continuously, achieving high-assurance is not too far.

Page 64: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Acknowledgement

This work was supported by Institute for Information & communications Technology

Promotion(IITP) grant funded by the Korea government(MSIP)

(R7117-16-0161,Anomaly detection framework for autonomous vehicles)

Page 65: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Q&A

Page 66: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Reference

Page 67: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Reference

[1] Debra S. Herrmann, “A practical guide to Security Engineering and Information Assurance”

[2] Sommerville, “Software Engineering, 9ed. 11 & 12, Dependability and Security Specification”

[3] Charlie Miller, Chris Valasek, “Remote Exploitation of an Unaltered Passenger Vehicle”

[4] Ross Anderson, “Security Engineering”

[5] ISO/IEC/IEEE 15288 : 2015, “Systems and Software engineering-System life cycle process”,

[6] Joe Jarzombek, “Software & Supply Chain Assurance : A Historical Perspective of Community Collaboration”, Homeland Security

[7] David Burke, Joe Hurd and Aaron Tomb, “High Assurance Software Development”, 2010

[8] Ron Ross, Michael McEilley and Janet Carrier Oren, “NIST SP 800-160 : Systems Security Engineering – Consideration for a Multidisci -plinary Approach in the Engineering of Trustworthy Secure Systems”, 2016

[9] Scott A.Lintelman, Krishna Sampigethaya, Mingyan Li, Radha Poovendran, Richard V. Robinson, “High Assurance Aerospace CPS & Implications for the Automotive Industry”, 2015

[10] NIAP, “Common Criteria-Evaluation and Validation Scheme, Publication #3, Guidance to Validators version 3”, 2014

[11] ISO/IEC 27034-2, “Information technology – Security techniques – Application Security”, 2015

[12] Paul R. Croll, “ISO/IEC/IEEE 15026, Systems and Software Assurance”, 21 st Annual Systems and Software Technology Conference, 2009

Page 68: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Reference

[13] EURO-MILS, “Secure European Virtualisation for Trustworthy Applications in Critical Domains, Used Formal Methods”, 2015

[14] Vijay D’Silva, Daniel Kroening, and Georg Weissenbacher, “A Survey of Automated Techniques for Formal Software Verification”, 2008

[15] Daniel Potts, Rene Bourquin, Lesile Andresen, “Mathematically Verified Software Kernals: Rasing the Bar for High Assurance Imple -mentation

[16] Bernhard Beckert, Daniel Bruns, Sarah Grebing, “Mind the Gap : Formal Verification and the Common Criteria“, 2010

[17] Gerwin Klein, Kevin Elphinstone, Gernot Heiser, June Andronick, David Cock, Philip Derrin, Dhammika Elkaduwe, Kai Engelhardt, Rafal Kolankski, Michel Norrich, Thomas Sewell, Harvey Tuch, Simon Winwood, “seL4 : Formal Verification of an OS Kernel”, 2009

[18] Gerwin Klein, NICTA, “Operating System Verification – An Overview”, 2009

[19] Jesus Diaz, David Arroyo, Francisco B. Rodriguez, “A formal methodology for integral security design and verification of network pro -tocols”, 2012

[20] Yoshikazu Hanatanil, Miyako Ohkubo, Sinichiro Matsuo, Kazuo Sakiyama, and Kazuo Ohta, “A Study on Computational Formal Verifi -cation for Practical Cryptographic Protocol: The Case of Synchronous RFID Authentication”, 2011

[21] Alexandre Melo Braga, Ricardo Hahab, “A Survey on Tools and Techniques for the Programming and Verification of Secure Crypto -graphic Software”, 2015

Page 69: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Reference

[22] Shinichiro Matsuo, Kunihiko Miyazaki, Akira Otsuka, David Basin, “How to Evaluate the Security of Real-life Cryptographic Protocol? The cases of ISO/IEC 29128 and CRYPTREC, 2010

[23] Bruno Blanchet, Ben Smyth, and Vincent Cheval, “ProVerif 1.94pl1: Automatic Cryptographic Protocol Verifier, User Manual and Tu -torial”, 2016

[24] Charles B. Weinstock, John B. Goodennough, “Toward an Assurance Case Practice for Medical Devices”, 2009

[25] CISCO, “Building Trustworthy Systems with Cisco Secure Development Lifecycle”, 2016

[26] Yannick Moy, Emmanuel Ledinot, Herve Delseny, Virginie Wiels, Benjamin Monte, “Testing or Formal Verification : DC-178C Alter -natives and Industrial Experience”, 2013

[27] Karen Scarfone, Murugiah Souppaya, Amanda Cody, Angela Orebaugh, “NIST SP 800-115, Technical Guide to Information Security Testing and Assessment – Recommandations of the National Institue of Standards and Technology”, 2008

[28] Steve Lipner, Microsoft, “The Security Development Lifecycle”, 2010

[29] Michael Felderer, Ruth Breu, Matthias Buchler, “Security Testing : A Survey”, 2016

[30] Vijay D’Silva, Daniel Kroening, George Weissenbacher, “A Survey of Automated Techniques for Formal Software Verification”

[31] John Rushby, Xidong Xu, Rangarajan and Thomas L. Weaver, “Understanding and Evaluating Assurance Case”, 2015

[32] David J.Rinehart, John C. Knight, Jonathan Rowanhill, “Current Practices in Constructing and Evaluating Assurance Case with Appli -cation to Aviation”, 2015

[33] The Government of Japan, “Cybersecurity Strategy 2015”

Page 70: [CB16] Using the CGC’s fully automated vulnerability detection tools in security evaluation and its effectiveness by InHyuk Seo & Jason Park

Reference

[34] Yasu Taniwaki, Deputy Director-General National Information Security Center, “Cybersecurity Strategy in Japan”, 2014

[35] “The NRL Protocol Analyzer : An Overview”, 1994

[36] Bruno Blanchet, “Automatic Verification of security protocols : the tools ProVerif and CryptoVerif”, 2011

[37] Tobias Nipkow, “Programming and Proving in Isabelle/HOL”, 2016

[38] Assistant Secretary of the Navy Chief System Engineer, “Software Security Assessment Tools Review”, 2009

[39] S.Santiago, C.Talcott, S.Escobar, C.Meadows, J.Meseguer, “A Graphical User Interface for Maude-NPA”, 2009

[40] NIST, "Source Code Security Analyzers"

[41] Cadar, Cristian, "KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs", 2008

[42] Cha, Sang Kil, "Unleashing MAYHEM on Binary Code", 2012

[43] Giovanni Vigna, "Autonomous Hacking: The New Frontiers of Attack and Defense", 2016

[44] Antonio Bianchi, "A Dozen Years of Shellphish From DEFCON to the Cyber Grand Challenge", 2015

[45] Jonathan Salwan, "Triton: Concolic Execution Framework", 2016

[46] Godefroid, "SAGE: Whitebox Fuzzing for Security Testing", 2012

[47] Michael Zalewski, "American Fuzzy Lop (http://lcamtuf.coredump.cx/afl/)", 2015

[48] Vegard Nossum, Oracle, "Filesystem Fuzzing with American Fuzzy Lop", 2016

[49] Hongzhe Li, "CLORIFI: software vulnerability discovery using code clone verification", 2015

[50] Stephens, "Driller: Augmenting Fuzzing Through Selective Symbolic Execution", 2016

[51] John Rushby, “The Interpretation and Evaluation of Assurance Cases”, SRI International Technical Report, 2015