1

Part 6: System Security

Malware Safe Coding Software Trust Virus Detectors Software Signatures “Kernel Integrity Checkers” “Application Integrity Checkers”

2

Computer Systems

Security – keep the systems safe No malware No applications that can be attacked No data stolen No corruption or harmful activity User accounts are not hacked ….. HOW?

Safe softwareVirus checksSandboxing

Integrity checkingCrypto File Systems

Smartcards

3

Safe Coding

Not well understood Software has vulnerabilities, ensure attack free code at

implementation time Stack Guard and Lib Safe and a host of techniques are available Type safe languages are available Not clear if these approaches work

Teach programmers to be aware of attack techniques on software

Safe coding practices

4

Bounds Checking

Array bounds often get exceeded and yet nothing happens Very common in C and C++ code Optimized compilers for all languages disable bounds checking Have to be checked in software, at all points

Buffer overflow of many kinds use this For every data transfer we need to check the size of the

data Function calls Data copying Reading input more

5

Input Validation

Input to a program has to be validated From network From files From user

Almost never done A program “expects” the data to be formatted correctly,

specially if that program generated the data Suppose WORD creates a DOC file When WORD reads it, the data is expected to be in WORD format Simple checks are NOT enough

This vulnerability is ingrained into all legacy software

6

Calling Routines

When routines are called, the caller and the callee must perform sanity checks on arguments

Specially when system calls and library routines are called Many stack and heap smashing attacks use such

vulnerabilities If a program has a call to the “system” routine in Unix If arguments are tampered with, any program can be run by an

attacker

7

Root Access Attacks

“set uid” programs can gain root access Used heavily for Unix Similar features in all operating systems

Many attacks have been designed to misuse this feature in many non-intuitive ways

Databases are particularly attackable via setuid attacks

8

Heap tricks

Put crafted data on heap Most input is read into the heap area anyway

Ensure that a particular pointer to an argument to another function call has been tampered with

Next time the function is called, it will be called with attacked arguments

Hard to detect and protect against

9

Open Source vs. Closed Source

What is safer – Open Source or Closed Source? Open Source:

Many eyes theory When found, vulnerabilities quickly fixed

Closed Source: Stringent code review and coding security policies may be in effect Even if vulnerabilities exist, they may not be found, as the code is

unknown Vendor may have liability

This debate has no obvious outcome, both have advantages and disadvantages

10

Data Execution

Many attacks inject executable code as data and then transfer control to it

Solution is to ensure data is never executed Code resides in code segment, data resides in data segment Code segment must be non-writable Data segment must be non-executable

Hardware support needed NX or XD bits in page tables Existed in the old VAX architecture

11

Software Issues

Software is inherently untrustable Even if source code is visible (e.g. Thompson Attack)

Many techniques for obfuscation Easy to make software do things other than what the source code

seems to do Programmers can inject things into programs that can be

harmful By design By accident

How do we know that software is to be trusted? OS Applications Drivers Downloaded things

No good answer

12

Feature Creep

Features of a program causes unforeseen vulnerabilities e.g. Visual Basic Scripting in Outlook

Why does an email reader have to be programmable? ….because it can be done!

Complex features of programs are attractive to designers and to end users

Impossible to analyze the security impacts Many tricks are then invented by attackers Old adage: Keep it simple

Code will be cleaner Features would be understandable Unforeseen consequences will be minimized

13

Malware

Virus – Trojan – Worm Adware, spyware Rootkits Can be

detectable, but only if detector in

operating system

Not detectable

14

Virus Detectors

Current state of the art in virus prevention is detection of viruses in all incoming files and existing files

Any file that a computer accesses is “scanned” by a virus detector

Hooks to the “open” and other system calls Makes system performance degrade

Scanning via hash matching Needs to know all the hashes of known viruses

Detects known viruses Does not detect changes to existing programs Have to frequently download “virus definitions”

15

Downside of Virus Detectors

Can be easily disabled (via buffer overflow, for example) Rootkits can uninstall it, or disable it

Cannot detect rootkits

Performance is slow New viruses can spread without fear for some time The virus detector is itself a virus?

Yes, but a good one No, but can be, via a little social engineering

Can store hashes of good files, but…. A virus can change the definitions and escape detection

16

Smart Viruses

Polymorphic Viruses Viruses that attack detectors Viruses that hide from detectors

Camouflaged Hidden names

17

Rootkits

Rootkits are kernel patches + processes Processes may run with Admin/root privileges Special routines are embedded in the kernel Many kernel routines may be changed or patched

How to detect Take checksum of kernel Take checksum of routines that may change Look at kernel tables and watch for changes Check for perturbations

18

Undetecatable

If a rootkit is not worried about rootkit detectors then it is detectable

Rootkits can be designed to be evasive If we take hashes of the kernel, it will patch the routine that

computes the hash If we track kernel tables, the tracking routine will be patched

A “new” rootkit detector will find an old rootkit As long as the rootkit writer does not know of the detector and its

techniques A rootkit can patch itself later to become invisible to new rootkit

detector Cat and Mouse game

19

Sandboxing

Run a program with some flags set in the kernel, designating it to be a sandboxed process

Every time this process calls a “system call” Check whether it is an allowable operation Check whether the arguments are “allowed” Interpret the call and ensure it falls under the “safe” category Policies have to be stated and enforced …. Loopholes?

20

Integrity Checking

Every executable file should have a pre-computed hash and a signature and an issuer certificate

Hash ensures any changes to the executable is detectable Signature ensures that no one updates the hash Issuer certificate ensures the signature is not faked This information is stored, separately from the file itself (hopefully

securely)

Integrity Checking Check hash and signature every time the file is executed Check hash occasionally while the program execution is in progress

(to detect dynamic corruption)

Problem: Who does the checking?

21

Application Integrity Checking

Assume: Operating System is not compromised OS does Application Integrity Checking

OS stores hashes and signatures Unsigned application either not allowed, or signed by OS Dynamic checking is also done

Ensures freedom from downloaded viruses Ensures some freedom from buffer overflow attacks

Cannot be completely detected, but the harm caused is limited

Runs only “trusted applications” Either signed applications or authorized by the user

Help prevention of undetected downloads, spyware and many such malware

22

Kernel Integrity Checking

How to check the OS for rootkits? Use a hardware device

TCG “Trusted Computing Group” approach “Co-Pilot” approach

Use a software device The VMM “Virtual Machine Manager” approach

Concept: A trusted device has keys and signatures for the operating system

components This device can access the OS and check it for integrity Possible to detect rootkits, as long as the checker is not

compromised

23

Hardware Checking

TPM (trusted platform module) from TCG A hardware device, that contains keys and hashes for checking

integrity of operating systems Attestation of software and digital signatures possible An elaborate framework of security modules, protocols and

monitoring of applications Can be used to enforce software licenses

Co-Pilot Project Hardware on the PCI bus, can take over the computer Performs memory checks on the OS and interrupt routines Can detect rootkits, even dynamic ones

24

Software Checking

Run the Operating System as a host on a Virtual Machine Manager

The VMM is hard to compromise Small, hence free of vulnerabilities (assumption) OS has to be rootkitted to attack VMM, but the VMM can detect

such rootkits

Check OS integrity from the VMM (Terra Project) Check OS integrity from a separate host operating system

called the Security Monitor ASU project

Work in progress, not tested well yet

25

Secure I/O

All security solutions outlined so far suffers from a spoofing vulnerability

A piece of software can be flagged to be “trusted” by faking human input by a program (virus)

TO ensure spoof freedom there is a need for a secure I/O channel from the core security system to the human user

TCG-TPM, Co-Pilot, Terra --- all are vulnerable

Secure I/O needs separate hardware channel, separate keypad

An SSL connection to a small communicator may be a solution.

26

Crypto File Systems

All files are stored encrypted Where is the key?

Derived from a passphrase Everything is “weak” as along the computer is “running”

MS-CFS uses PKI

Advantages Lost/stolen disks are secure Proper implementations may disallow some viruses from accessing

some files Diadvantrages

Files are decrypted on the fly and CFS is transparent to users and hance viruses

Attacking the key is possible

27

Current Status

Integrity checking is almost non-existent in deployed software

Hence, all systems are soft against viral attacks Big security problem, Device driver signing is used in Windows, quite ineffective

The TCG system is the closest to deployment Also the weakest system Has political implications

No other solutions are being deployed may not be entirely true