Building Security InEditor: Gary McGraw, gem@cigital.com

testing done properly goes deeperthan simple black-box probing on thepresentation layer (the sort performedby so-called application securitytools)—and even beyond the func-tional testing of security apparatus.

Testers must use a risk-based ap-proach, grounded in both the sys-tem’s architectural reality and the at-tacker’s mindset, to gauge softwaresecurity adequately. By identifyingrisks in the system and creating testsdriven by those risks, a software se-curity tester can properly focus onareas of code in which an attack islikely to succeed. This approach pro-vides a higher level of software secu-rity assurance than possible withclassical black-box testing.

What’s so differentabout security?Software security is about makingsoftware behave in the presence of amalicious attack, even though in thereal world, software failures usuallyhappen spontaneously—that is,without intentional mischief. Notsurprisingly, standard software test-ing literature is only concerned withwhat happens when software fails,regardless of intent. The differencebetween software safety and softwaresecurity is therefore the presence ofan intelligent adversary bent onbreaking the system.

Security is always relative to theinformation and services being pro-tected, the skills and resources of ad-versaries, and the costs of potentialassurance remedies; security is an ex-ercise in risk management. Riskanalysis, especially at the design level,can help us identify potential secu-rity problems and their impact.1

Once identified and ranked, soft-ware risks can then help guide soft-ware security testing.

A vulnerability is an error that anattacker can exploit. Many types ofvulnerabilities exist, and computersecurity researchers have created tax-onomies of them.2 Security vulnera-bilities in software systems rangefrom local implementation errors(such as use of the gets() functioncall in C/C++), through interpro-cedural interface errors (such as arace condition between an accesscontrol check and a file operation),to much higher design-level mis-takes (such as error handling and re-covery systems that fail in an insecurefashion or object-sharing systemsthat mistakenly include transitivetrust issues). Vulnerabilities typicallyfall into two categories—bugs at theimplementation level and flaws atthe design level.3

Attackers generally don’t carewhether a vulnerability is due to aflaw or a bug, although bugs tend to

be easier to exploit. Because attacksare now becoming more sophisti-cated, the notion of which vulnera-bilities actually matter is changing.Although timing attacks, includingthe well-known race condition,were considered exotic just a fewyears ago, they’re common now.Similarly, two-stage buffer overflowattacks using trampolines were oncethe domain of software scientists, butnow appear in 0day exploits.4

Design-level vulnerabilities arethe hardest defect category to han-dle, but they’re also the most preva-lent and critical. Unfortunately, as-certaining whether a program hasdesign-level vulnerabilities requiresgreat expertise, which makes findingsuch flaws not only difficult, but par-ticularly hard to automate.

Examples of design-level prob-lems include error handling in ob-ject-oriented systems, object sharingand trust issues, unprotected datachannels (both internal and exter-nal), incorrect or missing access con-trol mechanisms, lack of auditing/logging or incorrect logging, and or-dering and timing errors (especiallyin multithreaded systems). Thesesorts of flaws almost always lead tosecurity risk.

Risk managementand security testingSoftware security practitioners per-form many different tasks to managesoftware security risks, including

• creating security abuse/misusecases;

• listing normative security re-quirements;

• performing architectural riskanalysis;

BRUCE POTTER

Booz AllenHamilton

GARY

MCGRAW

Cigital

Security testing has recently moved beyond the

realm of network port scanning to include probing

software behavior as a critical aspect of system be-

havior (see the sidebar). Unfortunately, testing

software security is a commonly misunderstood task. Security

Software Security Testing

32 PUBLISHED BY THE IEEE COMPUTER SOCIETY ■ 1540-7993/04/$20.00 © 2004 IEEE ■ IEEE SECURITY & PRIVACY

Building Security In

• building risk-based security testplans;

• wielding static analysis tools;• performing security tests;• performing penetration testing in

the final environment; and• cleaning up after security breaches.

Three of these are particularlyclosely linked—architectural riskanalysis, risk-based security test plan-ning, and security testing—because acritical aspect of security testing relieson probing security risks. Last issue’sinstallment1 explained how to ap-proach a software security risk analy-sis, the end product being a set of se-curity-related risks ranked by businessor mission impact. (Figure 1 showswhere we are in our series of articlesabout software security’s place in thesoftware development life cycle.)

The pithy aphorism, “software se-curity is not security software” pro-vides an important motivator for secu-rity testing. Although security featuressuch as cryptography, strong authenti-cation, and access control play a criti-cal role in software security, security it-self is an emergent property of theentire system, not just the securitymechanisms and features. A bufferoverflow is a security problem regard-less of whether it exists in a securityfeature or in the noncritical GUI.Thus, security testing must necessarilyinvolve two diverse approaches:

1. testing security mechanisms toensure that their functionality isproperly implemented, and

2. performing risk-based securitytesting motivated by under-standing and simulating the at-tacker’s approach.

Many developers erroneouslybelieve that security involves onlythe addition and use of various secu-rity features, which leads to the in-correct belief that “adding SSL” istantamount to securing an applica-tion. Software security practitionersbemoan the over-reliance on “magiccrypto fairy dust” as a reaction to this

problem. Software testers chargedwith security testing often fall prey tothe same thinking.

How to approachsecurity testingLike any other form of testing, secu-rity testing involves determiningwho should do it and what activitiesthey should undertake.

WhoBecause security testing involves twoapproaches, the question of whoshould do it has two answers. Stan-dard testing organizations using atraditional approach can performfunctional security testing. For ex-ample, ensuring that access controlmechanisms work as advertised is aclassic functional testing exercise.

On the other hand, traditionalQA staff will have more difficultyperforming risk-based security test-ing. The problem is one of expertise.First, security tests (especially thoseresulting in complete exploit) aredifficult to craft because the designermust think like an attacker. Second,security tests don’t often cause directsecurity exploit and thus present anobservability problem. A securitytest could result in an unanticipatedoutcome that requires the tester toperform further sophisticated analy-sis. Bottom line: risk-based securitytesting relies more on expertise andexperience than we would like.

HowBooks like How to Break Software Secu-

rity and Exploiting Software help edu-cate testing professionals on how tothink like an attacker.4,5 Nevertheless,software exploits are surprisingly so-phisticated these days, and the level ofdiscourse found in books and articlesis only now coming into alignment.

White- and black-box testing andanalysis methods both attempt to un-derstand software, but they use differ-ent approaches depending onwhether the analyst or tester has ac-cess to source code. White-boxanalysis involves analyzing and under-standing source code and the design.It’s typically very effective in findingprogramming errors (bugs when au-tomatically scanning code and flawswhen doing risk analysis); in somecases, this approach amounts to pat-tern matching and can even be auto-mated with a static analyzer (the sub-ject of a future installment of thisdepartment). One drawback to thiskind of testing is that it might report apotential vulnerability where noneactually exists (a false positive). Nev-ertheless, using static analysis methodson source code is a good techniquefor analyzing certain kinds of soft-ware. Similarly, risk analysis is a white-box approach based on a deep under-standing of software architecture.

Black-box analysis refers to ana-lyzing a running program by probingit with various inputs. This kind oftesting requires only a running pro-gram and doesn’t use source-codeanalysis of any kind. In the securityparadigm, malicious input can besupplied to the program in an effort

www.computer.org/security/ ■ IEEE SECURITY & PRIVACY 33

Abusecases

Securityrequirements

Riskanalysis

Externalreview

Risk-basedsecurity tests

Staticanalysis(tools)

Riskanalysis

Penetrationtesting

Securitybreaks

Requirementsand use cases

Design Testplans

Code Testresults

Fieldfeedback

Figure 1. The software development life cycle. Throughout this series, we’ll focus onspecific parts of the cycle; here, we’re examining risk-based security testing.

Building Security In

to break it: if the program breaksduring a particular test, then wemight have discovered a securityproblem. Black-box testing is possi-ble even without access to binarycode—that is, a program can betested remotely over a network. Ifthe tester can supply the proper input(and observe the test’s effect), thenblack-box testing is possible.

Any testing method can revealpossible software risks and potentialexploits. One problem with almostall kinds of security testing (regard-less of whether it’s black or whitebox) is the lack of it—most QA or-ganizations focus on features and

spend very little time understandingor probing nonfunctional securityrisks. Exacerbating the problem, theQA process is often broken in manycommercial software houses due totime and budget constraints and thebelief that QA is not an essential partof software development.

An example: JavaCard security testingDoing effective security testing re-quires experience and knowledge.Examples and case studies like theone we present now are thus usefultools for understanding the approach.

In an effort to enhance payment

cards with new functionality—suchas the ability to provide secure card-holder identification or rememberpersonal preferences—many credit-card companies are turning to multi-application smart cards. These cardsuse resident software applications toprocess and store thousands of timesmore information than traditionalmagnetic-stripe cards.

Security and fraud issues are criti-cal concerns for the financial institu-tions and merchants spearheadingsmart-card adoption. By developingand deploying smart-card technol-ogy, credit-card companies provideimportant new tools in the effort tolower fraud and abuse. For instance,smart cards typically use a sophisti-cated crypto system to authenticatetransactions and verify the identitiesof the cardholder and issuing bank.However, protecting against fraudand maintaining security and privacyare both very complex problems be-cause of the rapidly evolving natureof smart-card technology.

The security community hasbeen involved in security risk analy-sis and mitigation for Open Platform(now known as Global Platform, orGP) and Java Card since early 1997.Because product security is an essen-tial aspect of credit-card companies’brand protection regimen, thesecompanies spend plenty of time andeffort on security testing. One cen-tral finding emphasizes the impor-tance of testing particular vendorimplementations according to ourtwo testing categories: adherence tofunctional security design andproper behavior under particular at-tacks motivated by security risks.

The latter category, risk-basedsecurity testing (linked directly torisk analysis findings) ensures thatcards can perform securely in thefield even when under attack. Riskanalysis results can be used to guidemanual security testing. As an exam-ple, consider the risk that, as de-signed, the object-sharing mecha-nism in Java Card is complex andthus is likely to suffer from security-

34 IEEE SECURITY & PRIVACY ■ SEPTEMBER/OCTOBER 2004

From outside→in to inside→out

Traditional approaches to computer and network security testing focus on network infra-

structure, firewalls, and port scanning. The notion is to protect vulnerable systems (and

software) from attack by identifying and defending a perimeter. In this paradigm, testing

focuses on an outside→in approach. One classic example is the use of port scanning with tools

such as nmap <http://www.insecure.org/nmap/> to probe network ports and see

what service is listening. Figure A shows a classic outside→in paradigm focusing on firewall

placement.

By contrast, we advocate an inside→out approach to security, whereby software inside the

LAN (and exposed on LAN boundaries) is itself subjected to rigorous risk management and

security testing.

Internet

Portable local area network

Publicnetwork

Hardware, firewall,usually part of a TCP/IP router

Secure private networkPublic network

Figure A: The outside→in approach. A firewall protects a LAN by blocking variousnetwork traffic on its way in; outside→in security testing involves probing the LANwith a port scanner to see which ports are “open” and what services are listeningon those ports. A major security risk associated with this approach is that theservices traditionally still available through the firewall are implemented withinsecure software.

Building Security In

www.computer.org/security/ ■ IEEE SECURITY & PRIVACY 35

critical implementation errors onany given card. Testing for this sort ofrisk involves creating and manipulat-ing stored objects where sharing isinvolved. Given a technical descrip-tion of this risk, building specificprobing tests is possible.

Automating security testingOver the years, we (the authors) havebeen involved in several projects thathave identified architectural risks inthe GP/Java Card platform, sug-gested several design improvements,and designed and built automatedsecurity tests for final products (eachof which had multiple vendors).

Several years ago, Cigital begandeveloping an automated securitytest framework for GP cards built onJava Card 2.1.1 and based on exten-sive risk analysis results. The end re-sult is a sophisticated test frameworkthat runs with minimal human inter-vention and results in a qualitative se-curity testing analysis of a samplesmart card.

The first test set, the functionalsecurity test suite, directly probeslow-level card security functionality.It includes automated testing of classcodes, available commands, andcrypto functionality. This test suitealso actively probes for inappropriatecard behavior of the sort that can leadto security compromise.

The second test set, the hostileapplet test suite, is a sophisticatedset of intentionally hostile JavaCard applets designed to probehigh-risk aspects of the GP on aJava Card implementation.

Results: Nonfunctional securitytesting is essentialMost cards tested with the automatedtest framework pass all functional se-curity tests, which we expect becausesmart-card vendors are diligent withfunctional testing (including securityfunctionality). Because smart cardsare complex embedded devices, ven-dors realize that exactly meeting

functional requirements is an ab-solute necessity for customers to ac-cept the cards. After all, they mustperform properly worldwide.

However, every card submitted tothe risk-based testing paradigm ex-hibited some manner of failure whentested with the hostile applet suite.Some failures pointed directly to crit-ical security vulnerabilities on thecard; others were less specific and re-quired further exploration to deter-mine the card’s true security posture.

As an example, consider that riskanalysis of Java Card’s design docu-ments indicates that proper imple-mentation of atomic transactionprocessing is critical for maintaininga secure card. Java Card has the capa-bility of defining transaction bound-aries to ensure that if a transactionfails, data roll back to a pre-transac-tion state. In the event that transac-tion processing fails, transactions cango into any number of possible states,depending on what the applet wasattempting. In the case of a stored-value card, bad transaction process-ing could allow an attacker to “printmoney” by forcing the card to rollback value counters while actuallypurchasing goods or services.

When creating risk-based tests toprobe transaction processing, we di-rectly exercised transaction-process-ing error handling by simulating anattacker attempting to violate atransaction—specifically, transac-tions were aborted or never commit-ted, transaction buffers were com-pletely filled, and transactions werenested (a no-no according to the JavaCard specification). These tests werenot based strictly on the card’s func-tionality—instead, security test en-gineers intentionally created them,thinking like an attacker given theresults of a risk analysis.

Several real-world cards failedsubsets of the transaction tests. Thevulnerabilities discovered as a resultof these tests would let an attackerterminate a transaction in a poten-tially advantageous manner, a criticaltest failure that wouldn’t have been

uncovered under normal functionalsecurity testing. Fielding cards withthese vulnerabilities would let an at-tacker execute successful attacks onlive cards issued to the public. Be-cause of proper risk-based securitytesting, the vendors were notified ofthe problems and corrected the coderesponsible before release.

T here is no silver bullet for soft-ware security; even a reasonable

security testing regimen is just a start.Unfortunately security continues tobe sold as a product, and most defen-sive mechanisms on the market dolittle to address the heart of the prob-lem, which is bad software. Instead,they operate in a reactive mode:don’t allow packets to this or thatport, watch out for files that includethis pattern in them, throw partialpackets and oversized packets awaywithout looking at them. Networktraffic is not the best way to approachthis predicament, because the soft-ware that processes the packets is theproblem. By using a risk-based ap-proach to software security testing,testing professionals can help solvesecurity problems while software isstill in production.

References1. D. Verndon and G. McGraw, “Risk

Analysis in Software Design,” IEEESecurity & Privacy, vol. 2, no. 4,2004, pp. 79–84.

2. C.E. Landwehr et al., A Taxonomyof Computer Program Security Flaws,with Examples, tech. report NRL/FR/5542—93/9591, NavalResearch Laboratory, Nov. 1993.

3. G. McGraw, “Software Security,”IEEE Security & Privacy, vol. 2, no.2, 2004, pp. 80–83.

4. G. Hoglund and G. McGraw,Exploiting Software, Addison-Wes-ley, 2004.

5. J. Whittaker and H. Thompson,How to Break Software Security,Addison-Wesley, 2003.

Bruce Potter is a senior associate with

Building Security In

Booz Allen Hamilton. He is also thefounder of the Shmoo Group of securityprofessionals. His areas of expertiseinclude wireless security, large-scale net-work architectures, smartcards, and pro-motion of secure software engineeringpractices. Potter coauthored the books802.11 Security (O’Reilly and Associates,2003) and Mac OS X Security (New Rid-ers, 2003); he’s currently coauthoringMaster FreeBSD and OpenBSD Security(O’Reilly and Associates, summer 2004).He was trained in computer science at theUniversity of Alaska, Fairbanks.

Gary McGraw is chief technology officerof Cigital. His real-world experience isgrounded in years of consulting withmajor corporations and software pro-ducers. He serves on the technical advi-sory boards of Authentica, Counterpane,Fortify, and Indigo. He also is coauthor ofExploiting Software (Addison-Wesley,2004), Building Secure Software (Addi-son-Wesley, 2001), Java Security (JohnWiley & Sons, 1996), and four otherbooks. He has a BA in philosophy fromthe University of Virginia and a dual PhDin computer science and cognitive sciencefrom Indiana University. Contact him atgem@cigital.com.

36 IEEE SECURITY & PRIVACY ■ SEPTEMBER/OCTOBER 2004