24 CROSSTALK The Journal of Defense Software Engineering March/April 2010

Application development and securitypractices are often at odds. Appli-

cation development is concerned withcreating software quickly with the mostfeatures possible in the minimum amountof time. Application security is con-cerned with finding and removing securi-ty vulnerabilities and releasing softwarewhen critical security risks have been mit-igated.

Many project stakeholders see appli-cation security practices as an increase inscope that adversely impacts the softwaredelivery schedule. In order to buildsecure applications, it is important toalign application development and securi-ty practices. Our analysis has found thatone of the best ways to do that is to inte-grate secure code analysis and securitytesting into CI. CI is a software develop-ment practice where members of a teamintegrate their work frequently, verifyingeach integration by an automatedbuild/test process to detect integrationerrors as quickly as possible [1]. Using CI,the time and effort to build security intothe development process can be mini-mized, making teams more likely toinclude security practices in their soft-ware development process and therebyreducing the risk of a successful attack.

Immediate NotificationCI ensures that ongoing changes to thesource code do not break the intent ordesign of the software. If a change doesbreak the software, that break is identi-fied immediately and can be fixed with aminimal cost and impact to the project’sschedule.

CI started with the notion that each CIcycle should make a clean build from an up-to-date checkout from the source coderepository, and that a set of unit testsshould be run against the clean build as aregression against the changes in the codebase. If the build and unit tests pass, thenthe recent checked-in changes did notbreak the software. If the build or unittests fail, then the changes broke the soft-ware, and the CI server immediately noti-fies the team that the software is broken.

Secure DevelopmentCI is now the foundation for serving manycrucial software development processtasks. Originally focused on compiling andunit testing, CI practices have grown andevolved over time. They now includeexpanded practices, such as functional test-ing and code analysis to evaluate the healthof a project. By integrating security testingand secure code analysis, CI can be furtherleveraged to include secure developmentpractices while minimizing the amount ofextra effort required to get the benefits ofsecure development. Since it is tied to CI,security testing and secure code reviewbegins when a project begins and runs con-tinuously throughout project development.With CI, security vulnerabilities testingbecomes part of the regression test bed,executed automatically with each successivebuild on the CI platform.

Changing Testing Economics Technology advances have changed theeconomics of testing, allowing moreaggressive approaches to testing than his-torically possible. Using CI for build, test,and analysis automation has increased thedepth and breadth of tests while also mak-ing them faster and less expensive. By mak-ing it cheap and easy to perform tests,teams are encouraged to test more and testsooner in the development cycle, reducingthe cost of fixing bugs. It has also made iteasier for managers and non-technicalstakeholders to understand projectprogress and health.

For many projects, testing has gonefrom a process that slowed deploymentdown to one that provides true qualityassurance, in turn helping stakeholdershave more confidence in their projects. Byreducing the cost of many quality controlaspects of application development, teamshave been able to use those controls moreoften and more effectively and have accel-erated development while improving quali-ty. This same change can be applied tosecurity practices integrated into CI. Thereduced cost of gathering security vulnera-bility data will encourage teams to collectthe data more often and sooner in their

development cycle, reducing the cost to fixissues such as cross-site scripting and SQLinjection.

Building Security TestingInto CI In order to integrate static code analysisand security testing into CI, a few keypieces of software are needed. Organiza-tions should select a specific application foreach of the software categories (as shownin Table 1), bearing in mind that someproducts might fit into more than one cat-egory. For example, Microsoft’s TeamFoundation Server (TFS) [2], fits into theCI Server, Source Code Repository, andIssue Tracking categories.

Choosing the Right ToolsIt is possible to build a completely opensource CI process that includes static codeanalysis and security testing. SecureCI isone example of a CI product that is madeof all open source tools and can be down-loaded and used free of charge [3]. Thereare also many good commercial productsthat, in some cases, provide more valuethan their open source alternatives, such asthe Web crawling capability of AppScanand WebInspect.

There are many tools to choose from,ranging from unsupported to company-supported open source software to com-mercial software that is developed andmaintained completely by the companythat created it. In other words, an organiza-tion can purchase support contracts tocommercial software—products that arethen developed and maintained completelyby the company that created them. Thereare even some commercial software prod-ucts that are free to use but come withsome restrictions, such as a limited numberof users. There is no correct or incorrecttool as long as the acquired software workstogether and can meet organizationalobjectives.

Tool IntegrationIn general, the CI server will be the inte-gration hub and orchestrator for a CIprocess. CI servers come with integration

Building Security In Using Continuous Integration

Building security into software is harder than it should be. This article explores a way to align application security practiceswith other software development best practices in order to make building security in easier to manage and more cost effective.In particular, this article looks at combining continuous integration (CI) with security testing and secure static code analysis.

Thomas Stiehm and Gene GotimerCoveros

Building Security In Using Continuous Integration

March/April 2010 www.stsc.hill.af.mil 25

application programming interfaces (APIs)for many of the tools an organization putsin its toolset so that working with thosetools will be easy. If there isn’t an API forone of the selected tools, CI servers can beextended using scripting languages andcompiled code plug-ins. This flexibilitymeans that an organization can add anytools that can be scripted or programmati-cally controlled into their CI process, mak-ing it possible to add security practices toan existing CI process without having toreinvent that process. For example, withHudson, in order to use PMD [4] orFindBugs [5] for static code analysis, allthat is needed is creation of a build job thatuses an Ant [6] build to run the tool. Youthen point the tool’s Hudson [7] plug-in tothe XML reports created during the Antbuild. The plug-in picks up the reports andparses them for display using HTML andgenerated graphics.

The CI server market is full of bothopen source and commercial products.Many of the open source CI servers havecommercial support contracts available.Because of the mature CI server market,there is no advantage in using either opensource or commercial products. The excep-tion to this statement is when an organiza-tion is considering development ecosystemsuites like Microsoft’s TFS and VisualStudio suite.

Integrating security testing toolsrequires a little more work. The applicationunder test needs to be deployed and run-ning because security testing tools work byinteracting with the application, analyzingthe requests and responses from the pointof view of a Web browser. This means thatthe CI server will have to deploy to anapplication server, start it, and then kick offthe security testing tool. While it seems likea lot of work, there are a number of goodexamples on the Internet to help get start-ed. The commercial security testing toolscan be configured to just launch and crawlan application.

Creating MultipleComplementary Builds toSupport Specific NeedsOne of the first things organizations willnotice about using static code analysis andautomated security testing tools is thatbuild times—the time it takes to gothrough a CI process—will increase signif-icantly. Because of this increase, it is com-mon for projects to use multiple CI jobs.The different jobs are set to run on differ-ent intervals and for different reasons. Forinstance, most teams have a quick job thatruns within 10 minutes of a check-in and

produces a result within 10 minutes. Thisquick job consists of a clean build and onlyexecutes unit tests. This quick build tells thedevelopers that the new code works andthat all of the unit tests pass (i.e., the codehasn’t introduced defects into previouslyworking code). Many teams will also uselonger running tests (a couple of hours),compiling the project and running the unittests, while also executing other kinds oftests (e.g., database tests and automatedfunctional tests). These tests take longer tocomplete and have a longer feedback cycle.By executing multiple jobs, an organizationcan provide the team with feedback asquickly as possible for a given type of feed-back. Finally, many projects have jobs run-ning from once a day to once a week thatperform static analysis or security testing.This allows the processes to run at theirown pace without slowing down other testprocesses.

The selection of a source code reposi-tory is generally based on what alreadyexists in an organization’s environment.From the overall goal of building securityinto applications, the choice of sourcerepository makes little difference—it justneeds to work with the CI server.

Utilizing Both Commercialand Open Source ToolsThere is a healthy marketplace for issuetracking applications having both opensource and commercial products.

Commercial issue tracking software hasan advantage over open source in terms of

the reporting and integration options.Commercial applications tend to have abetter and more customizable reportingsystem and tend to integrate more with(usually commercial) software that mightbe used in an overall development process.That said, many projects and organizationsdon’t need or use the extra capability ofcommercial issue tracking software. Forsmall project teams or small companies,open source issue tracking software worksjust fine. For large enterprises with multiplerelated development projects, a commercialissue tracking application may offer neededfeatures that can justify the cost of acquir-ing the software.

Open source unit testing tools are usu-ally frameworks that provide a core unittesting capability. These tools require adeveloper to write and maintain unit tests.If the developer follows the conventions ofthe framework, running unit tests is verysimple and easy and many CI servers canread and report on the results. The com-mercial products in the space build on topof the open source tools by adding thecapability to generate unit tests. The com-mercial tools will scan the source code anddetermine how to exercise the code paths(in the source code) in order to get 100 per-cent test coverage and possibly add nega-tive unit tests.

We have worked on projects that haveused both developer-written unit tests andtool-generated unit tests. When dealingwith security features, it is important towrite comprehensive unit tests that exerciseboth the positive and negative paths

SoftwareCategory

Open SourceToolsDescription

CommercialTools

CI Server Server software that monitors thesource code repository and runsthe build when changes to therepository are detected.

Hudson,CruiseControl

TeamCity,Bamboo,TFS

Source CodeRepository

Software that keeps source code,maintaining versions of the source filesand file groups (i.e., labels and tags).

ConcurrentVersionsSystems,Subversion

PolytronVersion ControlSystem,Clearview,TFS

Issue Tracking Software that is used to manage softwareissues and report their status.

Trac andSonar,Bugzilla

QualityCenter,TFS

Unit Testing Tools or frameworks used by developersto test at a source code level.

JUnit,NUnit

JTest

Functional Testing Tools, frameworks, or applicationsused to test the functionality of software.

Selenium,Watir

Quick TestProfessional,SilkTest

Security Testing Tools, frameworks, or applications usedto test the security aspects of software(i.e., penetration testing tools).

RatProxy,WebScarab

AppScan,WebInspect

Static CodeAnalysis

Tools, frameworks, or applications usedto inspect either source code or compiledfiles for known issues.

FindBugs,PMD,CheckStyle

FortifySource CodeAnalyzer

Table 1: Tools for Different Software Categories

Systems Assurance: Preparation and Promise

26 CROSSTALK The Journal of Defense Software Engineering March/April 2010

through the code. Many security defectscome from security features that don’t havea proper fail-safe method for when there areprogram or data processing errors.Negative testing—or the testing for failurepaths in the source code—addresses thecorrectness of failure states and can showthat a specific security feature can fail-safely when something unexpected hap-pens. If there is a large legacy code basethat doesn’t have unit tests, using a tool togenerate unit tests is a good way to addtests quickly. Keep in mind, however, thatthe project team still needs to review all ofthose tests, change some, remove others,and write new tests covering situationsthat the tools couldn’t.

Security Testing ToolsSecurity testing is an excellent way to dis-cover many of the security vulnerabilitiesin an application. These tools are runagainst an application, usually in a testingenvironment, in its production configura-tion. One goal of this testing is to deter-mine if the application has defects, mak-ing it vulnerable to outside attack while inproduction; another is to see if the appli-cation will fail safely when attacked.Failing safe has different meanings to dif-ferent applications, with the basic guide-line that an application should not giveusers unauthorized information or allowthem to take unauthorized actions.

In order to automate many opensource security-testing tools, they willneed to be used in conjunction with func-tional test tools. For example Ratproxy [8],a popular open source security-testingtool, can’t (unlike popular commercialsecurity testing tools) crawl a Web applica-tion. This means that people usingRatproxy need to use another tool to crawlthe Web application while Ratproxy isrunning. Another alternative is to have thetesters use Ratproxy while they are con-ducting manual functional tests.

Another area where commercial secu-rity testing tools have an advantage is inreporting. Commercial security testingtools, like AppScan [9], have a customiz-able reporting capability that gives usersnumerous ways to report their findings (inorder to conform to organizational stan-

dards or highlight different aspects of thefindings for different audiences). Thereporting capability also does a good jobof explaining the findings—that is, howthey can be exploited and remediated. Incontrast, Ratproxy has only one reportformat that provides little detailed infor-mation about an issue beyond its nameand where it was found. An analyst usingRatproxy has to figure out what the issuemeans and how to remediate it. Findingout this information isn’t hard, but it canbe time-consuming.

Static Code Analysis Tools Static code analysis tools examine the codebase for many problems including securityand code style issues, potential codedefects, and race conditions. These toolsare more akin to automated code reviewthan to unit and integration tests. The dif-ference between commercial and staticcode analysis tools mirror the differencesfound in security testing tools: Commercialtools have better reporting capabilities andprovide more information on the nature ofthe findings and suggested remediation.Another area where some commercialtools have added value is in the depth ofanalysis. Fortify Software’s Source CodeAnalyzer [10] creates an entire model ofthe application under analysis, examiningthe data and control flow of the softwareto determine if there is a larger-contextproblem. Most open source static codeanalysis tools only look for problems in alocal context (i.e., the immediate line ofcode, code block, or file).

Static code analysis can either be per-formed on the source code of an applica-tion or on the compiled binaries. For exam-ple, FindBugs runs against compiled Javaclass files. It can find bad practices, nullpointer dereferences, static use of non-thread safe code, and security issues. PMDruns against Java source files. It can finddead code, performance issues, style issues,and potentially dangerous code practices.While there is overlap with FindBugs, usingboth is not considered redundant.

Data Analysis and EvaluationOnce the vulnerability data has been col-lected from both security testing and code

analysis, it has to be analyzed and evaluat-ed. The goal of this step is to decide if thefindings are truly a problem and what todo about it. This step includes determiningthe priority and severity of the securityissues and putting them into an issue track-ing system. In order for application securi-ty practices to positively affect the project,issues uncovered in testing and analysisneed to be tracked and fixed. By integrat-ing security practices into CI, securityissues are discovered and dealt with morequickly, in turn preventing many securityrisks from entering production and mini-mizing the possibility of exploitation.

Building application security practicesinto a project can be done with minimumimpact to a project’s budget, schedule, andresources. Integrating security testing andsecure code analysis into CI is the firststep to building security into software.These practices help build security aware-ness by showing developers how and whytheir code is vulnerable. They give testersthe tools needed to find many security vul-nerabilities, and project managers a way todemonstrate the results of security prac-tices. While it would be instructive to pro-vide quantitative analysis of the benefitsof integration CI and security, the practiceis still new enough that no major studieshave been conducted.

Application security practices are hardfor many teams to adopt because theydon’t have the time, budget, or resourcesneeded. CI can help change the econom-ics of security testing and analysis by giv-ing project teams tools that can bedeployed in their environment. Thesesolutions can enable a team to quickly gofrom not considering security to a solidinitial step in finding and proactively fixingsecurity vulnerabilities.u

References 1. Duvall, Paul, and Steve Matyas. Contin-

uous Integration: Improving Software Qualityand Reducing Risk. New York: Addison-Wesley, 2007.

2. Microsoft. “Team Foundation ServerHome.” Team Foundation Server Home.2009 <http://msdn.microsoft.com/en-us/teamsystem/dd408382.aspx>.

3. Coveros. Coveros: Research & Insights –Free Secure CI Download. Fairfax, VA:Coveros, 2009.

4. InfoEther. “PMD” SourceForge.net.2009 <http://pmd.sourceforge.net>.

5. The University of Maryland. “Find-Bugs – Find Bugs in Java Programs.”SourceForge.net. 21 Aug. 2009 <http://findbugs.sourceforge.net>.

6. The Apache Ant Project. “Apache Ant– Welcome.” Apache Ant. 15 Oct. 2009

Application security is a priority for DoD applications, given the always on, globalnature of the DoD mission. At the same time, it is important to make applicationsecurity work within the development practices in common use among DoD devel-opment teams. By integrating application security practices with CI, we can addressboth the security needs of the applications as well as the efficiency and cost-effectiverequirements of the development teams.

Software Defense Application

Building Security In Using Continuous Integration

March/April 2010 www.stsc.hill.af.mil 27

About the Authors

Thomas Stiehm hasbeen developing applica-tions and managing thesoftware developmentteams for 16 years. AsChief Technical Officer

of Coveros, he is responsible for theoversight of all technical projects andintegrating application security practicesinto software development projects.

Coveros 4000 Legato RDSTE 1100Fairfax,VA 22033Phone: (703) 599-6243E-mail: tom.stiehm@coveros.com

Gene Gotimer has beenbuilding Web applicationsand working with securityfor the last 13 years. Hespecializes in CI andAgile Java development.

Coveros 4000 Legato RDSTE 1100Fairfax,VA 22033Phone: (703) 963-1620E-mail: gene.gotimer

@coveros.com

Get Your Free Subscription

Fill out and send us this form.

517 SMXS/MXDEA

6022 Fir Ave

Bldg 1238

Hill AFB, UT 84056-5820

Fax: (801) 777-8069 DSN: 777-8069

Phone: (801) 775-5555 DSN: 775-5555

Or request online at www.stsc.hill.af.mil

NAME:________________________________________________________________________

RANK/GRADE:_____________________________________________________

POSITION/TITLE:__________________________________________________

ORGANIZATION:_____________________________________________________

ADDRESS:________________________________________________________________

________________________________________________________________

BASE/CITY:____________________________________________________________

STATE:___________________________ZIP:___________________________________

PHONE:(_____)_______________________________________________________

ELECTRONIC COPY ONLY? YES NO

E-MAIL:__________________________________________________________________

CHECK BOX(ES) TO REQUEST BACK ISSUES:

MAY2008 o LEAN PRINCIPLES

SEPT2008 o APPLICATION SECURITY

OCT2008 o FAULT-TOLERANT SYSTEMS

NOV2008 o INTEROPERABILITY

DEC2008 o DATA AND DATA MGMT.

JAN2009 o ENG. FOR PRODUCTION

FEB2009 o SW AND SYS INTEGRATION

MAR/APR09 o REIN. GOOD PRACTICES

MAY/JUNE09 o RAPID & RELIABLE DEV.

JULY/AUG09o PROCESS REPLICATION

NOV/DEC09 o 21ST CENTURY DEFENSE

JAN/FEB09 o CMMI: PROCESS

To request back issues on topics notlisted above, please contact <stsc.customerservice@hill.af.mil> .

<http://ant.apache.org>.7. Sun Microsystems. “Hudson CI”.

Hudson: Extensible continuous integrationserver. 2009 <http://hudson-ci.org>.

8. Google. “Ratproxy: passive web appli-cation security assessment tool.”Ratproxy – Project Hosting on GoogleCode. 2009 <http://code.google.com/p/ratproxy>.

9. IBM. “Help ensure Web site securityand compliance.” IBM Web Site Securityand Compliance – Rational. 2009 <http://www-01.ibm.com/software/rational/offerings/websecurity>.

10. Fortify Software, Inc. “Source CodeAnalyzer (SCA) in Development.”2009 <www.fortify.com/products/detect/in_development.jsp>.