White Paper: Creating a World-Class Build System, and Getting It Right

Copyright Citrix Systems, Inc. All rights reserved.

1

Creating a World-‐Class Build System, and Getting It Right Scott Estes

Citrix Systems, Inc.


2

1 Table of Contents 2 Introduction ......................................................................................................................................... 4

3 SCM and Build are Critical .................................................................................................................... 4

4 Who Are the Vision Stakeholders? ...................................................................................................... 4

5 What are the Requirements? .............................................................................................................. 5

6 Evolution versus Revolution ................................................................................................................ 6

7 Build versus Buy? ................................................................................................................................. 7

8 Leveraging Others to Accomplish Common Goals ............................................................................... 7

9 SCM and the Build System at Citrix ...................................................................................................... 8

9.1 Engineering Structure .................................................................................................................. 8

9.2 XenApp/Desktop Component Structure ...................................................................................... 9

9.3 PLANB Structure .......................................................................................................................... 9

10 The Flow of the Build ..................................................................................................................... 11

10.1 Input .......................................................................................................................................... 12

10.1.1 Automation at Citrix .......................................................................................................... 12

10.2 Pre-‐Build Steps ........................................................................................................................... 13


10.3 Build ........................................................................................................................................... 14


10.4 Post-‐Build Steps ......................................................................................................................... 16


10.5 Output ........................................................................................................................................ 18


11 Optimizing the System ................................................................................................................... 20

11.1 Keep it Deterministic ................................................................................................................. 20

11.2 Keep it fast ................................................................................................................................. 22

11.3 Keep it simple ............................................................................................................................ 25

11.4 Keep to standards ...................................................................................................................... 25

12 Scheduling Updates and Maintenance .......................................................................................... 25

12.1 How to do it ............................................................................................................................... 25

12.2 When to do it ............................................................................................................................. 26


3

13 Final Thoughts ............................................................................................................................... 26


4

2 Introduction Anything worth doing has to begin with a vision, regardless of how mundane it may seem. Without a vision, whatever it is we’re trying to do is inevitably doomed to either go supernova in the short term, or die a drawn out, laborious death over a longer period. In both scenarios, the collateral damage is devastating and usually, the only way to recover is to trash everything and start over at the beginning again. An interesting study could be done to show how much time, money and resources are wasted by various industries, but that is a topic for another time. What is explained and demonstrated in this paper is how to define, implement and maintain an optimized Build/SCM/ALM vision, and avoid common (and not so common) pitfalls along the way.

3 SCM and Build are Critical It’s been said that the Build is the heartbeat of a project. This is a fundamental truth, yet it’s amazing how more often than not, person-‐months are invested in the Develop and Test visions and strategies of a project, and scant person-‐hours, (if that) are given to how the project code base will be managed and built, until the very last possible point in time. This “Build Avoidance” is understandable because everyone is usually focused on customer requirements, product features, schedules and a myriad of other things that siphon attention away from SCM and Build. All too often, the Product team will simply assume that the product will magically coalesce, or that someone else is thinking about how to build it. This is the point where the savvy Build Visionary can, with the proper support and resources, create order from the technical chaos, and provide a quality-‐built product that meets everyone’s expectations.

It must also be stressed that there must be executive level support for SCM. If the senior management staff, and those in charge of driving the overall Engineering roadmap(s) are not bought in to the Build Vision, there is little to no chance of obtaining the resources required to create a successful SCM and Build infrastructure.

4 Who Are the Vision Stakeholders? Identifying and knowing all the stakeholders of the Build system is the first step to eliminating any “blind spots” in the Build vision. Understanding the motives and mindsets of the stakeholders also goes a long way towards ensuring that the Build system is optimized and meeting all the requirements.

At the high level, the Build stakeholders can be organized as follows,

• Customers • Project Management • Product Development • Finance • Product Marketing/Sales • Quality Assurance/Test


5

• Lifecycle Maintenance/Customer Support • Build

These stakeholders will have varying levels of input and requirements to the Build vision and some may not immediately realize that they are stakeholders. This is acceptable as long as the Build Visionaries remain aware of all the stakeholders and are proactive at meeting their requirements. As an example, the customer is as key a stakeholder as is Product Development. That being said, it might be considered a bit odd to find a Build engineer consulting with a customer on how the product should be built. Here, the forward thinking Build visionary should be looking at project schedules and planning how the Build vision can facilitate delivering a quality product on time to the customer. Conversely, the Build Visionary should be in constant communication with Product Development to ensure that their requirements are being accounted for in the vision.

5 What are the Requirements? Once the stakeholders have been identified, the next step is to gather their requirements. The following table defines, at the high level, some of these requirements.

Stakeholder Requirements Customers • Quality product

• Delivered on time • Affordable • Maintainable

Project/Release Management

• Deterministic builds • On schedule • Of quality

Product Development • Fast builds • Efficient Build system • Fully supported

Finance • Adherence to budget • Proper forecasting

Product Marketing/Sales • Features • Quickly available

Quality Assurance/Test • Fast builds • Automated delivery and testing • Fully repeatable

Lifecycle Maintenance • Escrow-‐able, (“goldenized”) builds • Fully repeatable • Patchable • Business Continuity

Build • Stakeholder requirements • Adhere to SCM standards • Resourced and supported


6

There may be additional requirements to what is listed here, but having the ones listed here, well defined, will ensure a smooth departure on the path towards the world-‐class build system.

There is one additional stakeholder with requirements dependent on the maturity level of the company, and that is Security. The requirements here range from simply protecting the Intellectual Property of the company to full-‐scale security audits, threat models and so on. These requirements can be a stand-‐alone topic by themselves, and as such, left for another time.

6 Evolution versus Revolution At this point, the Build Visionary should have,

a) Executive sponsorship b) Identified stakeholders c) Stakeholder requirements

With these, he/she can then decide which of the two approaches, (Evolution or Revolution) best suit the needs of the product and company as a whole. Each approach has associated pros and cons, and often a combination of the two is what proves to be most effective moving the vision forward. The high-‐level pros/cons are listed in the following tables,

Evolution Pros Cons

• Usually a slow, stable rate of change • Can be too slow to affect meaningful change

• Training/documentation can keep pace • Impetus can be lost as others vie for resources

Revolution

Pros Cons • Shortcut/mitigate barriers to change • Amount of change can adversely affect

stakeholders’ ability to absorb it. • Can open avenues for other shift changes. • Training/documentation will usually not

keep pace. • Can cause unintended side-‐effects if not

carefully planned.

Focusing exclusively on only one of these two approaches is going to result in failure of the Build Vision at some point, either from stakeholder exhaustion or loss of confidence in the Vision/Visionary. While there are no hard and fast “rules” on how to combine these into an effective vision, experience indicates that Revolution works best with small, experienced teams that can then act as “ambassadors” to the larger organization when the vision is introduced at that scale. Evolution can, in fact, harness these


7

“mini” revolutions such that the larger teams can absorb the changes as schedules and resources permit. The most important task for the Build Visionary is to stay on top of the teams to ensure that they remain committed to the Vision.

7 Build versus Buy? There is no magic answer to this perennial question; it can be different to varying degrees, for every application at every company. The first major challenge the Build Visionary must overcome, is to adequately evaluate the available product offerings to see which ones could fit into the Build Vision compared to building and supporting the infrastructure in-‐house. It’s not enough simply to determine if the tools work technically, the evaluation and resulting recommendation has to include the overall return on investment, (ROI) and ongoing total cost of ownership, (TCO). The recommendation will be based on,

• The amount of funds/resources available, and • The amount of time allocated for the evaluations.

The second major challenge for the Visionary is to convince the company to allocate the amount of time needed to properly answer this question, while keeping pace with the ongoing requirements of the product release cycle(s).

8 Leveraging Others to Accomplish Common Goals Very rarely, if ever, will there be enough SCM/Build resources to singularly accomplish the Vision. Therefore, the savvy Build Visionary will look to others outside the immediate Build team to assist. This, aside from moving the Vision forward, has two important side effects,

1. People who help feel like their ideas are being heard and valued. As people see the results of their input and work, they will become more comfortable with continuing to present new ideas and processes. When channeled properly, this can become a self-‐sustaining cycle that further ensures the relevance of the Vision.

2. Those who help in turn become “Build Ambassadors” to the rest of their respective teams. Generally speaking, the best customer of an invention is the inventor. If a person on a development team has contributed to the Build Vision, that person will then most likely take the vision and spread it among the rest of the team, thus ensuring that the Vision is widely accepted without relying on the Visionary to do all the “selling”.

There are a couple of “gotchas” that must be mitigated when leveraging resources from other teams,

1. Don’t interfere with other teams’ mandates and objectives. Most development managers like to keep their teams 100% focused on the team’s particular roadmap, leaving no time for other pursuits. Most likely, the help provided will be in the form of “skunk-‐works”, or after-‐hours type projects. Sometimes the benefit to the overall group will be significant such that a Development


8

manager will assign a certain percentage of his/her resources to formally helping on SCM/Build objectives. In any scenario, the Build Visionary must take extreme care to balance the Build needs/wants with those of the supporting team(s).

2. Make certain to manage expectations. The last thing any of the Build stakeholders want is to feel, (whether real or imagined) that their ROI is not realized. Failure to manage the expectations is one of the quickest ways to destroy the credibility and success of the Vision.

In summary, the Build Visionary must exert all his/her diplomatic abilities when working with extended teams and other Stakeholders to ensure continued support for the Build Vision.

9 SCM and the Build System at Citrix Up to this point, the discussion has focused primarily on non-‐technical, logistical and/or philosophical aspects that must be considered when creating an ideal Build Vision. In parallel, the Build Visionary should be constructing the building blocks that will comprise the system that will execute as mandated by the Vision.

The core of the ideal Build system should revolve around well-‐defined processes and tools needed to facilitate the fundamentals of good SCM practices such as,

• “Determinism” ( the ability to determine attributes of a build) • Branching/Merging • Parallel development • Lifecycle maintenance • “Goldenization”/escrow • Business Continuity • Some level of Development automony/self-‐service • Etc.

Discussions around some of these areas can be separate topics, and as such will not necessarily be discussed in detail here, other than to show how the Citrix Enterprise and Desktop Applications Build team have leveraged the power of Perforce to facilitate the realization of our ideal Build Vision.

Before the technical details of the build system are discussed, the composition of the Engineering teams responsible for the product, and the structure of the product itself, will be explained to show how Citrix’ ideal Build vision has evolved to meet the requirements of the stakeholders.

9.1 Engineering Structure There are over 500 engineers organized in development and supporting teams located in the following sites,

• Bangalore, India • Bedford, Massachusetts


9

• Cambourne and Chalfont, UK • Fort Lauderdale, Florida • Santa Clara, California • Sydney, Australia • Tokyo, Japan

The global Build team consists of,

• 1 engineer in Bangalore • 1 engineer in Bedford • 5 engineers in UK • 4 engineers in Fort Lauderdale • 1 engineer in Santa Clara

The Build engineers are primarily responsible for maintaining the Build Vision and Systems at the site where they are located. Additionally, the team can leverage a “follow-‐the-‐sun” work schedule to be able to maximize Build resources and coverage without requiring any one of the teams to provide support outside of normal working hours at each site.

9.2 XenApp/Desktop Component Structure The code base that comprises the product is structured along core technology boundaries, known as Technology Components, (TCs), (e.g. Client, Server, protocols, etc.). The engineers are assigned into TC teams, and focus on their technology instead of a particular release of the overall product. Each TC team has a junior architect, (or Principal Design Engineer) who is responsible for the evolution of the technology, and coordinating what features can be delivered for a particular product release with the TC Manager, Release and Product Management teams. The deliverables of the TCs are released as SDKs which can consume/be consumed in such a way as to create a complete product. There’s a lot more that could be discussed about the TC methodology, and its advantages and disadvantages.

In synopsis for this discussion, the main advantage of this approach is that a huge code base is organized into small, manageable chunks that can be “snapped” together to create a product release. The disadvantage is that there is more responsibility placed on the architects, managers and Release Team to ensure that the TC teams are communicating, and any component dependencies are aligned for a particular release.

9.3 PLANB Structure The build system at Citrix is called the Perforce Linked Automated Network Builder, (or PLANB). It is comprised of six distinct components that communicate with the Perforce servers, and are responsible for,

• Determining when to build • Creating the Build environment • Building the binaries


10

• Creating/updating/publishing the TC SDKs • Creating the installers/DVD layout • Publishing the product DVD

There are two types of builds that PLANB controls,

• SDKs • DVD layouts

When building SDKs, there is an optional, manual SDK updater component, (using ASP.net) provided to “publish” the output as determined by the TC owner. This process is intended to be used when the TC output is at some level of quality as to be considered “locked” for the overall product DVD release. If a TC owner chooses to not publish his/her SDK, then the latest successful build of the output is used for the DVD layout.

A Watcher service (using WCF functionality) polls the Perforce servers for code submissions. When a submission is detected, the service determines if any client spec template changes are required, and make any needed changes before passing the change list(s) to be built, and control to the next component.

The Build VM creator (also using WCF), seeks out an available “bare-‐metal” machine, starts up a XenServer-‐based virtual machine template on it, and passes control to the Building component.

The Binaries Builder, (using C# and MSBuild functionality) synchronizes the code using the change lists sent from the Watcher, compiles the code, creates the SDK(s) and/or other components into a “save” area which can either be used as input to publish a TC’s component(s) or as direct input into the Installer/DVD creator.

The Installer/DVD creator, (using PowerShell scripts), collects the binaries, (SDKs, installers, etc.) from Perforce and the “save” area, and creates the DVD layout that will be published to Test.

The Publisher, (also using PowerShell cmdlets) then copies the DVD image out to an area where can be consumed by the various Engineering groups, and sends a semaphore to the Test infrastructure which initiates automated Build Verification Tests and subsequent Regression test suites.

When defining these components, it was realized that the overall system needed to be broken into smaller pieces or projects such that the architecture of each component could be properly designed without getting lost in the larger scope of the entire system. The projects were aligned as follows,

• Cumulonimbus, responsible for the Watcher and Build VM creator components, as well as the code synching process of the Binaries builder component. Utilizes a SQL database to store build information.

• Piston is a “toolbox” of PowerShell scripts used to create DVD layouts and associated artifacts. • Vulcan is the core build tool. It hooks into Visual Studio’s MSBuild engine to compile Windows


11

code.

Figure 9.3.1 maps out the components, and the interactions between them. Additional details of each component and the associated projects will be discussed in sections 10.1-‐10.5 below.

Figure 9.3.1:

PLANB

(Project) Piston(Project) Cumulonimbus

Watcher Build VM Creator

(Project) Vulcan

Binaries Builder Installer/DVD Creator Publisher

Synchs code

Notified of submissions and updates client views Synchs SDKs

SDK Updatermanual process

UpdatesSDKs

“Save” area

Perforce

Svr 1 Svr 2 Svr 3 Svr n...

10 The Flow of the Build Regardless of the complexity of the product/project, the flow of the Build should always follow the basic flow of control in the following diagram,

Figure 10.1:

Input Pre-‐Build Step(s) Build Post-‐Build Step(s) Output

This diagram illustrates the flow of control at its most componentized level. The Build Visionary should clearly define each of these components with the following attributes,

• Owner(s) • Stakeholder(s) • Input(s) • Control(s) • Output(s)


12

Keep in mind that there may be multiple builds following this model that feed as inputs into other builds and/or the ultimate layout of the product. Having the attributes for each “layer” of build well defined is key to ensuring a deterministic, simple and efficient build system.

10.1 Input Input can be loosely defined as any new “thing” that would cause a subsequent build to be different than the current build. Narrowing this down a bit, an input is usually a direct code submission, an SDK or some other “black box” component update. A “black box” update is, at some point, going to be the result of a direct code submission. Multiple or tiered Build systems, (which can be from where “black box” updates come will be discussed later).

Applying the previously defined attributes to the Input, defines the component as follows,

Table 10.1.1:

Attribute Details Comments Owner(s) • Project/Release

Management Responsible for ensuring builds are released as per schedule.

Stakeholder(s) • Project/Release Management

• Build • Product Development • Quality Assurance/Test • Product Marketing/Sales

Everyone in this list is vested in ensuring that the build is successful at all stages of the process.

Input(s) • Code submission(s) • “black-‐box” update • Previous build reports

Any new thing that needed in the build as well as any historical artifacts.

Control(s) • “Pre-‐flight” checks • Triggers • “Determinism”

Ensures quality and traceability of submitted code.

Output(s) • “Of quality” code Code has passed all controls and can be built.

10.1.1 Automation at Citrix The watcher service, (part of project Cumulonimbus), monitors the Perforce ports for submissions, and automatically queues them for building in the build farm. When a submission is detected, the service gathers the change list(s) that comprises the submission, and then checks the project’s configuration file to see if any changes in it require updates to the client view templates, (e.g. a new SDK is needed by the project). If any such changes are detected, the service will update the associated templates, before passing the change list(s), and a job ID associated with the change lists to the Build VM creator. This interaction is shown in Figure 10.1.1.

Figure 10.1.1:


13

Watcher

Code SubmissionPerforce

Submission notification received

Project config change?

Yes

Call VM CreatorNo

Update Client spec templates

Get Change List(s) and job ID

Cb database

Job ID

10.2 Pre-‐Build Steps There usually aren’t too many steps at this point; usually consisting of synching the code, and any build machine preparation needed prior to building the code. The attributes for Pre-‐Build steps are;

Table 10.2.1:

Attribute Details Comments Owner(s) • Build Responsible for ensuring the process and

infrastructure meets stakeholder requirements. Stakeholder(s) • Project/Release

Management • Build • Product Development • Quality Assurance/Test • Product Marketing/Sales


Input(s) • “Of quality” code • Build machine/infrastructure

Code and infrastructure on which to build it.

Control(s) • Available infrastructure • System resiliency

Must have reliable and available infrastructure.

Output(s) • Buildable code • Build environment

Code is now ready to be built.

10.2.1 Automation at Citrix The Build VM creator component, (also part of project Cumulonimbus), establishes and maintains a load balanced build farm consisting of dynamic and consistent build machines leveraging Citrix XenServer® technology.

When the Watcher service sends a cue to build, the creator determines which virtual machine, (from pool) to use based on the job ID sent from the Watcher. It then determines if the virtual machine is


14

available in the pool. If it is not, (i.e. it is building another job), the creator places the job in a queue to build; otherwise, it then calls into the Xen API functions to connect a data and tools drives to the virtual machine, assigns the machine to a host and starts it. The final step in this part of the Build process is a build script, (which resides in the Windows Startup folder) initiates to begin synching the code base. The control flow is shown in Figure 10.2.1.1 below.

One additional note, at this point the data and tools drives are the storage areas for the code base and build tools required for building the code base, respectively. These drives reside on a NAS infrastructure which will be discussed further in section 11.2.

Figure 10.2.1.1:

Build VM creator

Cue to build Determine VM type

Connect Data and Tools drives to VM

Is a VM available?

Wait for VM to become availableNo

Yes

Cb database

VM type

Start VM/Build scriptAssign VM to host

10.3 Build The Build Visionary has to be careful that this part of the system is maintained properly to remain in line with the Vision. Competing requirements from the development teams that for the most part, own the component build files (solutions, make files, etc.) are often introduced as shortcuts/hacks to accomplish short-‐term goals, (e.g. deadlines/milestones) at the component level. Failure to review and make necessary updates to the build files will result in a non-‐optimized product build. The following table defines the attributes for Build,

Table 10.3.1:

Attribute Details Comments Owner(s) • Build

• Product Development Responsible for jointly maintaining the component build tasks, solution files, makefiles, etc.


15


• Build • Product Development • Quality Assurance/Test

Everyone in this list is vested in ensuring that the build is successful at all stages of the process. Test will mostly be concerned with build times and availability.

Input(s) • Buildable code • Build environment

Control(s) • Build standards Component build scripts and associated files must conform to company standards.

Output(s) • Binaries • Installers*

Binaries ready to be signed, packaged, etc. in preparation for release to test. *Installers may be considered part of the Post-‐Build steps.

10.3.1 Automation at Citrix Project Vulcan provides a build framework around Microsoft’s Visual Studio MSBuild engine into which Windows component solution files are fed. This framework, shown in Figure 10.3.1.1 is kept as simple as possible for maintenance purposes. Only the Input, Post-‐Build and Output steps are developed and maintained in-‐house. The Pre-‐Build and Build steps are all MSBuild-‐centric, and no additional maintenance is required outside of the development teams maintaining the solution and project files.

That being said, the Build team has developed a Citrix-‐specific add-‐in, including project templates, (based on the out-‐of-‐the-‐box templates provided with Visual Studio), to Visual Studio which provides support for Engineering-‐wide requirements such as,

• Enforcement of banned Microsoft functions, (e.g. strcpy() versus sstrcpy()). • Consistent signing of binaries. • Automatic inclusion of test frameworks, (e.g. nUnit and code coverage). • Hooks into in-‐house developed tools used in the build, (e.g. custom graphics generator)


16

Figure 10.3.1.1:

Input Pre-‐Build Steps Post-‐Build Steps OutputBuild

Synch codeCreate SDKs,

installers, reports, etc.

ID components to build

Check and build deps Compile code

Sign, symbol index, copy output, etc.

Compile successful?

Log error and continue

No

More to build?YesConfiguration .proj/search

Yes

No

Non-‐Windows builds utilize standard make files with a similar framework. The non-‐windows build system does not support all the functionality that MSBuild provides, which means it is somewhat simpler, yet requires more developer support to ensure the make files are kept as clean as possible, (especially in the case of dependencies), to ensure fast, accurate builds. Additionally, system has not been updated to support the enhanced dependency checking and reporting capabilities found in the Windows system. This does not currently present a burden to the Development and Build teams, as the non-‐Windows code base consists of reasonably small client side applications. The adequate amount of Determinism is provided by the Perforce filelog functionality. The framework is shown in Figure 10.3.1.2.

Figure 10.3.1.2:

Input Post-‐Build Steps OutputBuild

Synch codeCreate SDKs,

installers, reports, etc.

Search for makefiles

Compile/Link code

Sign, symbol index, copy output, etc.

Compile/Link successful?

No

More to build?Yes

Yes

No

Log error and die

10.4 Post-‐Build Steps At this point, all binary components needed for the product build should be available for additional post-‐build processing. This post-‐build processing can consist of binary signing, installer creation, (some teams may choose to make these part of the Build steps), “layout” (also known as the DVD) creation, gathering data for the Build report, and any other tasks in preparation for releasing the build to Quality


17

Assurance/Test. The attributes are as follows,

Table 10.4.1

Attribute Details Comments Owner(s) • Build

• Product Development Responsible for jointly maintaining the install tasks. Build responsible for ensuring that the build is readied for release.


• Build • Product Development • Quality Assurance/Test

Everyone in this list is vested in ensuring that the build is successful at all stages of the process. Test will mostly be concerned with build times and availability.

Input(s) • Binaries • Installers (if any)

Control(s) • Build standards Install and “layout” scripts must conform to company standards.

Output(s) • Installable product image • Build reports • Email notification • Semaphore for automated

testing

Everything needed to continue with Build Verification and Regression Testing.

10.4.1 Automation at Citrix Project Piston provides a “toolbox” of PowerShell scripts which are used to create the product DVD. The request to create a DVD can either come from a schedule semaphore, or be manually requested from the Build Portal, (more on the portal later).

When the request to create a layout is received, the main PowerShell script extracts the data identifying the components needed for the product DVD from the project’s layout configuration file. This data is then passed to the next steps of creating the DVD image. Components that comprise the DVD contents are,

• Merge modules and installers from the various stand-‐alone sub-‐components. • Help files and supporting documentation. • Meta-‐installers. • Non-‐Windows clients

The next step in the process is to update the debug symbols server with the symbols generated during the build. The scripts then generate the various reports needed to provide the desired level of Determinism for the build,

• Location of the build • What changes went into the build


18

• Build time • Any major updates • Checksum for comparison with previous build. • What binaries changed • Etc.

The final stage of the process is to auto-‐generate and send an email notifying of the build’s availability, and sending a semaphore to the Test infrastructure to initiate automated testing. The process is shown in Figure 10.4.1.1.

Figure 10.4.1.1:

Piston

On-‐Demand/per schedule

Get layout information

Layout configuration

file

Release DVD for testingCreate DVD image Update Symbol

server

Symbol Server

Create reports

10.5 Output The final stage of the process, the build is now ready for Quality Assurance/Test to begin executing a set of “smoke tests” or Build Verification Test (BVT), to ensure the build will perform fundamental activities, (e.g. install). Based on the successful completion of the BVT, more extensive test suites will be executed on the build. Often the results of these tests will serve as input for subsequent builds. The attribute table for Output is as follows,

Table 10.5.1

Attribute Details Comments Owner(s) • Quality Assurance/Test Responsible for jointly maintaining the install

tasks. Build responsible for ensuring that the build is readied for release.


19


• Build • Product Development • Quality Assurance/Test • Product Marketing/Sales


Input(s) • Installable product image • Build reports • Semaphore for automated

testing

Control(s) • Test standards Test suites must conform to company standards. Output(s) • Test results

• Bug reports

10.5.1 Automation at Citrix The Citrix Automation Development Infrastructure, (CADI) is a SDK which provides a Test Automation framework for creating test suites which can then be executed in various test environments. Skynet is a NAS-‐based cloud infrastructure that provides a highly available virtualization infrastructure that will allow for fast, on-‐demand access to virtual machines that can be acquired by CADI. (Skynet is part of a larger quota-‐based storage infrastructure and will not be discussed in detail here).

CADI polls the build release area for a semaphore file which indicates a build is available. When it finds a file, it provisions a test infrastructure based on the product build requirements extracted from the Test infrastructure database. CADI then executes the BVT, which determines if the build is good enough for more exhaustive testing. If the BVT reports a failure, testing stops on the build, and it is marked as a failed or bad build. If the BVT reports success, the build is then submitted for additional testing to determine,

• Regressions • Performance • Scalability • Overall stability

Data from these tests is used to generate test, bug and other useful reports that indicate the state of the product and project. The high-‐level flow is shown in Figure 10.5.1.1.


20

Figure 10.5.1.1:

CADI

Build is ready Provision Test Environment

Create reports, generate CPRs, etc.Execute BVT Execute additional

test suites

No

BVT Pass? Yes

Report and die

Skynet

Test infrastructure

11 Optimizing the System The Build Vision must ensure the build system is optimized for the following,

• Determinism • Performance • Self-‐service • Standards

Having these areas optimized will ensure that the Build “Customers” are happy, and the overall Build needs and expectation of the organization are being met and exceeded.

11.1 Keep it Deterministic The term “Determinism” is overloaded from its usual, “fatalistic” definition, in that it means that the capability within the Build system exists to know everything about a product build. Not only does it have to exist, it has to be as efficient as possible. For example, the Perforce command, filelog will accurately report what changes went into a build. Having the additional capability to determine which code submissions caused which binaries to change is not necessarily as simple yet is critical for Quality Assurance/Test to be able to determine quickly, in order to properly seed their test bed.


21

Example: Historically, when a XenApp/XenDesktop build was released to Test, there was no capability to accurately determine which binaries changed from build to build, (checksums and binary comparison tools, while useful, could not provide the levels of detail required for targeted testing). As a result, test cycles for intermediate builds required the same amount of time at those for milestone builds, which slowed the entire release cycle. An in-‐house developed tool called the Binary Sleuth leverages the information stored in the debugging symbols created by the Windows Symbol Indexing functionality, (for Windows builds only), to accurately determine which binaries change from build to build. This has enabled Test to target specific areas to test based on the scope of the changes to obtain results much more quickly than by starting at the beginning of the test suites and working down to the areas that need quick turnaround verification. The format of the report is as follows, Figure 11.1.1:

Being able to track and report Build data as efficiently as possible is critical to the overall speed of the product release cycle.


22

Example: The XenApp/XenDesktop product has evolved from a monolithic build to a compilation of 30 major components, (with over 64 subcomponents). There are many different data sets that need to be examined on a per build basis, and having these automatically generated in report format is critical for the Release team to keep development notified of the schedule. A list of the currently available reports is as follows, Figure 11.1.2:

11.2 Keep it fast The ideal goal here is to provide builds faster than they can be consumed. While this may be unrealistic, keeping the build times as low as possible will definitely mean that the consumers will remain happy and able to perform their respective duties without having to wait for the build.


23

Example: At one point before the Technology Component methodology was introduced at Citrix, the build time for a monolithic XenApp build was approximately 16 hours. These extremely long build times made it very challenging to keep to aggressive Test and Release schedules. Several factors contributed to the time,

• Compiling code multiple times • Compiling “dead” code • Build infrastructure not optimized. • Dependencies not well defined. • Etc.

Over time and with concerted effort, the build times have been reduced significantly to approximately 5.5 hours to build all the components in serial. Having said that, build times have been further reduced as a result of SDKs and components releasing semi-‐autonomously, (as per the TC model) and simply being included as part of the DVD layout. Additional development work to simplify the component dependencies have reduced the times as well as optimized incremental build times. Build times reports have been added to the Build portal and can now be reviewed to determine the cause for any significant changes. A sample report is shown in Figure 11.2.1. Figure 11.2.1:


24

Leveraging Perforce functionality, like P4Proxy, as well as isolated network segments, fast hardware, etc. are starting points to a fast build. The Citrix XenApp/XenDesktop build farm topology shown in Figure 11.2.2 below, utilizes these optimizations as follows,

• Uses fast Bare-‐Metal machines, (64GB RAM, Quad core half-‐height blades) paired with a NetApp storage device.

• Maintains multiple and dedicated Perforce proxy servers across the sites to increase performance.

• Isolates the Build farm on its own optimized 10GB network segment to avoid traffic “jams”. • Maintains a well-‐defined Business Continuity process to ensure that build jobs can be quickly

moved from site to site.

Figure 11.2.2:

UK

Cam

Devs

US

Business Continuity Center

Build Farm

NAS

1GB

10GB

Servers

Build machines

P4 ProxyDataP4 Proxy

Perforce Svr 1

Builds NAS

Perforce Svr 2

Devs

1GB

Perforce Svr1/Proxy ports

Servers

Build machines (+BC for Chf)

Chf

Devs

Servers

Build machines(+BC for Cam)

1GB

Perforce Svr1/Proxy ports

Builds NAS(+BC for Chf)

Builds NAS(+BC for Cam)


25

Additional tweaks, like incremental builds, parallel component builds will also further reduce build times. Some compilers, such as Microsoft’s MSBuild engine capitalize on the multi-‐processor capabilities of current hardware to spawn non-‐dependent components across the available processors to more fully optimize build times.

11.3 Keep it simple Most people enjoy a good puzzle for fun. Most software engineers don’t have the same feeling when they’re trying to meet feature milestones and project milestones, and find themselves lost in an intricate, cryptic build system. The cardinal rule of build systems is to keep the user experience as close to an “out of the box” install as possible. This means, don’t create a set of elaborate scripts/steps that modify the build environment needlessly. Recall Microsoft’s “Razzle” build system? It worked great as long as nothing changed, and when something within the environment did, then the results were unknown, and there was not an obvious way to debug. Happily, Microsoft learned their lesson, and the current Visual Studio/MSBuild system is very extensible to meet developers’ needs. The ideal Build Vision should support a core build “framework” into which additional layers or components can plug to facilitate the organization’s requirements, (e.g. binary signing, security settings, etc.)

At Citrix, for Windows-‐based builds, Visual Studio and Perforce comprise the core framework, and as few MSBuild plug-‐ins as required are leveraged to create the flagship XenApp\XenDesktop products. The details of the Vulcan Build system were previously discussed in section 10.3.1.

11.4 Keep to standards Use standards where it makes sense to do so, and resist the dark embrace of the “cookie cutter” approach because this is ultimately doomed to fail. How the Build Visionary chooses to define the standards should be based on development feedback and the problems they are trying to solve. For example, there could be a standard way to build all C#/Managed code components, or a way that all Windows-‐based components are signed. The reality is that the combinations are many, and to reiterate should be based on the needs of all the Build stakeholders.

12 Scheduling Updates and Maintenance

12.1 How to do it The Build Vision should account for system maintenance and updates, as this is an important part of keeping the Build system up to date and meeting new requirements. As mentioned earlier, keeping the various parts of the system componentized as plug-‐ins to the core framework will facilitate ongoing maintenance. Additionally, the following guidelines further ensure seamless transitions as a result of updates,

• Stage updates to get maximum ROI. Group similar updates together to minimize the amount of change to the system. This may seem obvious, but is often overlooked by the overzealous Build


26

engineer. • Automate as much as possible. scripting updates such that they require minimal manual

intervention also mitigates any potential human error • Test to trust. This is the most critical part of maintenance. Nothing will destroy the credibility

of the Build team quicker than a botched upgrade resulting in downtime for Development teams. Trust between the stakeholders is a requirement for a successful Build Vision, so test, test and test some more before implementing updates into production.

• Keep the Stakeholders in the loop on progress. Notification of key events during the maintenance, (start time, end time, benefits, changes in behavior, etc.) will also shore up trust in the Build team and Vision. The Stakeholders will feel like they are “in on” events, even if they don’t need to be, plus, the Build visibility doesn’t hurt the reputation of the team either.

12.2 When to do it Updates come with some overhead in that they can, (and usually do) come with some amount of change to the system. It is critical that they are scheduled with the knowledge and consensus of the Build stakeholders. Following the following general guidelines will ensure that there will be minimal confusion as the result of Build System maintenance,

• Align updates, (as much as possible) with project schedules and milestones. The last thing the Build Visionary wants is to conduct maintenance during a critical project milestone, (e.g. the last weekend before the project achieves final function/code freeze).

• Attach updates to a particular project. At the end of the day, it’s really all about getting the product to the customer, so having a Build Vision in general, (in addition to the maintenance), attached to a project ensures that the Vision will be funded properly. The savvy Build Visionary will work closely with the Release and Development teams to embed needed Build System updates into the project schedule.

13 Final Thoughts Like everything else in the technical industry, the Build Vision never ends; it is always evolving to meet new requirements, tools and processes. As we move forward into whatever the next new “thing” that will become the norm in Software Development, and companies increasingly focus on trimming costs and increasing revenues, the Build Visionary must take care to nurture the Vision, keep it looking forward, and constantly demonstrating the ROI to everyone who will listen. This will help secure the Build Vision’s position as a critical, core team within the organization. The Build Visionary will find himself/herself wearing many hats, from the technical to the salesperson, ensuring longevity of the Build Vision.