Pre- Commit Builds with Perforce Helix StreamsRakhee Kulkarni

Continuous Integration Builds

Pre-Commit CI builds gate checkins into CM for a set of pre-defined smoke tests. This ensures software quality on the branch. These can be serialized and slow.

Post Commit builds can be triggered on a bundle of commits. Since the test are run post commit, the time for failure resolution on these is critical. Finding the bad changelist and backing it out or fixing in a short time is necessary.

Pre-Commit CI Implementation

Usage of streams to manage code base for HW and SW builds Virtual streams for defining build types Restricting client specs

Pre-Commit Builds restrict submits to depot for changelists that are not tested for smoke test Advantages

• Keep TOT clean• Global teams – advantage managing cross site build failures• Smoke test coverage is protected

Disadvantages• Serialized and slow

Tools and Resources Used

CI tool from Electric Cloud• Setting up CI based on a configurable property• High Availability feature

Proof of concept Load sharing facility for job launching

• C++, VCS builds• Tests

CI Queue Implementation Highlights

Builds defined using virtual streams Extensive use of P4Python API for automation Gathers shelves based on a keyword in shelve description Aggregates shelves for speedup on throughput Conflicting files blocked by triggers Configurable for branches

CI build Implementation Flowchart• High level Flowchart• Bundling of non

conflicting CLs• Automated submits of

bundled CLs• Increased througput

using bundling• Serialized in case of

bundle failure

Highlights of Implemented CI Queue

Aggregate shelves – bundling algorithm for non-conflicting changelists

Error handling at all steps, tests Configurable design to run on other perforce branches Cleaner artifact storing policy Cleaner clients creation using streams Use of keywords to manage shelves, EC_COMMIT,

EC_PASSED, EC_FAILED

Submit Code • Submit done through

automation • Multiple p4 sessions as p4

superuser and owners of CLs

• Added check for up-to-date files at shelve with shelve-submit trigger

• Added checks for bad shelves

• Added checks for conflicting files in queue

• Added check for unit test on HW side

Perforce Triggers to support submit automation

Added shelve-submit triggers• Checking out of date files in user WS• Checking conflicts with the files in queue • Additional checking for empty shelves

Post Submit Triggers• Block submits for code with certain keywords

CI StatisticsThe chart below describes the number of passing builds for bundling in a week.

Multiple CI queues on main/release branches

Build Statistics

Number of CLs shipped - ~4500 Number of CLs on a day Max ~72 with smoke test of

~50minutes Aggregating produced better results than serializing

Conclusion

Aggregating CLs improved throughput on the serialized builds

Use of perforce streams was helpful with automation to build Use of p4python was helpful in achieving automation Perforce triggers supported automated submits Pre-commit CI builds ensure stability of the main stream at

the cost of Serializing

Thank you!