EA and Certification TestingAndrew PritchardEA Certification Microsoft Platform Supervisor Electronic Arts Inc.

Agenda

• Who We Are• EA Certification Services• EA Certification Structure• Sharing Knowledge • Aiming to Align• Tips

Who We AreIntroducing EA Certification• EA Certification is responsible for the Final Certification process on all EA Xbox 360

products

• An audit function post QA Final through which all EA products must pass before submission to the Microsoft Certification Group

• EA Certification manages the submission process into Microsoft, as well as handling TCR Exceptions

• Main Certification liaison between EA and Microsoft Certification Group (via Release Management)

• Service Provider to EA Studios, Consultation, Pre-Final Reviews, Age Ratings, Knowledge Sharing

• To provide consistent level of service, support and quality to Microsoft despite disparate EA teams around the world

Who We Are Where We Are, Who We Work With

EAC – Burnaby ,Canada (Black Box, EA SPORTS +

QA)

Singapore(Cert and QA)

Bucharest, Romania

(QA)

EA Montreal, Canada

(Studio + QA)

EALA – Los Angeles (Danger Close + QA)

EA Tiburon, Florida

(Studio + QA)

Stockholm, Sweden(DICE+ QA)

Baton Rouge, Louisiana

(QA)

Buenos Aires, Argentina (QA)

Guildford, U.K.(Bright Light Studio, Criterion

+ QA)EA Certification (Canada)Microsoft DLC Specialist

EA Certification (Redwood Shores)

MS Submissions TeamMicrosoft DLC Specialist

EA Certification – Madrid, Spain

Microsoft Test GroupMS Submissions Team

Microsoft DLC SpecialistTester Pool

EA Redwood Shores (EARS)

(Visceral Games + QA)

Edmonton, Canada

(BioWare)

Who We AreWhat Certification Means at EA • Ensuring Release Readiness – become a key contributor to the drive for quality and

predictability, through effective audit, knowledge sharing and strong partnerships between Microsoft and Electronic Arts.

Any game whose quality is certified by EA Certification…

...is ready for release to Microsoft & EA’s Customers

• Xbox 360 Test Group• Submissions Group

• Project Management• Technical Operations

Software Quality

• Pre-Certification (Compliance)• Pre-Final Audit (PFA)• Submission Reviews

• Age RatingsOnline Performance +

CompatibilityCross-Continent Online Checks

Post Launch Support

Additional Services

Who We Are What EA Certification Offers• High Level Audit Function: Testing encompasses Xbox 360 Build Verification Tests, Technical Certification Requirements, Functional Test Cases and EA Standards, working towards alignment with Microsoft

• Focused Coverage: Test ALL EA titles. Aim for 1st Time Pass/Release Readiness/No Repeat Fails

• Balancing Turnaround Time With Coverage: More Streamlined Service Level Agreement. Expert testers on all Product types: Disc Game, Digital Download Preview (DPP), Xbox LIVE Arcade, Title Updates, Marketing Beta, Tech Beta, Games on Demand (with code changes), DLC.cab

• Separate from Studio Teams: Internal and EA Partners (EAP) studio teams are our customers, Worldwide QA are our partners

• Team of experts united in meeting Microsoft and consumer expectations

• By actively engaging our business partners early in development and through continued support and guidance, we ensure software and game quality and customer satisfaction

• An essential partner to our customers, the EA Certification team builds trust through timely interactions, adaptability and transparent relationships

Who We AreOur Customers• EA Certification interact with all EA Studios developing for Xbox 360, including:

• As well as certifying games developed by third-party developers via the EA Partners program,

• Harmonix (Rock Band® 3)• Paramount Digital Entertainment (Rango™)• Spicy Horse (ALICE: MADNESS RETURNS™)• Crytek (Crysis®2)• Epic Games/People Can Fly (BULLETSTORM™)• GRASSHOPPER MANUFACTURE INC.(SHADOWS OF THE DAMNED™)

Rock Band ®3: © 2010 Harmonix Music Systems, Inc. All rights reserved. Covered by the following U.S. Patents: 6,429,863, 6,482,087, 7,164,076, 7,459,624, 7,625,284 and patent applications pending. Harmonix, Rock Band, Rock Band 2, Rock Band 3, Rock Band Network and all related titles and logos are trademarks of Harmonix Music Systems, Inc., an MTV Networks company. Rock Band, Rock Band 2, Rock Band 3 and Rock Band Network developed by Harmonix Music Systems, Inc. MTV, MTVGames and all related titles and logos are trademarks of MTV Networks, a division of Viacom International Inc. EA and the EA logo are trademarks of Electronic Arts Inc. All rights reserved. All other trademarks are properties of their respective owners.Rango™ : © & ™ 2011 Paramount Pictures. All rights reserved. EA and EA logo are trademarks of Electronic Arts Inc. All other trademarks are the property of their respective owners.ALICE: MADNESS RETURNS™ : © 2011 Electronic Arts Inc. EA, the EA logo and Alice: Madness Returns are trademarks of Electronic Arts Inc. All other trademarks are the property of their respective owners.Crysis®2: © 2011 Crytek GmbH. All Rights Reserved. Crysis, Crytek and CryENGINE are trademarks of Crytek GmbH. EA and the EA logo are trademarks of Electronic Arts Inc. All other trademarks are the property of their respective owners.BULLETSTORM™: © 2011, People Can Fly Sp. z o.o.  All rights reserved.  People Can Fly, the People Can Fly logo, Bulletstorm, and the Bulletstorm logo are trademarks or registered trademarks of People Can Fly in the United States of America and elsewhere. Epic, Epic Games, Unreal, the Powered by Unreal logo and the Epic games logo are trademarks or registered trademarks of Epic Games Inc. in the United States of America and elsewhere. EA and the EA logo are trademarks of Electronic Arts Inc. All other trademarks are the property of their respective owners.SHADOWS OF THE DAMNED: © 2011 GRASSHOPPER MANUFACTURE INC. Shadows of the Damned is a trademark of GRASSHOPPER MANUFACTURE INC. EA and the EA logo are trademarks of Electronic Arts Inc. All other trademarks are the property of their respective owners.

Who We AreEA Certification Road To Release (generic timeline)

Proj

ect

Gre

en L

it

+8 Days-35 Days

EA Certification Final Review

(8 calendar days)- BVTs/TCRs/FTCs/

EA Standards- CertNet/ProdNet Server

Prep #2

- Pre-Certification (TCRs)

- Network Performance & Compatibility

-Cross-Continent Playtests with WW

QA

- Concept Submission Form

- XeMID/Title IDs Obtained

- PartnerNet Config

Project Manager/Submissions Stakeholder Communication & Planning• Confirm Scope, Cost Estimate, Pre-Plan• Testing Resources• Understand Game Team Needs via Face to Face/Calls• Milestone Dates Communicated• Risk Prediction

Microsoft Optional Pre-Certification

- Pre-Final Audit (TCRs/FTCs)

- Submission Reviews

- Packaging Reviews- CertNet/ProdNet

Server Prep #1

Age Rating Submissions

-15 Days 0 Days Final

Final Submission

to Microsoft

Certification

Microsoft Optional

Final Certification

Green = Reviews at MicrosoftBlack = Reviews at EA Certification

RTM

EA Certification ServicesTest

Pre-Certification ReviewCompliance Testing

TCR SweepNetwork Performance

Pre-Final AuditCompliance TestingFunctional Testing

Submission Reviews

Printed Material Reviews/In-game Manuals

Final Certification Review Compliance TestingFunctional TestingFull TCR/FTC Pass

EA Standards

EA Certification ServicesSubmissions

Technical ChecksAsset Verification

Worldwide Age RatingsSubmissions to Age Rating Bodies

Asset/Paperwork/Code Verifications

Microsoft POC via Release Manager

Co-Ordinate submission to Microsoft

Reduces issues on

submission assets and

details

Ensures that all

submissions are done

consistently

Highlights areas of

concern prior to submission

Consistent communicatio

n with MSNo Repeat

Fails

All Submissions to Microsoft are consolidated to minimal POCs

EA Certification ServicesLate Shift/Technical Operations

Late Shift

• Shift that overlaps with the regular Xbox 360 Test Hub: 2pm-11pm (CET) / 5am-2pm (Pacific)•Regression support•Out of hours point of contact for our North American studios

Technical Operations

•Develop Certification processes for EA wide adoption• Improve internal systems/tools, propose new technical solutions• Provide key Xbox 360 knowledge company wide•Align with Studios/QA, find out what’s important to them, provide continuous process improvements• Identify topical/top Hold, Rejection and Failure areas in Process and Code, direct internal process improvements

EA Certification StructureXbox 360 Test Hub Structure

Xbox 360 Platform Supervisor

Xbox 360 Project Lead

Xbox 360 Test Specialist

(Compliance)

Xbox 360 Senior Tester

Xbox 360 Project Lead

Xbox 360 Test Specialist (Online)

Xbox 360 Senior Tester

Xbox 360 Project Lead

Xbox 360 Test Specialist

(Functional)

Xbox 360 Submissions

Specialist

The Xbox 360 Test Hub is made up from the following positions:

• Platform Supervisor• Submissions Specialist• Project Lead• Test Specialist• Senior Tester

This team is complimented by a dynamic pool of testers which expands or decreases based on work demand

All members are focused on and highly knowledgeable in the certification of Xbox 360 titles

EA Certification Structure Xbox 360 Test Hub Roles

•High Knowledge of Xbox 360 guidelines•Leads the EA Certification Xbox 360 team

Xbox 360 Hub Supervisor

• Responsible for scheduling/managing testing on specific titles• Works closely with QA/Studio to ensure any concerns from our own testing are communicated and discussed (1x Lunch/1x EOD Meeting)

• Project accountability on a title by title case –Testing, Planning

Xbox 360 Project Lead

•Dedicated to specific Microsoft areas; Compliance, Online, Functional, Submissions

•Use of specialised Microsoft tools (APImon, XBWatson etc..)

Xbox 360 Specialist Compliance, Online,

Functional, Submissions

•Test all full products, Title Updates, DLC, Demos and more•Works closely with Test Specialists•Directs and mentors/oversees results from the Test Pool

Xbox 360 Senior Tester

All roles within the Xbox 360 Test Hub have different areas of expertise and accountability

Working together for the common goal of improving the quality of EA Xbox 360 titles and striving for passing at Microsoft 1st time, every time

Knowledge SharingCertification Excellence

Strong Partnerships Release Readiness

• Certification is a secure central hub all titles must pass before Microsoft Certification; providing quality and predictability for frontline EA products

• Established Compliance, Functional, Online and Submission teams with EA & Microsoft wide knowledge of process, best practice and guidelines

• Direct relationship with Microsoft Release Management; monitoring and providing real-time status updates across EU and NA time-zones• Dedicated POCs between EA Cert – EA Studios –

MS• Close relationship with Worldwide QA

• Our review is the final checkpoint and approval step within EA before release to Microsoft• Consolidate submissions process to minimise risks• Assessing key attributes of a release against quality

thresholds and standards, ensuring the highest quality of an EA title/brand

• Visibility of Microsoft test activity, allows Cert to analyse and report on common failure trends, helping provide company wide-knowledge sharing

• Certification Support Service, answering game queries within 24 hours

• Knowledge base for Xbox 360 requirements

EA Certification StructureBenefits

Ensure Release Readiness, Target 1st Time Pass, Plan Customer Satisfaction

Sharing KnowledgeResponding to a Rejection or CFR

Step 1.EA Cert receives the Fail Report

from MS

Step 2.Communicate CFR to QA and

Dev teams

Step 3.Enter CFR into

our online Xbox 360 fail tracker

database

Step 4.Investigate Root Cause of failure

Step 5. Create “Action Plan” within 24

hrs

Step 6.Update Test Cases for worldwide

QA/Cert adoption

This quick response action plan is put into place with immediate effect as soon as EA Certification is alerted to a failure at Microsoft

EA Certification will have a fully realised Action Plan in place within 24 hours of first receiving the failure

This plan leads to our goal of “No Repeat Fails”

Sharing KnowledgeCreating Worldwide StandardsEA Standards – an EA maintained database that

contains:• Microsoft, Sony, Nintendo, PC/Mac, Facebook

standards• Compliance Docs, Tools & Training Material• First Party News• First Party Failure Databases• TCR/Guideline Downloads• Xbox 360 Wikipedia

QA/Cert Forums/WW QA Committee - a shared forum for all platform related queries• Submit Compliance, Functional or Online queries

related to Microsoft Standards• 24 Hour Turnaround on responses – Guidance of

TCR implementation in titles • Review of in-game guideline messages/terms/in-

game manual

Aiming to AlignLearning from Microsoft PFA Service – Offered to our dev

teams as a companion to the MS Optional Final, during which EA Certification run a full TCR/FTC pass on a near Final build this

benefits us by ensuring we receive better quality Final builds,

which results in better submissions to MS

PreCert/Final – EA Certification run the same reviews as MS,

sometimes in tandem, sometimes with a degree of overlap

Issue Reporting – EA Certification use an issue

reporting template based on the MS format. This clear and concise format has resulted in simple to

read and easy to understand issues meaning dev teams can fix

issues quicker

Test Cases – All test cases are kept up to date in real time with all MS updates and fail examples

Submissions Checks – EA Certification run a formalised set of Submissions Checks based on

MS requirements

BVT Process – A series of checks designed to determine suitability for submission to MS. Detailed

on next slide

EA Certification strive to align with MS Certification in as many ways as possible, including the following:

Aiming to AlignBVT Process • To improve our Certification process and to align with MS testing practices we implemented a Build Verification Test (BVT) system to avoid any potential rejections

• This system is based directly on the 17 BVT tests run at MS Cert

•While we will not hold of testing if BVT items fail, a title will not be submitted to MS until all applicable BVT tests are in a “PASS” status

Title Submitted to

EA CertBVT Checks will

be run

BVT Pass/Fail notification will be sent to game

team via BVT Template email

Game team will respond to

notification with required

items/information

Aiming to AlignBVT Process• Example of BVT notification email:

• The implementation of this process has seen a decline in rejections at Microsoft Mastering Lab

TipsGeneral Advice on CertificationUnderstand Game Team Needs - Preparation

• Face to Face, Direct Calls to Pre-Plan your Test Plan, understand resource/hour coverage, and align your services to studio expectations

• Explain your own Internal Review processes and communicate Microsoft dependencies, pre-empt any risks• Partner with QA (if a separate entity) to build relations and provide certification consultation

Communication, Communication, Communication• Communicate with Studio and associated QA Teams (daily reports, daily/twice daily check-ins, Office/Windows IM, Live

Meeting)• Ensure Bugs are clearly written with steps to reproduce, link to TCR/FTC (if applicable), report issues as soon as they

are found• Create detailed Test Plans with clear test team assignments, consider Matrix’s for

Achievements/Options/Quadrant/Features testing

Submission, Submission, Submission• Verify all assets in your submission to avoid On Holds & Rejections, ensure no grey areas exist between testing

functions• Run Build Verification Tests (BVT’s) all the time, every time, ensure LSP is configured!• Have a dedicated POC in your organisation responsible for submission/communication with your Release Manager

Analyse and Share Failure/Rejection information company-wide • Learn from previous mistakes to avoid repeat occurrence – timely updates to processes and test cases to ensure No

Repeat Fails!• Know your organisation’s Top 10 Mastering, Compliance and Functional Failures for all submission types

and…

TipsKnow your Resources and Friends at Microsoft!

• 360cert for Certification Technical Support to help improve your processes and remove ambiguity

• ATG for general help on Xbox 360

• Your Release Manager for • General Enquires• Troubleshooting• TCR Exceptions• Submission Management• LSP Configuration• Publisher & Cert Liaison• RTM Management• Platinum Program Updates

• Game Developer Network Portal (GDNP) - Resources• Process Documentation and White Papers on GDNP – dissect, understand and filter relevant information to

your immediate teams and company-wide

Thank You from EA Certification!

Q&A

© 2011 Electronic Arts Inc. Trademarks belong to their respective owners. All rights reserved.

www.microsoft.com

© 2011 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.