SCO Payroll Project Report, Senate Office of Oversight and Outcomes

7/27/2019 SCO Payroll Project Report, Senate Office of Oversight and Outcomes

1/55

Crash Course:Failure to heed earlywarnings, troubles of thepast contributed to payrollsystem collapse

Cali ornia Senate Ofce o Oversight and Outcomes

A report prepared or Senate Budget andFiscal Review Subcommittee No. 4,Senator Richard Roth, Chair

August 12, 2013

Prepared by Jim Sweeney and Dorothy Korber John AdkissonJohn HillSaskia KimDorothy Korber

Jim Sweeney


2/55

Cali ornia Senate Ofce o Oversight and Outcomes August 12, 2013

2


3/55

august 12, 2013

Cali ornia Senate Ofce o Oversight and Outcomes

Prepared by Jim Sweeney and Dorothy Korber

Crash Course: Failure to heed earlywarnings, troubles of thepast contributed to payrollsystem collapse


4/55


5/55

Table of Contents

Executive Summary

Missteps and Lessons Not LearnedA Decade of Challenges

Reports to Legislature Sugar-Coated, Ignored Problems 29

Project Timeline

References


6/55


6


7/55


System Error:It will be difcult to determine, i anyone ever does, exactly whatdoomed the project, the largestpayroll modernization in thenation The system was designedto take over human resourcesand payroll responsibilities or240,000 state employees At

the end, the project had nearlytripled in cost, to $373 million,and was years behind schedule

The expensive misadventurehas once again le t manywondering why in a state thathas given the world Google,Apple, Facebook and Twitter Cali ornia consistently strugglesto modernize its own publiccomputer systems

Executive Summary

When state Controller John Chiang terminated the contract or a newstate payroll system on Feb. 8, 2013, Cali ornia was le t with little to show

or a project that burned through more than a quarter o a billion dollarsover 10 years.

Unlike other big state computerailures, the 21st Century Project

collapsed not once, but twice,despite multiple layers o oversight designed to spot troubleearly and keep the complex andmassive undertaking on track.

A review o hundreds o pages o documents and interviews withmany o those involved show theproject su ered rom lapses indue diligence, a ailure to resolvecore issues raised early and o ten,chronic turnover in leadershipand what may have beenunrealistic expectations.

At the same time, the SenateO ce o Oversight andOutcomes ound a lack o candor

at times with the Legislatureabout the projects di culties. Atsome o its darkest moments, theState Controllers O ce (SCO)delivered upbeat reports to legislators that only hinted o the turmoilchurning within the project.

It will be di cult to determine, i anyone ever does, exactly whatdoomed the project, the largest payroll modernization in the nation.The new system was designed to take over human resources and payroll


8/55


2

responsibilities or 240,000 state employees. By the time the last contractwas cancelled, the project had nearly tripled in cost, to $373 million, andwas years behind schedule.

Yet the so tware provider and prime contractor, SAP Public Services, Inc.,argues that it is un air to call the projects demise a ailure. The globalin ormation technology company was pushing at the end to plow ahead despite persistent errors with an initial live pilot with a second, largerroll-out.

The Controllers O ce and SAP blame each other or the projectscollapse, with the dispute expected to be settled in court. As much as$190 million may hang in the balance up to $135 million the statehopes to recover and $55 million that SAP believes it is owed, accordingto the Legislative Analysts O ce.

The expensive misadventure has once again le t many wondering why in a state that has given the world Google, Apple, Facebook and Twitter Cali ornia consistently struggles to modernize its own public computersystems.

The Senate O ce o Oversight and Outcomes was asked to examine notthe technical details, but where the planning and management processmay have ailed. As such, this report compiles in ormation gathered or anoversight hearing by Senate Budget and Fiscal Review Subcommittee 4,chaired by Sen. Richard Roth, D-Riverside.

While the state has had a number o high-pro le IT ailures, the 21stCentury Project stands out because it has now skidded o the railstwice. Moreover, unlike some other ailed projects, such as the digitalconsolidation o court records, overhauling the states human resourcesand payroll computers is not optional. The existing, Vietnam War-erasystem is maintained and operated by a dwindling number o retirement-age specialists who still understand the outdated computer language. Theoriginal easibility study warned the existing system needed to be replaced

as soon as possible. That was more than 10 years ago.The Controllers O ce issues $1.2 billion in payroll checks every month.Should the existing system crash, the Controllers sta said all o thenecessary data is stored in a backup system that could reproduce thestates payroll. That has never happened, so it is unknown how disruptiveit might be.

I think everyone knows that at some point, that payroll system is goingto blow up or good, warned Vince Brown, who was involved in the


9/55


early stages o the project as chie operating o cer under ormer stateController Steve Westly.

Conceived in the late 1990s, the 21st Century Project was launched inJuly 2003 with a easibility study or a new payroll system or 294,000state employees, which included the Cali ornia State Universitywork orce. The system, named MyCalPAYS, was expected to modernizeand improve management o payroll and bene ts administration whileo ering sel -service access, much like the states MyCalPERS system.

In April 2005, the Controllers O ce signed a contract with SAP oro -the-shel so tware that would be customized to meet the states needs. A year later, the SCO hired BearingPoint Inc. to serve as the systemintegrator, the prime contractor. The $132 million project was to be

nished by June 2009.

Be ore work began, however, there were troubling reports about anotherSAP payroll system newly installed at the Los Angeles CommunityCollege District. In July 2005, just three months a ter the state purchasedits SAP so tware, LACCD o cials were publicly describing theirtransition to an SAP payroll system as horri c.

Eighteen months later, in January 2007, the Los Angeles Uni ed SchoolDistrict experienced similar results when it transitioned to a new SAPpayroll system. The asco paralyzed Los Angeles Uni ed or nearly ayear as administrators scrambled to correct thousands o paychecks andstabilize the payroll.

At the time, the Controllers sta issued public assurances that theywere aware o what had happened at the nations largest communitycollege district and second largest K-12 district, and would make surethe states new payroll system would not su er a similar ate. The SCOsaid it dispatched a team to review what happened in Los Angeles, butits unclear how thorough that review was. When the Senate OversightO ce asked or documentation, the SCO could nd no report or memo

detailing the teams ndings or recommendations.While SAP enjoys a reputation as a global leader in payroll and humanresources systems, IT specialists told the Senate Oversight O ce that itso -the-shel so tware was designed largely or private sector clients, whopayroll and HR systems are much di erent and much simpler than thestates, which has 160 departments and 21 di erent labor or bargainingunits.


10/55


4

The Controllers project team noted that the state used a di erent systemintegrator BearingPoint than the Los Angeles Community CollegeDistrict and Los Angeles Uni ed. As the prime contractor, the systemintegrator does the hardest work, customizing the so tware as necessary tobuild the new system.

In October 2007, as Los Angeles Uni ed continued to grapple with itspayroll, the Controllers O ce served notice that its project also was inserious trouble. State attorneys sent a cure notice and breach o contractwarning to BearingPoint. A ter an attempt to regroup and restart theproject, BearingPoint was red and the project was suspended in January2009. BearingPoint later led or bankruptcy.

The project was restarted in February 2010 with the SAP so tware andSAP under contract as the new system integrator. The estimated costhad more than doubled to $305 million and the completion date wasextended more than three years, to September 2012. The CSU systemalso had been dropped rom the project, reducing the payroll by 18percent, rom 294,000 to 240,000 employees.

Eighteen months later, in August 2011, the Controllers O ce issuedanother cure notice, this time to SAP, threatening to cancel the contractunless a number o issues were resolved. The biggest problem was a

ailure to complete data conversion, a process to enable the trans ero employee data rom the legacy computers to the new system. Dataconversion di culties had dogged the project rom the beginning.

In a telling response, SAP agreed to address the SCOs concerns, butnoted: As you are well aware, there is a disagreement between SAP andSCO as to the contractual responsibility or data conversion. Thus, morethan eight years into the project, the state and its vendor were hagglingover a core responsibility that should have been clearly assigned in thecontract. The dispute was resolved during two months o con dentialnegotiations, a ter which the state agreed to pay an additional $15 millionto bring in a data conversion specialist.

Less than a year later, in October 2012, the SCO issued a second curenotice to SAP and both sides began bracing or a legal ght over whowould pay or a project on the brink o collapse. When SAP issued ademand or mediation on Feb. 5, 2013, the SCO terminated the contractthree days later.

The October 2012 cure notice revealed what looks to be anothersigni cant lapse in due diligence. Nine years into the project, the letter


11/55


questions whether SAP has the ability and experience to deliver such alarge payroll conversion.

The sudden collapse o the project caught legislators, as well as somebudget and policy analysts, by surprise. Some have since questionedwhether the SCO was completely candid with the Legislature about theprojects status.

The Senate Oversight O ce ound that in August 2011, SCOrepresentatives told the Little Hoover Commission that the project wasproceeding as planned and was in the midst o rigorous testing. Whatthe projects leadership didnt tell the commission which includes twolegislators was that the Controllers O ce six days earlier had issued acure notice threatening to cancel the project. Similarly, in its quarterlyreport to the Legislature a month later, the Controllers O ce did notmention the August cure notice.

When the second cure notice was issued in October 2012, legislativeanalysts said they did not nd out about the startling development untildays or weeks later. One said he rst learned o the cure notice throughthe rumor mill.

A review o independent oversight reports that tracked the project alsoshows that key issues such as the states payroll complexity, the need toprepare existing sta or the new system, project sta vacancies and dataconversion di culties shadowed the project rom the outset. Many o those same issues continued to be cited in oversight reports during thesummer and all o 2012 as the project oundered.

The ailure to resolve major issues identi ed early on may refect alack o ownership and requent turnover at the top o the project team.Controller John Chiang inherited the 21st Century Project rom ormerController Steve Westly in 2007. Westly had inherited it rom ormerController Kathleen Connell. I it is resurrected, it will likely be under a

ourth controller. Term limits will orce Chiang rom the o ce a ter nextyear.

In addition, documents show that over the past decade the SCOs teamhad at least ve di erent project directors and our di erent projectmanagers. Over the nal 18 months alone, it had three project directorsand two project managers. The IT vendors, SAP and BearingPoint, alsochanged their project team leaders, although that is more di cult totrack.


12/55


6

As a result o the many changes at the top, the independent overseerswho raised concerns about payroll complexity, sta training, chronicvacancies, data conversion and other risks o ten were taking up thoseissues with an ever-changing audience a new project director, or a newproject manager, either o whom might be working or a new controller.

Finally, the independent oversight reports issued repeated warnings thatthe state was asking or more customization than perhaps it should romso tware sold as a an o -the-shel product not designed to undergoextensive modi cation. One veteran state IT specialist said its notuncommon or IT vendors to promise more than they might be able todeliver in such modi cations. But the Senate Oversight O ce oundbroad agreement that the more a client attempts to modi y commercialo -the-shel so tware, the greater the degree o di culty and likelihood o

ailure.

In the end, the projects leadership also ailed to heed repeated warningsthroughout the nal year about the adequacy o testing designed toidenti y and x mistakes be ore the rst payroll was issued on July 1, 2012.

Lack o quality in testing may result in MyCalPAYS errors and inemployees not receiving correct pay, a March 2012 oversight reportdeclared. The warning proved prescient.

(This report ocuses on just one project and should not be construed to in er that any o the problems cited are similar or dissimilar to those experiencedby other projects. In addition, the report re ers requently to the Cali orniaTechnology Agency, which existed until June 30, 2013, when it wasreconfgured as the Department o Technology. Similarly, CTA o fcialsinterviewed or re erenced or work done while employed by the ormer agency are identifed by their CTA positions.)


13/55


Missteps and Lessons Not Learned

On Feb. 8, 2013, when state Controller John Chiang cancelled a contractto build a new computer system to take over the states 240,000-employeepayroll, he suspended a $373 million project that has been in the works

or more than a decade.

The State Controllers O ce (SCO) had been planning the projectsince at least 1999, when it concluded the states aging computer systemneeded to be replaced as soon as possible. Introduced in the early 1970s,the legacy system is based on dated computer language understood by adwindling number o aging computer technicians.

At the end o a long planning process, the SCO signed a contract in April2005 with SAP Public Services, Inc. or o -the-shel so tware that wouldbe customized to accommodate the states needs. At the same time, theSCO began the search or a system integrator to modi y and deploy theso tware.

SAP is a global leader in providing payroll and human resources so tware.But many o its clients are private sector rms with payroll systemsthat are simpler than those o government agencies. Few payrolls public or private are believed to be as complicated as Cali ornias,which maintains historic data or roughly 300,000 employees in 160departments with 21 di erent labor or bargaining units.

The Controllers O ce and SAP said several states Maryland,North Carolina, Pennsylvania and South Carolina have success ullyconverted to SAP systems. But the SCOs project team said recently thatit was unsure whether any o those states have payrolls as complex asCali ornias.

About the time the Controller was shopping or payroll so tware, severalother public employers were experiencing major problems with new SAPsystems. Three months a ter the state purchased SAP so tware, the Los Angeles Community College District the nations largest communitycollege district was sounding an alarm over its new SAP payroll system.


14/55


8

The districts July 2005 changeover to an SAP system was horri c,according to the districts payroll manager.

Three months later, on Oct. 4, 2005, Los Angeles Uni ed School Districtsigned a $95 million contract or SAP payroll, nancial managementand procurement so tware to be modi ed and integrated by DeloitteConsulting.

On Oct. 6, 2005 just two days a ter LA Uni ed signed its SAP-Deloittecontract the Irish Health Service gave up on two SAP systems, one o which was to handle payroll and related systems or 120,000 employees,hal the size o the states payroll. The Irish Health Services SAP payrollsystem had been expected to take three years and $10.7 million tocomplete. It was scrapped 10 years and $180 million later.

LA Uni ed, the nations second largest school district with nearly 100,000employees at the time, began using SAP so tware or payroll in January2007. The results were disastrous, with thousands o employees receivingincorrect paychecks. It would take more than a year to stabilize Los Angeles Uni eds payroll in what became an ongoing public spectaclethat was not lost on the Controllers O ce.

In February 2007, when asked about the pending conversion o the statespayroll to the same brand o so tware involved in the spectacular ailuresat LA Uni ed and the community colleges, state o cials said they haddispatched a lessons learned team to Los Angeles to make sure they didnot repeat mistakes made there.

The State Controllers O ce has de nitely been paying attention towhats going on and is on top o this, an SCO spokesman said at thetime.

Asked recently about the lessons learned rom the Los Angelesexperiences, Collin Wong-Martinusen, the controllers chie o sta , toldthe Senate Oversight O ce that the SCO made two signi cant changes.

It established a steering committee as a new layer o oversight anddecided to phase in the SAP so tware, rather than deploying it in a singlebig bang introduction as Los Angeles Uni ed did.

But the Controllers O ce could nd no memos or reports that wereprepared by its lessons learned team, either rom its interaction with thecommunity college district or Los Angeles Uni ed.

Two years a ter the SAP rollout at LA Uni ed, a civil grand jury had toresort to legal arm-twisting to persuade the district to give up pertinent


15/55


documents and other in ormation about the asco. Following severalailed attempts over a ve-month period and the threat o a subpoena,

LAUSD produced the documents the (grand jury) determined to beuse ul and relevant, the grand jurys report said.

The grand jury report disclosed that the botched conversion to SAPso tware resulted in an overpayment o $60 million to some 35,000 LA Uni ed employees. In late 2009, nearly three years a ter the conversion,approximately $9 million o that overpayment still had not been collected

rom 2,400 employees, the grand jury ound.

In its report, akin to a lessons learned analysis, the grand jury said that despite assertions to the contrary simulated parallel payroll runs couldhave been executed and compared to existing payroll. This might havebeen a lesson that could have averted the epic problems o the states rstlive payroll run.

A thorough review o the LA Uni ed experience also would have revealedthat, despite the district spending an extra $40 million to address andcorrect all o the payroll problems, a third and nal phase o the SAPconversion was postponed inde nitely in May 2009. It was not completeduntil last month (July 2013). As a result, the district continued to bear theexpense o operating and maintaining its legacy system or procurementactivities, according to a December 2009 report prepared by the districtsinspector general.

Two months a ter the release o that inspector generals report, theSCO signed a new contract with SAP to serve as the system integratoro its so tware or the state. The projects original system integrator,BearingPoint, had been red a year earlier.

Wong-Martinusen, the controllers chie o sta , said that although SAPso tware was the common denominator in the troubled new systems at thecommunity college district, LA Uni ed and the state, those projects useddi erent system integrators, which served as the prime contractors.

When the state red BearingPoint, the state did not have any problemswith the so tware, Wong-Martinusen said. Regardless, he said, di culttransitions and ailures are not uncommon in the IT industry.

For any major IT vendor, you could easily nd a dozen public or privateailures, he said.

As part o a pending assessment o the 21st Century Project, the state plansto reassess the so tware, to review why it was selected initially, whether it


16/55


10

can be modi ed to accommodate the states complex payroll and whetherit has been success ully customized or other states with similarly complexpayroll systems.

Was the SCOs due di ligence adequate?

Other excerpts gleaned rom hundreds o pages o documents reviewed bythe Senate O ce o Oversight and Outcomes raise additional questionsabout how well the Controllers O ce reviewed vendor proposals when it

rst embarked on the 21st Century Project in 2003, and again in 2009-10,when it sought to salvage the project a ter ring the rst system integrator,BearingPoint.

When the SCO dismissed BearingPoint in January 2009, one o the keyactors the state noted was the companys ailure to develop accurate,reliable data conversion programs, scripts and computer instructions thatwould allow the new system to incorporate and use the historical datastored in the SCOs legacy computers.

Data conversion is one o the most crucial aspects o a success ulproject, the Department o General Services wrote in a Dec. 3, 2008,letter that served as a notice o intent to terminate the BearingPointcontract.

Yet, three years later eight years into the 10-year project dataconversion remained a major stumbling block and the new systemintegrator, SAP, was disputing who was responsible or it.

As you are well aware, there is a disagreement between SAP and SCO asto contractual responsibility or data conversion, an SAP executive wrotein an Aug. 21, 2011 response to another cure notice, the second issued onthe project.

The statement raises the question: Why, on a project o such magnitudeand complexity, would a critical core task such as data conversion not be

clearly assigned in the contract?SAP and the SCO ultimately agreed to bring in a third party, BackO ce Associates, to do the data conversion. The state agreed to pick up theadded $15 million cost, although the payment is among those the statewithheld when it cancelled the SAP contract.

Finally, in a third cure notice, dated Oct. 25, 2012, the SCO raisedquestions about SAPs ability to deliver and the companys experiencewith such a large payroll conversion approximately 240,000 state


17/55


1

employees. In May 2012, the cure notice reveals, the SCO asked thecompany to identi y an SAP client where 191,000 employees aresuccess ully processed in a single payroll area using SAP so tware.

Leadership turnover a problem from the outset

State Controller John Chiang inherited the 21st Century Project rom aprevious controller, Steve Westly, who also inherited it rom a previouscontroller, Kathleen Connell. In all likelihood, Chiang who is termedout next year will hand o the remains o the project to his successor.

The SCOs original project director and manager le t shortly a ter theproject was launched. Early oversight reports starting in 2006 also warned

repeatedly about prolonged vacancies in top positions.In addition, documents show the SCO had at least ve di erent projectdirectors and our di erent project managers over the course o the 10-year project. During the nal 18 months alone, it had three di erentproject directors and two project managers. The last project director tookover on March 13, 2012, just three months be ore the June 11 go-livedate. The vendors, SAP and BearingPoint, also changed leaders on theirteams, although that is more di cult to track.

The independent project oversight consultant (IPOC) position, a key

observer embedded with the project, turned over three times in the last,tumultuous year o the project. In addition, the SCO, the IPOCs andan independent veri cation and validation (IV&V) consultant all citedturnover at the top o the vendors team as a serious problem.

The constantly changing cast o leaders le t ew, i any, with realownership or start-to- nish responsibility or the project. Most likely knewthey would be gone be ore the project was nished or any other pointwhere they could be held accountable.

Procurement process critical to successful project

As the Controllers top sta refected on the demise o the 21st CenturyProject, Chie Administrative O cer Jim Lombard suggested the newpayroll system may have been doomed rom the start.

It comes back to procurement not bringing the right vendor on boardto do the job, Lombard said.

The Controllers O ce never had many vendor choices at any stage o the project. Just two companies bid on the so tware contact in 2004. Only


18/55


12

one bid to be the rst system integrator in 2005 and only three bid on thesecond system integrator contract in 2009.

Wong-Martinusen, Chiangs chie o sta , said the project has persuadedthe Controllers O ce that we need to treat IT procurement di erentlythan we treat procurement or other goods and services.

The existing system is driven by a risk-adverse mentality designed toprotect the state rom costly, time-consuming protests rom bidders,Wong-Martinusen said. The rigid process was developed to insure that allbidders are treated equally. But it can also prevent the state rom pressinga bidder or added in ormation in critical areas, such as proposed plansand experience, Wong-Martinusen said.

A ter the initial system integrator ailed, the state paid SAP and a secondbidder, Accenture, $500,000 each to demonstrate in a bakeo howthey would design a new system to take over the states payroll. Duringthat process, one o the Controllers consultants warned that SAPs datamigration tool a critical component in the modernization o a payrollsystem as large and complex as Cali ornias appeared to be inadequate

or the daunting project. But the states procurement rules prevented theControllers team rom questioning SAP about the data migration tool,said Tony Davidson, the SCOs current project director.

When the Controllers O ce issued its rst cure notice to SAP in August 2011, threatening to cancel the contract, the step was takenbecause o the companys inability to migrate data, according to anSCO document.

A ter the BearingPoint ailure, the Controllers O ce asked i it couldtry the success ul strategy Caltrans has used on some projects bonuses

or early completion and nancial penalties or delays. Attorneys atthe Department o General Services concluded state law did not allowthat or IT projects, SCO o cials said. The state also has taken steps,including the repeal o a per ormance bond requirement, to osterincreased interest and bidding on IT projects.

In response to a number o recent IT ailures, the state last monthtrans erred the responsibility or IT procurement, along with a smallteam o specialists, rom General Services to the new Department o Technology.


19/55


1

Dirty data, few records to decipher original coding

The Controller issues paychecks or employees o 160 state departments.Most, i not all, o those departments have or have had the ability to entertheir own employee data into the legacy system. This has resulted in anunusual amount o unreliable or dirty data, such as employee start andexit dates rom di erent departments. (None o those interviewed orthis report said aulty data contributed to paycheck errors. But a payrollcomparison done or the project did discover that the legacy system wasdeducting too much rom some employees paychecks or state disabilityinsurance. An extra $14 million had been collected rom 100,000employees dating back to 2005. The Controllers O ce has startedre unding that money.)

The legacy system also contains applications, or code, that was developedas needed over the years with little record o how the code was written orwhat it says. That compounded the di culty o converting the massiveamount o historic data into a orm that could be used by the new system.

Everyone knew the data in the Controllers system was bad raughtwith errors ed in rom all o the di erent departments, said VinceBrown, CEO o the Alameda County Employees Retirement Association.Brown worked on the project as chie operating o cer under ormerController Steve Westly.

Adequacy of testing

As early as December 2010, an independent oversight consultantembedded in the project began warning that testing had allen behindschedule. A year later, both o the projects independent oversightconsultants (IPOC and IV&V) were raising concerns about delays andshortcuts taken in testing.

We were concerned about not having enough test scripts to vet, becausewe knew our payroll is complex. ... We were concerned the test plan

didnt appear thorough enough, said Mary Winkley, an assistant secretarywith the Cali ornia Technology Agency (CTA), which employed theoversight team.

The testing concerns raised in January 2012 were repeated every monthor the balance o the year as the project team pushed toward the rst live

test on June 11. Because testing typically occurs near the end o a project,it is o ten something that gets condensed and shortchanged, Winkley andothers said.


20/55


14

The independent oversight reports sounded increasingly urgent alarmsabout the state o testing as the project approached the June 11 go-livedate. Three months out, an oversight report cautioned that a lack o quality in testing may result in MyCalPAYS errors and in employeesnot receiving the correct pay. A month out, an oversight report warnedominously: The project has had quality challenges in documentingtesting results, which makes it di cult to ascertain i testing actuallyoccurred and the results o testing.

In its October 2012 cure notice, the SCO fagged testing lapses as a keyactor in the payroll errors experienced in Pilot 1.

SAP ailed to develop, plan and, critically, execute a comprehensive testprogram, the SCO alleged. Among the testing lapses, the SCO citeda ailure to execute a complete payroll test which led to undetectedcon guration errors and contributed to paycheck errors. The SCOsletter stated that it had in ormed SAP multiple times that its testing wasinadequate.

In its response, SAP maintained that Pilot 1 was a success, that mosto the mistakes resulted rom user errors by poorly prepared andoverwhelmed SCO employees rather than system de ects or which SAPwas responsible.

As with data conversion, the Controllers O ce and SAP disagreedsharply about who was responsible or testing.

Did the state seek too much customization of thesoftware?

The SCOs project team has maintained that contracts or the 21st Century Project always required the new system to accommodate thestates complex payroll practices. But the two system integrators bothcomplained that the state was trying to replicate its legacy system withexcessive customization o the SAP so tware.

The SCOs project team said it worked with the state Department o Human Resources to simpli y state practices when possible, but couldnot change many other practices, some o which are required by law orlabor agreements. At the outset, and again when the project was restartedin 2010, SCO o cials said, SAP assured the state that its so tware couldhandle or be customized to accommodate all o the states businessespractices.


21/55


1

In the nal design o the new system, the Controllers O ce saidSAP agreed that 283 customizations would be required. At projecttermination, 251 o those customizations were in operation, said TonyDavidson, the SCOs current project director.

In written responses to questions rom the Senate Oversight O ce,SAP said the Controllers customization o the base so tware wasextraordinarily high compared to other large SAP payroll systems.During the course o the project, SAP said the Controllers O cerequested 359 changes, including 126 a ter the June 11 go-live date, tothe base so tware or the original design.

The 21st Century Project should have been an opportunity to correct theerrors that had accumulated over decades o separately built, separatelymaintained legacy payroll systems operated as a patchwork to servehundreds o state agencies and departments, SAP said in its responses tothe Senate Oversight O ce.

The independent oversight reports warned repeatedly that the state wasasking or more customization than perhaps it should rom the so tware.The Senate Oversight O ce ound broad agreement that the more aclient attempts to modi y commercial o -the-shel so tware, the greaterthe degree o di culty and likelihood o ailure.

Institutional resistance

Big IT projects o ten run into serious trouble when existing payroll andhuman resources sta get involved, several veteran IT specialists told theSenate Oversight O ce. Employees o ten do not want to change long-established practices and will insist they cannot do their work in the newway. In addition, the state o ten doesnt have enough time or resources toproperly train sta .

As an added disincentive to the rank and le, new IT systems o ten areexpected to produce e ciencies, meaning ewer jobs. The 21st Century

Project was no exception, promising sta reductions through attrition.I you cant overcome that problem, i people dont want to go to thenew so tware, you can get into some very di cult situations, said TeriTakai, chie in ormation o cer at the Department o De ense and thestates CIO rom 2007 to 2010. Thats generally a big part o ailures in industry or government.

SAPs response to the Controllers cure notice alleged that state humanresources personnel o ten re used to actively participate in SAP training


22/55


16

sessions. As evidence, SAP quoted an email rom Lisa Dean, theControllers work orce transition leader.

General eel in the class is one o negativity and hostility Oneparticipant slept quite a bit One participant very rustrated, not willingto try and learn Im at a loss or the lack o respect being shown inclass, Deans May 2012 email said.

No parallel payroll test

The SCOs team said it relied on assurances rom SAP that three payrollcomparison tests all but guaranteed that, a ter accounting or knowndi erences, the payroll would run within $1 o the legacy system ormost employees.

We did expect problems, but nothing o the magnitude experiencedwhen we turned it on, said Jim Lombard, the SCOs chie administrativeo cer.

It is expensive and labor intensive to run an actual parallel payroll test,which requires all o the sta time to do an actual payroll, the SCO teamexplained. But a parallel payroll test would have allowed the SCO andSAP to analyze mistakes and work to correct them in a calmer, clinicalsetting, without hundreds o angry SCO employees clamoring orimmediate corrections.


23/55


1

A Decade of Challenges

Monthly reports prepared by independent oversight consultants showthe 21st Century Project su ered rom delays and challenges rom thestart, new problems that emerged immediately a ter it was resurrected inJanuary 2010 and other serious issues that, in some cases, dragged on ormonths.

The independent project oversight consultants, or IPOCs, were employedby the Cali ornia Technology Agency (and its predecessors). They wereembedded in the project with onsite work space and access to meetingsand documents and led reports with the agency and the project team.It is important to note that issues cited in the reports were those thatremained unresolved a ter in ormal discussions between the oversightconsultants and the project team. The reports were known as IPORs, orIndependent Project Oversight Reports.

The oversight consultants worked with a second, similar overseer knownas the independent veri cation and validation (IV&V) consultant.The latter, employed by a private company under contract with thestate, ocused on technical issues, while the IPOCs largely reviewedmanagement o the project.

The ollowing summaries and excerpts rom key IPOC reports show the21st Century Project was plagued by known challenges identi ed at theoutset and that continued, in some cases, or a decade. These basic, coreproblems included the states payroll complexity, data conversion, sta vacancies, organizational change management (the process o teachingand selling the new system to existing sta ), the absence o a qualityassurance team, and an inability to stay on schedule:

July 2004.Projects rst independent project oversight report(IPOR) questions whether the state will be able to continueits semi-monthly pay schedule in the new system. The payrollschedule and complexity will remain an issue throughout theproject.


24/55


18

(Eight years later, as the project is overwhelmed by errors in its frstlive pilot, an IPOR concludes the payroll calendar complexity maycreate an unsustainable operations model.)

August 2004.Projects second IPOR warns, in a telling prediction,that ailing to gain timely union agreement and acceptance o new payroll system changes could increase the scope o the project,require more time and extend the implementation schedule (andcost).

October 2004.Five months in, the project is already severalmonths behind schedule.

July 2005. New independent oversight consultant sounds anearly warning about potential high risk the project aces inorganizational change, the teaching and selling o the newsystem to o ten-resistant existing sta .

A ully implemented 21st Century Project potentially impacts thebusiness processes o every state entity, and the methods used byevery state employee to record time and manage leave. Continuouscommunications with all project stakeholders will be crucial toproject success, IPOR says.

(Seven years later, as the project careens toward ailure, an IPORconcludes the level o organizational change management andtraining had been insu fcient to prepare users or the new system.)

December 2005. Selection o a system integrator, to be theprojects prime contractor, has allen up to nine months behindschedule. IPOR also notes high risks that:

The state may not be able to recruit required sta or theproject.

The state team may not be able to make timely decisions onproject design due to a lack o knowledge or authority.

August 2006IPOR identi es a number o developing problems:

Un lled State Controllers O ce (SCO) positions or theproject are o great concern. Project business manager andtechnical manager positions are vacant.


25/55


1

With key decision makers not available, delays in theapproval process are inevitable, the report states.Additionally, the project could venture down thewrong path or some time be ore it is noticed and coursecorrections are made.

Without the appropriate expertise available to reviewproduct deliverables, quality assurance will su er.

Project delays are causing work to be rescheduled resulting inunknown long-term impacts.

November 2006IPOR repeats many o the same warnings withadded emphasis in some cases:

Speci c positions un lled are o great concern both theSCO project business manager and technical manager positionsremain vacant. The 21st Century Project will be at risk untilthese key positions are lled.

Oversight team remains very concerned with schedulemanagement.

Poor quality assurance practices could result in delivery o asolution that does not satis y all project requirements.

As with the project schedule, we have reported on theseconcerns repeatedly.

(Seven years later, when the state cancels a second contract or the project, oversight reports were still warning there is no qualityassurance team.)

March 2007IPOR suggests project is in serious trouble:

Project is behind schedule and much o the work continues tobe completed later than planned.

In an ominous orecast, report states that because onlya limited number o departments participated in blueprintworkshops, critical requirements may not have been captured Missed requirements could result in incorrect data in theSAP systemwhich has the potential to prevent employees rombeing paid correctly(emphasis added).


26/55


20

Project rebaselined with increased budget, extendedcompletion date

May 2008. With the project ar behind schedule, all involved agreeto rebaseline it, increasing the cost to $179 million and extendingthe completion date by a year, to June 2010.

Seven months later, inJanuary 2009, the state terminates thesystem integrators contract. Failure to develop accurate, reliabledata conversion programs and scripts are a key actor. Dataconversion problems would continue to haunt the project.

Second attempt to build 21st Century Project

encounters immediate problems February 2010. The project restarts with SAP as the new system

integrator. Scope has been narrowed by dropping the Cali orniaState Universitys 54,000 employees. Cost increases to $305million and completion date is pushed back another two years, toSeptember 2012.

February 2010.Oversight report or the rst month since therestart warns the project already is experiencing challenges:

Three deliverables have been rejected and three others,including the project schedule and deployment strategy, havebeen delayed.

SAP submits change requests to address the late deliverables. Inan early sign o confict, the SCO rejects the change requests.

SAP prepares an in ormal project recovery plan to outline latedeliverables and establish new due dates.

SAP has only 18 o 24 planned sta on site. This sta ng

shortage is having an impact on completion o majordeliverables, report states.

March 2010IPOR notes that, two months into the restartedproject:

There still is no project schedule. The SCO submitted 505comments outlining de ciencies in schedule submitted by SAP.


27/55


2

SCO and SAP sta cannot move into the project work spaceuntil Aug. 1, six months a ter the project restart.

June 2010IPOR delivers blunt warnings about delays, shortcutsand critical vacancies:

Report fags a major faw in the Requirements TraceabilityManagement Plan (RTMP), a key document that describesthe process that will deliver the agreed upon scope to meetproject requirements. The RTMP discusses trackingproject requirements, notbusinessrequirements (emphasis added).

Project requirements are at a higher level Businessrequirements are the lowest level o unctionality that isrequired to be built.

It is common and accepted best practice to separately identi yand track each business requirement through development,unit and integration testing and back. That is not being donehere (emphasis added).

September and October 2010IPORs fag SCO vacancies in threecritical project positions.

December 2010IPOR nds:

Project director has split project management between twopeople, blurring responsibility or some critical tasks, such asorganizational change management (OCM), the teachingand selling o the new system to existing sta . The OCMprocess is essential to a smooth transition. Twice, the IPOR warns, the divided project leadership may be a challenge orSCO to manage.

SCO still has three critical vacancies.

January 2011IPOR discloses the SCO has requested signi cantchanges to the system, the total number and complexity o whichis not known. Report warns the late changes to be detailed in apending Change Request 34 will delay the project and increasecosts. Among the consequences, a completed test deliverable mustbe reopened and a new testing cycle developed.


28/55


22

February 2011IPOR outlines several major new concerns:

For the rst time since the project was restarted a year earlierwith SAP as the system integrator, the oversight consultantsounds an alarm over the state o data conversion.

Legacy data conversion has experienced signi cant delays andtechnical di culties. Further delay will result in signi cantimpact to testing cycles.

The project has moved into unchartable territory, with noobjective measurement to gauge impacts that changes soughtin Change Order 34, and the added testing required, will haveon the projects critical path, a sequence o activities that mustbe completed on time or the project to nish on schedule.

Major departments CalFire, the State Water ResourcesControl Board, Caltrans and, surprisingly, the Cali orniaTechnology Agency are behind schedule and at serious risko missing deadlines to develop inter aces to interact with thenew system.

The SCO still has three vacancies in critical positions, ouryears a ter Chiang took over the project.

March 2011IPOR reveals a signi cant shortcut taken as a result o the data conversion troubles:

To prepare or a payroll comparison test, the project attemptedto convert legacy payroll data or 22,000 records chosento refect populations o the Pilot 1 and Pilot 2 departments, aswell as a representative sample o the remaining state employeepopulation.

This legacy data extraction, per ormed by the SCO, hadlimited success. As a result, the extraction resulted in about5,800 records (26 percent o the planned 22,000 sample) beingconverted. The SCO determined it was a good enoughsample to proceed with Payroll Comparison Test 1.

However, the report warned, it is anticipated that, o the5,800 records, a signi cant number may drop out as the dataconversion progresses.


29/55


2

April 2011IPOR reveals the project has been brought to its kneesby data conversion challenges:

A data conversion deadline, already missed, is pushed backanother two months.

For the rst time, oversight consultant warns the project may beunable to meet planned Sept. 1, 2011 go-live date.

Eclipse Solutions, which analyzed the data conversion problemsat the request o the Cali ornia Technology Agency, issues ajarring assessment that cites critical discrepancies:

These include, but are not limited to, inaccurate mappings,incomplete conversion cycles a lack o ocus and priority lack o ormal review and signo or critical arti acts,lack o clear communications, a lack o collaboration,overloaded resources (project sta ), and a lack o adequatemanagement involvement.

Project team starts developing another project recovery plan.

SCO still has not lled all o the critical project vacancies,with one remaining.

Controller suspends second attempt to complete the21st Century Project

On Aug. 19, 2011, the SCO issued a cure notice to SAP, threatening toterminate the contract largely because o the inability to resolve the dataconversion problems. SAP responded with a letter that declared: As youare well aware, there is a disagreement between SAP and the SCO as tothe contractual responsibility or data conversion.

Two months o con dential negotiations produced a Nov. 17 settlement

agreement to resolve most di erences and continue the project withsigni cantly higher state costs and a one-year schedule extension. Addedexpense included $15 million or a specialized vendor to take over dataconversion.

December 2011IPOR, completed just one month a ter the mid-November restart, immediately sounds new concerns, includingsigni cant new test quality issues uncovered.


30/55


24

February 2012IPOR repeats the warning about testing and raisestwo more major issues:

It is unclear whether Payroll Comparison Test 2, to comparelegacy payroll results to those produced by the new system,

ully covers essential actors such as wage types and bene ts.With ewer SCO employees than planned, substantially lessrecords (were) manually validated by expert end-users thanplanned.

The project has stopped monitoring progress in knowledgetrans er, the comprehensive preparation and training necessaryto enable state workers to take over operation o the new systema ter launch.

March 2012IPOR, covering the three months since theNovember restart, notes the project continues to have di cultiesdocumenting testing results, and urther warns: The projectlacks quality assurance sta , which has had an adverse impact onensuring quality in testing, coding and con guration.

In a prescient admonition, the reports states: Lack o qualityin testing may result in MyCalPAYS errors and in employeesnot receiving correct pay.

(From the outset, oversight consultants warned that the project hadno ormal quality assurance practices or quality assurance team.)

April 2012IPOR ampli es warnings about test quality andshortcuts, knowledge trans er and SCO sta vacancies, and adds anumber o other key concerns, including:

Continued overlap o multiple test cycles is stretching projectsta .

Second April 2012IPOR, prepared by a new independentoversight consultant, issues rank warnings about testing andtraining o SCO sta or the transition:

One phase o critical per ormance testing was unsuccess ul andanother could not even be attempted because o systemlimitations. Given that the ull per ormance testing had notbeen completed, the (oversight consultant) could notcon rm that the current sizing o the system was su cient or

ull deployment.


31/55


2

(In other words, a little more than a month be ore the June 11launch o Pilot 1, the oversight team couldnt say or sure that thesystem ultimately could handle the states ull payroll.)

An employee survey clearly shows most o the unctionalteam members have not attended SAP training courses onhow to operate the new system. In addition, SCO projectvacancies include the organizational change manager, a keyposition to guide o ten-resistant existing sta in the transitionto the new system.

May 2012IPOR, prepared less than two weeks be ore the June 11launch o Pilot 1, adds several new key concerns to those that hadbeen repeated and le t unresolved, in some cases, or months:

The project team has made program-level decisions withregard to how many pay cycles may be run and whether theyneed to be closed out be ore the next is initiated. These businessdecisions mimic existing legacy unctionality and allowfexibility but are not aligned to the way SAP is designedto operate and will create complexity and extra workload oroperationsthat could prove unsustainable (emphasis added).

The project has had some quality challenges in documentingtesting results,which makes it di fcult to ascertain i testingactually occurred and the results o the testing (emphasis added).

It is unclear what the need and plan is or a ter-hours supportduring the rst cycles o production runs.

The Pilot 1 back-out plan is incomplete and not wellcommunicated. While the probability o the plan beingexecuted is low, the impact o ailure to restore is high.

Pilot 1 errors doom project

A ew weeks later, the Pilot 1 payroll to some 1,300 SCO employees less than 1 percent o the work orce or which the system was designed contained hundreds o errors in pay and deductions. The errors causedan internal uproar and proved so di cult to correct over eight monthsthat the SCO suspended the live test and, on Feb. 8, terminated thecontract with SAP. The termination came three days a ter the statereceived a demand rom SAP to mediate di erences over the causes andresponsibility or the errors.


32/55


26

Immediately a ter the Pilot 1 launch, the independent oversightconsultant raised issues that essentially repeated and con rmed warningsthat had shadowed the project rom the outset.

June 2012IPOR notes con usion between the SCOs humanresources sta and HR sta at other state departments over themanagement o miscellaneous deductions.

Report warns bluntly: I the payroll calendar and legacy operationsare not simpli ed, the level o complexity may overwhelmMyCalPAYS and legacy operations sta .

Source: State Controllers O ce, which noted that a de ect may impact a single employeeor all employees. The Controllers O ce said the chart refects a conservative view o system de ects SAP was required to correct. SAP contends that most o the Pilot 1 problemsresulted rom user errors, committed by state employees.


33/55


2

July 2012IPOR restates testing concerns with the most revealingand pointed warning yet: Decisions were made during the testingphase that (it) was impractical to test every possible paycheckscenario, so test scripts were not run or every possible scenario. Itis unclear whether that decision may have missed detecting de ectsand/or missing unctionality.

August 2012IPOR portrays a project on li e support and revealspreviously unreported problems:

Pilot 2 postponed, and project team to re-examineproblemsin most o the basic development areas organizational changemanagement, schedule, testing, payroll/operations and usertraining (emphasis added).

Organizational change management and training isinsu cient to prepare users or the new system.

Payroll calendar complexity may create an unsustainableoperations model as more waves are rolled out.

A relatively large number o new issues were added to the(issue) log this month. This is because a number o existingissues were being tracked outside o the issue log and notnecessarily refective o how many issues were actually openedthis month.

(The August IPOR was prepared by a new oversight consultantwho had assumed the assignment in May, just be ore the Pilot 1launch.)


34/55


35/55


2

Reports to Legislature Sugar-coated,Ignored Problems

A ter the payroll systems rst collapse, Sen. Denise Ducheny thenchairwoman o the Joint Legislative Budget Committee asked theController to give the Legislature regular updates on the 21st CenturyProject. In response, Controller John Chiang committed in a February2010 letter to provide written quarterly reports, as well as brie ngsto legislative sta at major project milestones and ad-hoc brie ngs atthe request o the Legislature. The rst o those quarterly reports wassubmitted to the joint budget committee in April 2010.

Lawmakers wanted to more closely monitor the beleaguered project,which was overdue and over budget. But a review by the Senate OversightO ce ound that the quarterly reports o ten lacked candor, sugar-coatingsome problems and ignoring others. This ailure to be transparentcompromised legislative oversight and stymied accountability.

That nding is based on a comparison o the legislative reports withdocuments prepared by others monitoring the project. Every month, theCali ornia Technology Agency issued a detailed Independent ProjectOversight Report (IPOR) which ticked o per ormance measurementsand potential problems in the 21st Century Project. These were pairedwith more technical memos rom an Independent Veri cation and Validation (IV&V) team. Those monthly reports along with others thatdocumented problems provide the basis or comparison.

Collin Wong-Martinusen, the Controllers chie o sta , also noted thatproject managers brie ed sta rom the Legislative Analysts O ce 19times in 2011 and 22 times in 2012. Many o these touch points weretelephone conversations. It is impossible or the Senate Oversight O ceto review these oral communications. The quarterly reports were theControllers o cial updates to the Legislature and, as written documents,can be scrutinized.

***


36/55


30

So tware giant SAP Public Services, Inc. took over as integrator o the21st Century Project on Feb. 2, 2010. By January 2011, the Cali orniaTechnology Agencys independent overseer was expressing seriousconcerns about data conversion, calling it the projects top priority. Thiswas not conveyed in the Controllers quarterly report to the Legislature,however. In the February 2011 report, Chiang wrote that the projecthas made signi cant progress. The words data conversion were notmentioned in the 15 Vital Signs or the 5 Risk Pro les, two matrixes usedto mark progress.

In the Controllers April 2011 report to the Legislature, data conversion islisted as a high priority issue, but it is characterized as largely contained:SAP retained data management consulting rm, Utopia, to help expeditethe process. The situation was described very di erently by Robert Eich,the technology agencys independent oversight consultant, in his Aprilreport. Eich said a corrective action plan was being developed to allowthe project to move orward in light o signi cant delays caused by dataconversion. Meanwhile, he rated seven areas as acing high risk and saidnumerous contract deliverables were behind schedule.

Things were to worsen later that summer. On August 19, 2011, the StateControllers O ce (SCO) sent a strongly worded cure notice to SAP,initiating termination or de ault procedures against the contractor.The main issue, again, was data conversion. The Controller accusedSAP o breach o contract and lack o good aith. According to thenotice, SAP leaders had ailed to show up at high-level meetings to dealwith the problems. The project was running ar behind schedule 24deliverables were overdue. And, contrary to the April quarterly report tothe Legislature, SAP had still not secured the services o a quali ed dataconversion vendor by June 1, 2011.

John Hiber, the Controllers project director, wrote this in the Augustcure notice: While the SCO continues to work in good aith underthe terms o the contract, to date SAP has not responded to the SCOsrequests to resolve issues o SAPs ailure to per orm, and little progresshas been made toward achieving the goals agreed upon during the June27, 2011, meeting.

The slow pace o the negotiations was also a major concern or Eich, theoversight consultant. The extensive timeline to nish the negotiationsbrings additional risk to the project, Eich wrote in his August 2011report. He listed those risks: Signi cant additional costs. Sta retention.Loss o momentum. Damage to sta morale.


37/55


3

In the midst o all this, however, SCO sta ers conveyed a rosy pictureduring a act- nding hearing be ore the Little Hoover Commission,which includes members o the Legislature.

On August 25, 2011 just six days a ter the cure notice was issued the commission held a public hearing on in ormation technologygovernance. A portion o the hearing ocused on the progress o the21st Century Project. Project Manager Laurye Gage testi ed or theControllers O ce.

In written materials presented to the commission, Gage acknowledgedthat data conversion was more complex and labor intensive thanoriginally anticipated. But she also wrote: The Project has consistentlybeen ully sta ed on both the State and SAP teams. This statement iscontradicted by Eichs and other oversight reports, which repeatedlyidenti ed sta vacancies as a serious and ongoing risk.

Nothing was reported to the commission about the cure notice or therancorous relationship with vendor SAP. In act, according to Gage,the relationship between the SCO and SAP was a positive: Co-locatingthe state and contractor team, with counterparts jointly reporting status,combined with leadership example, has helped to orge a positive, results-oriented team culture.

In a similar vein, the Controllers quarterly report to the Legislature inSeptember 2011 was positive overall. It did cite data conversion as asigni cant problem that would delay the project six to 12 months. Butnothing was said about the cure notice or a lack o good aith on the parto the contractor. Instead, the report cites substantial progress.

Over the next six months, the SCOs reports to the Legislature continuedthe upbeat theme. The April 2012 report, or example, said dataconversion was working well and the project was on track or the rstpilot: creating actual paychecks or 1,300 SCO employees beginning thatJune.

Each legislative report includes a scorecard o 15 Vital Signs, with agreen, yellow or red box indicating the relative status o each. The colorsare like tra c signals: go, caution, stop. In April 2012, this matrix showedno red boxes and just one yellow (Unresolved Issues, which was ratedlate with no impact). Other Vital Signs like Team E ectiveness,Strategy Alignment, Vendor Viability and Technology Viability glowedgreen, green, green. (See 3/31/12 Project Scorecard on pages 34 and 35,which was part o the April 2012 report to the Legislature)


38/55


32

The technology agencys independent observers were not so optimistic.On March 6, 2012, the agencys oversight report noted that thetechnology team was unable to veri y some test results. It said the impacto inadequate tests could be pro ound: Lack o quality in testing mayresult in MyCalPAYS errors and in employees not receiving thecorrect pay, wrote oversight consultant Eich.

A month later, in the April 30 report the same date as the legislativereport with all those green lights oversight consultant Chi Emodi redanother warning shot. This time the issue was training: The latestemployee survey clearly shows that most o the unctional team membershave not attended the SAP Training courses. It is still not clear howand when the SCO payroll unctional resources will receive knowledgetrans er.

On June 11, 2012, the 21st Century Project hit its Waterloo. That wasthe start o Pilot 1, when the new payroll system was tested on SCOemployees. When the actual paychecks rolled out on July 1, top SCOo cials said in an interview, the number o mistakes le t them shockedand dismayed. Wong-Martinusen, the Controllers chie o sta , said,We fipped it on and the number o errors were unsettling. SAP was asfabbergasted and surprised as we were.

The payroll was riddled with mistakes, despite years o work and hundredso pages o risk identi ers, key concerns, top issues, project scorecards and all those optimistic reports to the Legislature. Heres how theLegislative Analysts O ce described the problems:

Incorrect paycheck deductions were made, payroll and pension wageswere erroneously calculated, and medical bene ts were denied orsome employees and their dependents. In one case, employees thattook vacation time during the rst payroll cycle received compensationin addition to their base salary. Attempts to correct these errors created

urther problems in the ollowing payroll cycle.

On Oct. 25, 2012, the Controller sent another cure notice to SAP thisone 37 pages long and bristling with accusations o breach o contract.

Some are speci c: SAP also ailed to noti y SCO that it was a bestpractice to enter employee time on a regular basis rather than once amonth.

Some are broad: SAPs breaches span the entire breadth o the Contractrom the project planning and project implementation, to the execution

o the plans, and nally to the result or output o the execution.


39/55


3

One might ask what happened to that positive, results-oriented teamculture.

The October 2012 quarterly report to the Legislature mentioned the curenotice. It did not include a copy o it, however. And now all the risks arelisted as high and the Vital Signs scorecard is dotted with red boxes. ButTeam E ectiveness still gets a green: highly e ective.

The quarterly reports cover letter, signed by Jim Lombard, SCOschie administrative o cer, states that SAP has 30 days to correct thede ciencies. Lombard manages to nd a positive note: Notwithstandingthe signi cant design and operational issues cited in the [cure] letter, theSCO still believes that the SAP so tware is the plat orm or payroll andpersonnel administration or the uture. He concludes: SCO is workingwith SAP to resolve the issues and will keep the [Joint Legislative BudgetCommittee] in ormed o the outcome.

On Feb. 8, 2013, the Controllers O ce announced that it wasterminating the contract with SAP a project that cost taxpayers morethan a quarter o a billion dollars. On Feb. 12, Carlos Ramos, the head o the Cali ornia Technology Agency, noti ed the Legislature that he wassuspending the 21st Century Project.

A ter the contract was terminated, the tone o the SCOs quarterlyreports to the Legislature struck a di erent note. Instead o scorecards o Vital Signs, the May 2013 legislative report o ers a post mortem titledContract Termination Problems.

Lombard also wrote the cover letter or the May report. The SCOworked with SAP in good aith to go through the errors and ensure thatSAP was addressing them, Lombard said. However, eight months o payroll runs have yet to produce any pay cycle without material errors andhave instead exposed a system riddled with grave weaknesses.

* * *

In its report on IT governance, the Little Hoover Commission made onecomment that seems particularly instructive or the Legislature. Thecommission counseled the Cali ornia Technology Agency to be wary o sel -assessment by departments. The reason was simple: Such sel -ratingis vulnerable to bias toward making projects appear more success ul thanthey might be.


40/55


34

(From the Controllers April 30, 2012 Quarterly Report to the Legislature. ProjectScorecard o Vital Signs showed 14 o 15 categories rated green, the highest possible.On June 11, 2012, the project would go live with its rst payroll).


41/55


3


42/55


43/55


3

Project Timeline

July 2003 State chie in ormation o cer approves easibility study ornations largest payroll/human resources modernization.

May 2004 Department o Finance approves easibility study. Projectbegins with initial cost estimate o $132 millionand mid-2009 completion date.

April 26, 2005 Project contracts with SAP Public Services, Inc. orso tware, begins search or integration vendor to design, develop anddeploy the so tware.

July 2005 Los Angeles Community College District starts using newSAP payroll system. District o cials describe transition as horri c.

June 2006 Controller contracts with BearingPoint to serve as systemintegrator.

January 2007 John Chiang sworn in as new state controller. ReplacesSteve Westly.

January 2007 Los Angeles Uni ed School District launches SAPpayroll system. Thousands o paychecks are incorrect. Mistakes continue

or months.

February 2007 Amid restorm o criticism over LA Uni eds SAProllout, Controllers O ce vows state will not make similar mistakes.

Oct. 19, 2007 State issues cure notice to BearingPoint, declaring thesystem integrator in breach o contract and threatening termination.

Oct. 26, 2007 BearingPoint responds with letter blaming theControllers O ce or the projects troubles.


44/55


38

May 2008 Controller, Department o General Services andBearingPoint agree to rebaseline the project, which is months behindschedule. Cost increases to$179 million, completion date pushed back ayear, toJune 2010.

January 2009 State terminates BearingPoint contract.

February 2010 Project restarts with SAP as system integrator. Cali orniaState Universitys 54,000 employees removed rom project. Cost increasesto $305 million,completion date pushed back two more years, toSeptember 2012.

Spring 2011 Data conversion di culties and other seriousimplementation issues emerge.

May 18, 2011 Controller initiates problem escalation process orSAPs alleged ailure to resolve contractually material issues.

Aug. 19, 2011 Controller sends cure notice to SAP. Primary problemis stalled data conversion.

Aug. 21, 2011 SAP cure notice response notes disagreementbetween SAP and the SCO as to the contractual responsibility or dataconversion.

Aug. 25, 2011 Little Hoover Commission oversight hearing includesprogress report on 21st Century Project. No mention o Aug. 19 curenotice.

Sept. 1, 2011 Go live date cant be met and is pushed back to March2012.

Sept. 7, 2011 SCO and SAP enter into con dential negotiations toresolve di erences outlined in the August cure notice and SAPs response.

Nov. 17, 2011 Con dential negotiations between SCO and SAP yielda 39-page settlement agreement to resolve most di erences and continueproject with signi cantly higher state costs.

March 2012 SCO requests an additional $179 million, raising theprojects cost to$373 million.Completion date pushed back anotheryear, toSeptember 2013.

June 11, 2012 First pilot goes live with Controllers 1,300 employees.Payroll checks contain numerous errors.


45/55


3

Aug. 9, 2012 Pilot 2, to incorporate another 15,000 state employeesstarting Sept. 1, is postponed inde nitely because o recurring errors inPilot 1.

Oct. 25, 2012 Controller sends cure notice to SAP, initiating contracttermination. Notice states: SAPs breaches span the entire breadth o thecontract rom the project planning and project implementation, to theexecution o the plans, and nally to the results.

Nov. 30, 2012 SAP response to cure notice states it has not breachedthe contract in any material respect and cannot be terminated orde ault.

Feb. 5, 2013 SAP issues demand or mediation to resolve projectdisputes.

Feb. 8, 2013 Controller terminates the contract with SAP. At the sametime, CTA suspends the project.


46/55


47/55


4

References

Abdollah, Tami. State to adopt pay system despite errors; Thousandso L.A. Uni ed and community college district workers did not receivechecks or were paid the wrong amount, o cials say. Los Angeles Times.Feb. 19, 2007.

Alamo, Chas, scal and policy analyst, Legislative Analysts O ce.Interviews. February 2013.

BearingPoint, Inc.:

Letter notifying State Controllers Of ce it is in material breachthe contract or the 21st Century Project. Oct. 26, 2007.

Response to states termination of contract for 21st Century ProjeJan. 6, 2009.

Boghossian, Naush. L.A. schools may sue so tware rm. Contra CostaTimes. Feb. 16, 2007.

Brandt, Steve. District payroll glitch is teacher nightmare; Its anotherblack eye or Minneapolis schools.Star Tribune. July 23, 2006.

Brown, Vincent, chie executive o cer, Alameda County EmployeesRetirement Association. Interview. March 2013.

Cali ornia State Auditors Updated Assessment o High-Risk Issues the

State and Select State Agencies Face. Chapter 6, Providing E ectiveOversight o the States In ormation Technology Projects. August 2011.

Cali ornia Technology Agency:

Letter from CTA Secretary Carlos Ramos to state Controller JohnChiang suspending the 21st Century Project. Feb. 8, 2013.

Letter from Secretary Ramos to Joint Legislative BudgetCommittee disclosing order to suspend 21st Century Project.Feb. 12, 2013.


48/55


42

Chaudhry, Shanda, assistant deputy director or legislation, Cali orniaTechnology Agency. Interview. February 2013.

Crowe, Lisa, division chie , Personnel/Payroll Services Division, StateControllers O ce. Interviews. March, April, July, August 2013.

Cruz, Marvin, deputy chie in ormation o cer, Los Angeles Uni edSchool District. Interview. March 2013.

Davidson, Tony, project director, 21st Century Project, State ControllersO ce. Interviews. March, April, July, August 2013.

Davidson, Scott, SOMS project manager, Cali ornia Department o Corrections and Rehabilitation, ormer independent oversight consultanton the 21st Century Project. Interviews. May, June 2013.

Department o General Services:

Letter on behalf of the State Controllers Of ce notifyingBearingPoint, Inc. that it is in breach o contract or the21st Century Project. Oct. 19, 2007.

Letter on behalf of the State Controllers Of ce initiatingtermination o BearingPoint contract. Dec. 3, 2008.

Johnson, Nels. Fixing a scal asco: County so tware system still notworking right.Marin Independent Journal. Feb. 22, 2009.

Kunkle, Fredrick. Fair ax suspends work on computer system overhaul.Washington Post. April 20, 2012.

Legislative Analysts O ce:

2009-10 Budget Analysis Series: General Government, StateControllers 21st Century Project.

Summary of LAO Findings and Recommendations on the 2013-14Budget, State Controllers O ce. Recommend pause on troubledpayroll IT project and assessment o state IT oversight.March 6, 2013.

Summary of LAO Findings and Recommendations on the 2013-14Budget, State Controllers O ce. Recommend assessment o ITproject in addition to reconciliation and legal e orts.May 20, 2013.


49/55


4

Lewis, Anthony, general counsel, Cali ornia Technology Agency.Interview. February 2013.

Li, Erika, assistant principal budget manager, in ormation technologyoperations and consulting, Cali ornia Department o Finance, ormeranalyst or in ormation technology, Legislative Analysts O ce. Interview.March 2013.

Little Hoover Commission:

Letter to the governor and Legislature. Sept. 27, 2011.

Hearing on Information Technology Governance. Aug. 25, 2011.

Lombard, Jim, chie administrative o cer, State Controllers O ce.Interviews. March, April, July 2013.

Los Angeles County Civil Grand Jury Report 2009-10.Los AngelesUnifed School District Payroll System Implementation: Where Are theControls?

Los Angeles Times. Garbage in, garbage out; A so tware asco causedL.A. Uni eds payroll debacle, but the district was programmed to ail.Unsigned editorial. Feb. 2, 2008.

Los Angeles Uni ed School District, O ce o the Inspector General:

Attestation Report.Review o Internal Control Design: EnterpriseResource Planning Implementation. Release II Human Resources/ Payroll Regular Payroll Processing. July 18, 2006.

Audit Report.SAP Segregation o Duties. Sept. 2, 2008.

Audit Report.O -Cycle Payroll Processes. Sept. 8, 2008.

Audit Report.Final Pay Calculation within the Payroll ServicesBranch. Sept. 9, 2008.

Audit Report.SAP Application Security Review. Dec. 10, 2009.

Audit Report. Focused SAP Confgurable Controls Review Phase 1 Blueprinting.Dec. 3, 2012.

Martinez, Julio, director o legislative a airs, State Controllers O ce.Interviews. March, April, May, June 2013.


50/55


44

Megerian, Chris. State payroll system near collapse; Costs have tripledor high-tech system, which is years late and unable to pass tests.Los

Angeles Times. Dec. 22, 2012.

Morales, Lourdes, scal and policy analyst or in ormation technology,Legislative Analysts O ce. Interview. February 2013.

Morgan, Jenni er, president, SAP Public Services. Testimony be orethe Subcommittee on Federal Financial Management, GovernmentIn ormation, Federal Services, and International Security, U.S. SenateCommittee on Homeland Security and Government A airs.May 24, 2012.

Ortiz, Jon. Chiang cancels big so tware contract.Sacramento Bee. Feb. 8, 2013.

Press-Telegram(Long Beach), System Failure. Unsigned editorial.June 22, 2008.

SAP Public Services, Inc.:

Response to request for proposal for 21st Century Project. Aug. 28, 2009.

Best and Final Offer in response to request for proposal for the21st Century Project. Oct. 14, 2009.

Twenty First Century SAP HCM Project Escalation Management Summary. Aug. 17, 2012.

SCO Pilot 1 Post Go-Live Project Review, Aug. 24, 2012.

Response to Oct. 25, 2012 cure notice. Nov. 30, 2012.

Demand for mediation of disputes involving the 21st CenturyProject. Feb. 5, 2013.

Responses to questions from the Senate Of ce of Oversight andOutcomes. April 16, May 22 and Aug. 5, 2013.

Songini, Marc L. Irish agency halts work on two SAP applicationprojects.Computerworld. Oct. 17, 2005.


51/55


4

State Controllers O ce:

21st Century Project Feasibility Study Report. RevisedApril 27, 2004.

Special Project Reports 1 through 5 for 21st Century Project. Ma2004, March 2008, May 2009, December 2009 and February 2012.(SPRs must be submitted to the Cali ornia Technology Agencywhen a project deviates by 10 percent or more rom the mostrecently approved project budget or schedule.)

Contract with SAP, plus amendments, to serve as system integratoon the 21st Century Project. Feb. 2, 2010.

Formal cure notice initiating termination of contract with SAPPublic Services. Aug. 19, 2011.

Agreement and Settlement with SAP to modify 21st Century Projectcontract, bring in a data conversion specialist, extend the scheduleand restart the project. Nov. 17, 2011.

Formal cure notice initiating termination of contract with SAPPublic Services. Oct. 25, 2012.

Independent Project Oversight Reports, monthly from July 2004through January 2013.

Independent Veri cation and Validation Reports, monthly fromJanuary 2010 through August 2012.

Quarterly Reports to the Legislature on the 21st Century Project. April 28, 2010 through May 9, 2013.

21st Century Project Steering Committee Status Reports,April 2009 through September 2012.

Draft 21st Century Project Eight Point Recovery and Go ForwardPlan. January 2013.

Letter to SAP, terminating the contract for the 21st Century Project.Feb. 8, 2013.

Takai, Teresa, chie in ormation o cer, U.S. Department o De ense,ormer chie in ormation o cer, state o Cali ornia. Interview. April 2013


52/55


46

Wilmoth, Ken, director, ERP Business Focus, Eclipse Solutions.Independent veri cation and validation (IV&V) consultant on the21st Century Project. Interview. March 2013.

Winkley, Mary, assistant secretary, program & port olio management,Cali ornia Technology Agency. Interviews. February, July, August 2013.

Wong-Martinusen, Collin, chie o sta , State Controllers O ce.Interviews. March, April, July 2013.


53/55


54/55

Senate Ofce o Oversight and Outcomes

1020 N Street, Suite 560

Sacramento, CA 95814

Telephone: (916) 651-1518

Facsimile: (916) 324-5927

http://www.sen.ca.gov/oversight

Special Counsel John Adkisson [email protected]

Ofce Manager Monique Graham [email protected]

Principal Consultant John Hill [email protected]

Principal Consultant Saskia Kim [email protected]

Principal Consultant Dorothy Korber [email protected]

Principal Consultant Jim Sweeney [email protected]


55/55

1549-SAdditional copies of this publication may be purchased for $4.50 per copy

(includes shipping and handling), plus current California sales tax .

Senate Publications & Flags1020 N Street, B-53

Sacramento, CA 95814

Documents

SCO Payroll Project Report, Senate Office of Oversight and Outcomes