Advancing Return on Investment Analysis for Government IT · 2018-06-11 · CENTER FOR TECHNOLOGY IN GOVERNMENT —ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT : A PUBLIC

Advancing Return on Investment

Analysis for Government ITA Public Value Framework

Anthony M. Cresswell

G. Brian Burke

Theresa A. Pardo

Center for Technology in Government

University at Albany, SUNY

187 Wolf Road, Suite 301

Albany, NY 12205

Phone: (518) 442-3892

Fax: (518) 442-3886

E-mail: [email protected]

www.ctg.albany.edu

September 2006

This material is based upon work supported by SAP. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of SAP.

©2006 Center for Technology in GovernmentThe Center grants permission to reprint this document provided this cover page is included.

CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK

THIS PAGE INTENTIONALLY LEFT BLANK


Acknowledgments

This paper and the project it is based on—”Public ROI: Advancing Return on InvestmentAnalysis for Government IT”—was made possible with funding and guidance from SAP, a

provider of enterprise software for the public sector. In particular, we would like to thank RussLeFevre and Peg Kates, who have been partners from the very beginning of this work, andspecial thanks to Tom Shirk, Ingo Hoffman, Rod Massey, Ian Swann, Mor Sagmon, CarstenFriedland, Alex Bratzler, and Bonnie Rothenstein.

Several members of the Center for Technology in Government staff made many importantcontributions to this white paper and the overall project: Sharon Dawes, Jane Krumm-Schwan, Alison Heaphy, Paula Hauser, and Lucy Dadayan.

Finally, this white paper and the case study reports depended on the enthusiasticparticipation of many government officials from the five case study sites. In particular, wewould like to thank Christian Ihle from the Ministry of Finance in Austria, Adam Jansen fromthe Washington State Digital Archives, Ronny Jacobowitz and Yitshak Cohen from theMerkava Project in Israel, Andy McIntyre from the Integrated Enterprise System in theCommonwealth of Pennsylvania, and Darrell Fowler from Service New Brunswick in Canada.We thank them for their generous hospitality and commitment in helping us conduct our sitevisits and creating this resource for improving government IT investments.




Executive SummaryExecutive SummaryExecutive SummaryExecutive SummaryExecutive Summary 11111

IntroductionIntroductionIntroductionIntroductionIntroduction 44444

Section ISection ISection ISection ISection I 55555 Why Assess Public ROI for Government ITWhy Assess Public ROI for Government ITWhy Assess Public ROI for Government ITWhy Assess Public ROI for Government ITWhy Assess Public ROI for Government IT

Section IISection IISection IISection IISection II 66666 Research Summary of Public Value Case StudiesResearch Summary of Public Value Case StudiesResearch Summary of Public Value Case StudiesResearch Summary of Public Value Case StudiesResearch Summary of Public Value Case Studies

Section IIISection IIISection IIISection IIISection III 88888 A Public Value Framework for Government IT AssessmentA Public Value Framework for Government IT AssessmentA Public Value Framework for Government IT AssessmentA Public Value Framework for Government IT AssessmentA Public Value Framework for Government IT Assessment

10 A.The Framework Strategy.

11 B. How Does Government IT Investment Link to the Public?

13 C. What Kinds of Impacts Matter for Public Value.

14 D. What is the Investment? Linking IT to Goals and Business Processes.

16 E. What Kinds of Public Value are Produced?

19 F. Who Receives Value? The Stakeholder Analysis.

20 G. How to Demonstrate the Value? Identifying Variables and Methods.

26 H. What Can Interfere With or Prevent Public Returns? The Risk Analysis.

29 I. Overview of Using the Framework.

32 J. How to Summarize and Present Results?

Section IVSection IVSection IVSection IVSection IV 3737373737 Reflections on the Framework and the Value of Public ValueReflections on the Framework and the Value of Public ValueReflections on the Framework and the Value of Public ValueReflections on the Framework and the Value of Public ValueReflections on the Framework and the Value of Public Value

AppendixAppendixAppendixAppendixAppendix 3838383838 The Research Basis for the New FrameworkThe Research Basis for the New FrameworkThe Research Basis for the New FrameworkThe Research Basis for the New FrameworkThe Research Basis for the New Framework

38 A. Consultative Workshop

40 B. Case Study Summaries

EEEEEndnotesndnotesndnotesndnotesndnotes 4343434343

Table of Contents



CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 1

Executive Summary

The range and complexity of governmentinformation technology (IT) investments

makes assessing their returns a dauntingprospect. Projects can range fromsystemwide transformations, to improvingfinancial transparency, to more efficient doglicensing. The returns may be large or small,obvious or obscure, and can run from a fewminutes saved in a routine transaction toimproving the trust and legitimacy of anentire government. In spite of the difficulty inassessment, however, knowledge aboutpublic returns can be vital to fully informedand justified IT investment decisions.

Assessing these returns remains a coreproblem in IT planning and decision making.That problem results from shortcomings inthe available methods and models forassessing public returns, what we callpublic return on investment (ROI). In lookingover these existing methods and models wesaw three significant shortcomings:

n Incomplete analysis of public value,

resulting in too narrow a scope of whatcan be considered returns to the public.

n Lack of systematic attention to how

government IT investments generateresults of value from the point of view ofthe public.

n Weak or absent methods for tailoring a

public ROI assessment to the specificcontext and goals of a government ITinvestment.

Existing methods also deal with risks togovernment IT investment, but primarily fromthe point of view of technology developmentand implementation. The risks that involvethe public beneficiaries of the investmentsmerit more attention.

This white paper presents a public ROIassessment framework that addresses theseshortcomings. We call it a public value

framework to emphasize the point of view ofthe public, not the government, as the basisfor the assessment. The frameworkdescribes how to identify and assess publicvalue through the kinds of activities shownbelow in Figure 1.

The framework’s strategy is simple inconcept, but complex in application: connectwhat happens in the government (on theleft) with the impacts on stakeholders in thepublic domain (to the right), then report andapply the results. The activities on the leftidentify the potential value mechanisms andoutcome goals. Those are linked tostakeholder interests, impacts, and risks tothe right. The curved arrows indicate that in

Figure 1. Public Value Framework

CENTER FOR TECHNOLOGY IN GOVERNMENT—ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK2

practice the process would seldom be linear,requiring reflection and backtracking toadjust for learning and new information. Thefull paper presents a detailed version of thisschematic, showing the links among theseactivities.

The public value proposition takes centerstage. This value proposition must bebroadly conceived to do justice to the scopeof government and how it affectsindividuals, groups, and both public andprivate organizations. This frameworkpresents a new and more comprehensiveway of describing public value, based on sixkinds of impacts government IT can have onthe interests of public stakeholders:

nnnnn Financial – impacts on current or

anticipated income, asset values,liabilities, entitlements, and otheraspects of wealth or risks to any of theabove.

nnnnn Political – impacts on personal or

corporate influence on governmentactions or policy, role in political affairs,or influence in political parties orprospects for current of future publicoffice.

nnnnn Social – impacts on family or community

relationships, social mobility, status, andidentity.

nnnnn Strategic – impacts on economic or

political advantage or opportunities,goals, resources for innovation orplanning.

nnnnn Ideological – impacts on beliefs, moral

or ethical commitments, alignment ofgovernment actions or policies or socialoutcomes with beliefs, or moral orethical positions.

nnnnn Stewardship – impacts on the public’s

view of government officials as faithfulstewards or guardians of the value ofthe government itself in terms of publictrust, integrity, and legitimacy.

Expanding the view of stakeholder interestsin this way brings into focus two distinct butequally important types of public value: thedelivery of benefits directly to citizens andenhancing the value of government itself as

a public asset. An IT investment that makesgovernment more transparent, more just,and a better steward has added publicvalue, a non-financial but nonethelessimportant return. This framework describeshow to include both in public valueassessments.

The framework also identifies the basicways government IT investments link topublic value. The simplest link results froman IT investment embedded directly in aservice delivery process (Figure 2:A) in away that enhances its value to citizens orofficials. The total value may be a compositeof several separate measures: cost savings,quality, enhanced trust. An indirect link(Figure 2:B) can result from infrastructureimprovements to business processes.

This link may be more difficult to trace andcan involve more risk. Enterprise Resource

Citizen/Official

government with old IT

government with new IT

V1

V2

Return = V2 – V1

Citizen/Official

old government business process

new government business process

V1

V2

old IT infrastructure

new IT infrastructure

Return = V2 – V1

Figure 2. Direct and Indirect Links to Public Value

A - Direct B - Indirect


Planning (ERP) software implementations inour case studies are excellent examples ofthis kind of investment. Value can also resultfrom a mix of direct and indirect links (Figure3) when new IT changes the environment.

Identifying these links is just part of the story.Each link can involve several value-generating mechanisms. The frameworkdescribes how to trace these mechanismsworking through specific businessprocesses to produce different kinds ofvalue. The value-generating mechanismsare:

nnnnn Increases in efficiency – obtaining

increased outputs or goal attainmentwith the same resources, or obtainingthe same outputs or goals with lowerresource consumption. In our Austrianand Pennsylvania case studies, forexample, new ERP systems helpedachieve substantial efficiencies infinancial management.

n Increases in effectiveness –

increasing the quality and/or quantity ofthe desirable thing. Our case study ofService New Brunswick, for example,repor ts how an online registry for landdata can contribute to improvements inproperty tax administration.

n Enablement – providing means or

allowing otherwise infeasible orprohibited desirable activity, orpreventing or reducing undesirableevents or outcomes. In our Washington

State Digital Archives case study, forexample, putting birth and marriagerecords online enabled research bylocal historians and genealogists.

n Intrinsic enhancements – changing the

environment or circumstances of astakeholder in ways that are valued fortheir own sake. For example, our Israelcase showed how enhanced financialaccounting and reporting in theMerkava ERP system openedgovernment financial decision makingto greater transparency.

An IT investment project can deliver publicvalue through any or all of thesemechanisms.

The framework is deliberately presented at amoderate level of generality to make it mostwidely useful. Every government IT projectwill have its own unique goals, valuepropositions, and stakeholders. So thisframework can be used to plan and guide apublic value assessment, in combinationwith measurements, analysis tools, andreporting techniques chosen for the specificsituation. To aid in this regard, the paperpresents an overview of more detailed andhighly specified assessment methods thatcan be used in conjunction with thisframework. This includes a more detailedflow chart for the assessment process, asummary of several ROI models andmethods, and suggestions for analysis andreporting of results.

Figure 3. Mix of Direct and Indirect Links to Public Value

Citizen/Official

governmentwith new IT

Return = Vd + Vi

Citizen/OfficialV i

Vd

V i = indirect value Vd = direct value

environment


Introduction

Every investment decision requires aleap of faith—sometimes a large one—

into an uncertain future. When much of thatuncertainty involves the value of theexpected returns, risk increases and theinvestment decision is all the more difficult.This is an even bigger problem forgovernment IT projects, which arenotoriously risky and aimed at public valuereturns that are often very difficult to defineand even harder to measure. This whitepaper takes on one important part of thatproblem: the question of describing andmeasuring public value.

We chose to focus on public value becauseit is both the most poorly understood and themost complex par t of the overall problem.We can divide the overall analysis of returnon investment (ROI) in government ITexpenditure into four parts: costs, internalreturns (agency efficiencies, cost avoidance,etc.), public returns, and risks. Of these, thepublic returns element receives the leastamount of attention, though interest in thistopic is growing. The key question aboutpublic returns, of course, is what do wemean by public value and how can weobserve, measure, and document itscreation. The goal of this white paper is tohelp answer that question.

The approach we take to answering thatquestion takes a point of view based on ourexperience, analysis, and the backgroundresearch for this paper. From this point ofview we see two sources of public returns:(1) value to the public that results fromimproving the government itself, and (2)value that results from delivering specificbenefits directly to persons or groups. Wealso see potential value creation that goesfar beyond the traditional financial andservice evaluation data. Value creation cancome as much from increasing the integrityand transparency of government as fromreducing costs through online tax payments.This expanded scope of value includes anoften wide range of stakeholders, each withtheir special interests and expectations fromgovernment. This point of view dictates anexpansive way of seeking public value. Wecall this a public value framework, meaningthat it is less a specific measurement tooland more a way of identifying and assessing as wide a range of public value aspossible. This is not a small task, andcannot be completed in one such effor t. Butthis approach can advance the search for away to effectively measure public value.


Section I: Why Assess Public ROI

for Government IT?

The scope of government investment in ITand the associated problems certainly

deserve serious attention. Both the levelsand growth rates in government IT spendingare substantial. The most recent dataavailable shows that the level of thisinvestment in both the industrializedcountries and the developing world hasgrown to a very large scale.

n European government IT spending is

expected to increase from US$110billion in 2005 to US$119 billion by2007, with US$26 billion in the UKalone, which is about 40% aboveGermany or France.

n IT spending by Asia-Pacific region

governments, excluding Japan, isexpected to reach US$31.7 billion by2010, from US$22.7 billion in 2006.

n The Chinese government expenditures

of USD$5 billion in 2004 are expectedto grow 16% in the following five years.

n India spent US$943 million on

e-government in 2002, and this figure ispredicted to increase by 15% a year toUS$3.3 billion by 2009.

n The U.S. federal budget for fiscal year

2007 provides $64 billion in funding forIT investments, approximately a 3%increase from the 2006 enacted level of$62 billion. Total state and localgovernment IT spending was about $55billion in FY2004 and is expected togrow to $62.4 billion by FY2009.

Though what is included in the IT categorymay vary across these examples, the overallscale of expenditure remains substantial.1

This level of expenditure is receivingincreasing scrutiny. The reason for thisscrutiny is clear, according to AndreaDiMaio, a Gartner Vice President focusing

on the public sector: “If governments do notaccurately measure the full value of their ITinvestments, they risk a serious politicalbacklash. They will be accused of wastingbillions of pounds of taxpayers’ money onunnecessary technology.” This sentimenthas reached the US Congress. In July of thisyear the Senate Appropriations Committeerecommended no funding for theadministration’s 2007 e-governmentinitiatives, reporting that “… the committeehas no confidence that the amounts beingassessed have any relationship to thebenefits anticipated to be returned.” And inthe House of Representatives, the currentappropriations bill calls for a cost-benefitanalysis of all e-government initiatives.According to Mike Hettinger, staff director atthe House Government Reform Committee’sGovernment Management, Finance andAccountability Subcommittee, “the languagespeaks for itself and reiterates what thesubcommittee has been saying for the pastyear, that in order for this initiative to besuccessful, we need to have a betterunderstanding of the costs and benefits andclearer guidance for the agencies to follow.”2

Certainly this enhanced attention to ROI forIT must include a comprehensive andeffective way to deal with the public valueside of the problem.


Section II: Research Summary of

Public Value Case Studies

This framework is based in part on theresults of five case studies that

examined how government IT investmentprojects came to deliver value to the public.3

The projects were the Integrated EnterpriseSystem in the Commonwealth ofPennsylvania and the Washington StateDigital Archives in the US, the MerkavaProject in Israel, the Austrian FederalBudgeting and Bookkeeping System inAustria, and Service New Brunswick inCanada. All five case study reports and thewhite paper are available on CTG’s Website.2 In the case studies we examined howthe IT investments were conceived anddeveloped, with particular attention to therole of public value in the process. We tookthe view that government IT investmentgenerates public value in two ways:

n By improving the value of the

government itself from the perspectiveof the citizens, and

n By delivering specific benefits directly to

persons, groups, or the public at large.

The first is based on the idea that, assuminga benign government, the better it functionsoverall, the better off its citizens will be. Thegovernment is an asset to the community ornation that delivers a wide range of values.Internal improvements make it a morevaluable asset to the public. The secondway of generating value has three forms:financial, political, and social. Financialvalue results from lowering the cost orincreasing the efficiency of government ordelivering direct financial benefits to thecitizens. Political value consists ofincreasing participation, fairness,transparency, legitimacy, or conferringpolitical capital to elected officials orcitizens. Social returns include increasedsocial status, stronger relationships, oropportunities; increased safety, trust ingovernment, and economic advantage.

These general understandings of publicvalue guided the data collection andpresentation of the case results.

The case studies examined public returns tothe overall IT investment, including returnsgenerated by a government IT investmentand the possible mechanisms to producethem. We did this by considering the linksbetween goals, implemented systems,government performance, and public returnsin terms of where they represent value inthe chain shown in Figure 4 (next page).

The public returns from the investment canflow from the internal improvements in theagency resulting in returns to individualcitizens and the public at large (the mainflow through the center). Other returns canflow to the political system and the economicenvironment (below the center), or througheffects on other agencies (secondaryperformance gains). This general view ofpublic returns informed the case studies andhelped summarize the results.


PublicCost

PrivateCost (?)

IT Investment

Internal Agency Returns

Direct Citizen Benefit

Public-at-Large Benefit

*

*

�

�

�

* *

*

**

***

**�

*

Secondary Performance

Gain

General Economic

Development

Policy/Electoral Benefit

Vendor/IndustryBenefit

� Development Risk

Benefit Risk*Direct Benefit Flow

Indirect Benefit Flow

Direct Benefit Accumulation

Indirect Benefit Accumulation

Business Goalor Problem

Figure 4. Public ROI Value Propositions


Section III: A Public Value

Framework for Government IT

Assessment

This framework is designed to assistgovernment IT executives and analysts

in understanding and measuring the value tothe public of government IT investments. Themain goal is to produce an assessment ofpublic returns that is credible, persuasive,and highly relevant to the investmentdecisions faced by the government. We usevalue here, rather than return to emphasizethe broad scope of the framework. Mostmethods for assessing return on investmentfocus on financial or economic metrics; thisapproach includes a much broader view ofhow IT investments can produce results ofvalue to citizens or to the society as awhole.4 This concept of value includes morethan the usual financial or economic metricscommon in ROI analysis. It is a new andexpanded way of understanding the resultsof government IT expenditures.

We call this set of ideas a framework toindicate that it is more than a particularmethod for public value assessment. It isbroad in scope so that it can be applied tovirtually any government IT investment, fromsimple Web sites to governmentwideinformation systems and architectures. It isbroad in scope because this range ofinvestments requires a comparable range ofassessment methods. Our framework,therefore is a way of thinking about andorganizing the analysis of a family ofproblems that can encompass manymethods. A spreadsheet, for example, is aframework for working on a broad class ofproblems or analytical tasks. Any par ticularspreadsheet may include specific methods,such as scenario analysis or a net presentvalue calculation. Instead of guiding theassessment process in terms of calculationsin a matrix, however, this framework providesan analysis process that starts with a highlevel view of the IT investment and thendrills down through successive steps toidentify the specific measures and methodsthat will reveal and document public value.

In this way, the framework offers both asystematic way of thinking about publicvalue and a way to apply that thinking topar ticular IT investment decisions. Thedrilling down process is necessary to tailor aspecific public value assessment to thenature of a par ticular investment decision.The framework shows how to take intoaccount how public value can changeacross the many interests of citizens andgroups in interacting with governments. Inthe morning, for example, an executivedoing business with the government maythink about how IT speeds payments on hergovernment contracts, in the evening whilehelping with homework she may observehow computers improve the quality ofschools, and while watching the news on TVat night she might learn how a new crimemapping system makes the neighborhoodsafer. These ways of thinking about publicreturns include both easily measured value,like improved financial flows, as well ashighly subjective ones like public safety,service quality, or government integrity. Aframework for public value assessment mustprovide a way to deal with these manyperspectives and possible measures ofpublic value.

Not every aspect of public value is relevantfor a par ticular IT investment. TheWashington State Digital Archives project,for example, had no particular public safetyrelated goals, but is of considerable value togenealogists and local historians. Ourframework begins the process of narrowingand focusing by starting with the three basicelements of analyzing public value: theinvestment, the government operationsaffected, and the stakeholders. At thebeginning of a public value assessment, ananalyst often knows that there are manyconnections among theses three elements,but those connections are poorly specifiedand understood. The situation might bedescribed by the overlapping parts of


Figure 5 below. The public value is to befound by unpacking, so to speak, the area ofoverlap. The task of the assessment,therefore, is to identify the connections andgather data about how the IT investmentproduces value for the relevantstakeholders.

Close attention to all three elements isessential. The connections among theseelements are the keys to a fully informedpublic value assessment. The frameworkprovides a way of describing theseconnections to show how public value isgenerated and the risks involved therein.Focusing on one or two elements alonecannot reveal the necessary scope ofpublic values involved in an IT investmentor how they can be assessed. The risks ofslighting one or another of these elementsin an assessment can be substantial. Inaddition to missing significant publicreturns, such limited thinking can lead tostakeholder resistance, flawed technologydecisions, or poor integration with ordisruption of business processes.

Figure 5. The Basic Elements in the Public Value Framework

Analyzing those connections can, inprinciple, start in any part of the problem.Ultimately, of course, these three elementsmust be considered together in order toensure that the value assessment for anyparticular investment project or system istailored to the specific value context,investment situation, and the beneficiariesinvolved. In practice, however, theassessment must begin somewhere. Theframework presented below describes thelogic and methods that guide theassessment process.

The sequence of activities shown for anassessment is not designed or intended,however, to suggest how IT investmentdecisions should be made.

The Stakeholders

Government Programs & Operations

The TechnologyInvestment

Public Value


A. The FrameworkStrategy

The framework describes public valueassessment as a series of steps to gatherspecific data and use it to answer questionsthat lead to public ROI results. The resultscan include measures and documentationof public returns, risk analysis of the threatsto achieving those results, and suggestionsfor presenting results to decision makers.Each component of the framework dealswith a different set of questions andpossible data to use in subsequent steps,and so for th. By working with the ideas andmethods presented in the framework, theanalysis can produce a public ROIassessment that is broad in scope, yettailored to the par ticular investment ofinterest.

The steps are not intended to be used in alock-step manner. The questions andinquiry involved in one part of theframework may be obvious and easilyanswered in some cases and quite complexand difficult in others. It may be necessaryto explore later steps before preceding onesare finished, and then cycle back. In

general, however, the sequence follows thepath shown in Figure 6 below.

The steps illustrate how the process ofvalue identification and measurementcarries through from the goals of an ITinvestment, through the value generatingmechanisms of the business processes,connection with stakeholders, to specificdata and reporting. These are the maincomponents of the framework.

By starting with a deliberately broad scope,then narrowing to specific questions foreach project, this approach addresses oneof the shortcomings of previous work onROI for government IT, namely the narrowscope of possible value questionsconsidered. By expanding the initial scopeof analysis to include a broad view of publicreturns, the framework can lead to a morecomprehensive result. The framework thusdoes not replace or supersede the existingmethods developed by others for assessingreturns to government IT investments.Rather the framework shows how thesemethods can be used in assessment, whatadditional value issues should beaddressed, and how additional assessmentmethods may be used.

Figure 6. Steps in Applying the Public Value Framework


B. How Does GovernmentIT Investment Link to thePublic?

The question of how an IT investment canimpact public value can be answeredinitially in terms of three impactmechanisms. These three, alone or incombination, describe the general ways ITinvestments can connect with the public, asrevealed in our research. Identifying theselinking mechanisms is an important step inthe analysis process because it leads to amore detailed examination of impactpathways than might otherwise occur. ITinvestment can enhance public value inmany ways, some of which are onlyindirectly related to the performance of aparticular IT system. Attention to theseindirect and more complex mechanisms canhelp ensure a full picture of public valueresults is developed.

1. Direct Service Impacts

Direct service impacts (Figure 7) occur whenIT is embedded directly in a service deliveryprocess, generating service changes thatenhance value to the citizen or officialinvolved in the service process. The netpublic value is just the difference betweenthe value of the new and the old. Value inthis sense, and in the other mechanisms aswell, is assessed from the point of view ofthe public participant in the transaction, notthe government. The total value may be acomposite of several separate measures,such as lower cost to the citizen, plusincreased satisfaction with quality ofservice, plus enhanced trust in government.The specific kinds of value involved aredescribed in more detail in a later section.

The e-government investments in our casestudies and much of the literature one-government provide many examples ofthis kind of investment. These includesystems for fee and tax payments, licenseapplication and renewals, obtaininginformation, filing forms, etc. The public

value propositions for these investments gobeyond the important but obvious cost andtime savings to include attention to servicequality, access, equity, and the full range ofvalues described below.

2. Indirect Service Impacts

Indirect service impacts (Figure 8) occurwhen back office or infrastructureinvestments produces changes in agovernment business process. As in Type 1mechanisms, the value to the public isreflected in the changed interaction ortransaction with a government businessprocess. However, the process changeresults from an IT investment at least onestep removed from interaction with thepublic. Because of this indirect route, it maybe more difficult in Type 2 cases to trace thelinks from the IT investment clearly andunambiguously to the public. Risks to theeffectiveness of the investment are higheras well, due to dependence on businessprocess changes that are independent ofthe IT itself. The IT investment may havepotential results spread over many businessprocess and may interact with othertechnologies, further obscuring the impactlinkages.

The ERP system implementations in Austria,Israel, and Pennsylvania that wereexamined in our case studies are excellentexamples of this kind of investment. Theprimary impact of these systems is on theinternal, back office operations of thegovernment. How to identify and describe

Figure 7. Type 1: Direct Service

Impact

Citizen/Official

government with old IT

government with new IT

V1

V2

Return = V2 – V1


Figure 8. Type 2: Indirect Service or Impact

Citizen/Official

old government business process

new government business process

V1

V2

old IT infrastructure

new IT infrastructure

Return = V2 – V1

these linked back office operations isdiscussed in a later section of theframework in terms of business referencemodels (see Figure 10 on page 15). Theimpacts on these internal operations may bevery positive and extensive, and still remainhard to trace to many kinds of value gainsfor individual citizens or organizations. Thisis an important part of the public valueproblem dealt with in detail in the valuetypes discussion to follow. ERP systems alsocontribute to public value generationthrough the Type 3 mechanism discussedbelow.

3. Mixed Service & EnvironmentalValue Impact

In this third type of impact mechanism(Figure 9 below), the links to public value

are more complex. The direct valuemechanism shown here as V

d, is the same

as a Type 1 or Type 2 mechanism,producing a value increment for a citizen,official, or group (shown in a simplified way).In this type of system, the new IT is alsolinked to changes in the environment andrelationships between the direct beneficiaryand other entities (persons, groups,organizations) in the public arena. Thisindirect mechanism can result in additionalpublic value flowing from interactionsoutside the government, between otherpersons or organizations. The changes inthe environment may also produce directvalue beyond the direct transactions.

There are many examples of this type ofvalue mechanism in our cases and in othersettings. In the Washington State DigitalArchives case at least two such impacts

Citizen/Official

governmentwith new IT

Return = Vd + Vi

Citizen/OfficialV i

Vd

V i = indirect value Vd = direct value

environment

Figure 9. Type 3: Mixed Direct, Indirect Service & Environmental Impact


occurred. The ability of the county auditorsto shift responsibility for preserving digitalrecords to the state archive producedsavings in cost and workload at the countylevel (V

d). This allowed county auditors to

improve services to their citizens in otherareas (V

i). In addition, the accessibility of

the government records online increasesthe overall environment of transparency forthe government, benefiting all citizens. Inthe Service New Brunswick case, Webaccess to company registrations provideddirect value to citizens such as accountants,lawyers, financial institutions, and thegeneral public searching for such businessinformation (V

d). In addition, the decision by

CGI to locate its Global E-GovernmentHeadquarters in Fredericton, NewBrunswick, due in large part to its strongpartnership with Service New Brunswick,benefits the economic development of thecity of Fredericton and surrounding areas inthe province (V

i). In a different way, putting

some forms of information online, such asthe Toxic Release Inventory in the US,enable citizens to obtain benefits from othertransactions, such as lawsuits againstpolluters (V

i).5

C. What Kinds of Imp actsMatter for Public V alue?

Just identifying these operationalmechanisms in general terms, however,does not tell the full value story. Eachmechanism can involve more than one kindof value generator. These mechanisms showhow IT investments can link to increasedpublic value, but not how that increase isproduced. Each mechanism has thepotential to generate more than one kind ofpublic value increase, depending on thedetails of the situation. The frameworkrecognizes four basic kinds of public valuegenerators, listed below, each with adifferent range of measurements andimplications for assessment:

nnnnn Increases in efficiency – obtaining

increased outputs or goal attainmentwith the same resources, or obtaining

the same outputs or goals with lowerresource consumption. In our Austriaand Pennsylvania case studies, forexample, new ERP systems helpedachieve substantial efficiencies infinancial management and other coreadministrative functions of government.

nnnnn Increases in effectiveness –

increasing the quality and/or quantity ofthe desirable thing. Our case study ofService New Brunswick, for example,repor ts how an online registry for landdata can contribute to improvements inproperty tax administration.

nnnnn Enablement – providing means or

allowing otherwise infeasible orprohibited desirable activity, orpreventing or reducing undesirableevents or outcomes. In our WashingtonState Digital Archives case study, forexample, putting birth and marriagerecords online enabled research bylocal historians and genealogists.

nnnnn Intrinsic enhancements – changing the

environment or circumstances of astakeholder in ways that are valued fortheir own sake. For example, our Israelcase showed how enhanced financialaccounting and reporting in theMerkava ERP opened governmentfinancial decision making to greatertransparency.

The examples above illustrate public returnsthat accrue in addition to, and largelyindependently of, internal efficiency gainsthat may accompany the IT investment.These value generators can also operatetogether, increasing the overall return.Research by the local historians mentionedabove was enabled by access to previouslyinaccessible records; online access to theserecords also made research much moreefficient compared to paper records. Accessto crime mapping information by citizenscould also improve the efficiency ofchoosing a place to live or locate abusiness.


D. What is the Investment?Linking IT to Goals andBusiness Processes.

1. Links to the Business ofGovernment

This analysis of public value generationrequires identifying how the IT investmentproject links to government goals andactivities. IT investments generate value inrelation to the policy and administrativesetting in which they operate. The goals andbusiness processes provide links betweenhow the technology operates and theinteractions with stakeholders that generatevalue. The analysis, therefore, includeslinking the investment to the relevantgovernment goals, operations, and businessprocesses.

This linking process is more complex than itmay appear, requiring a comprehensive andreasonably detailed picture of governmentgoals and operations. Such a picture willensure that all the relevant links betweenthe technology and business processes willbe identified. Many government ITinvestments have potential links acrossmany agencies and processes. TheWashington State Digital Archives, forexample, collects records from hundreds ofstate and local agencies. The Merkavaproject in Israel will eventually involve allgovernment agencies. The Pennsylvania IESsupports human resources management,budgeting, and other administrativefunctions for all executive agencies in theCommonwealth. The relevant businessprocess setting of an IT investment,therefore, can be quite extensive. To dealwith this contingency, this part of theframework is based on just such acomprehensive picture. This is a usefulanalysis strategy because it helps drawattention to value generating aspects of theinvestment that might be missed becausethey result from indirect effects or complexinteractions across many agencies orprocesses.

The first part of the linking takes advantageof the comprehensive integrateddescriptions of government goals andprocesses that are found in the enterprisearchitecture work currently underway for USand other governments. For our purposeshere, the most useful comprehensivedescription of government processes is inthe US Federal Enterprise Architecture(FEA) Business Reference Model (seen inFigure 10).6 With some minor modifications,the components in this FEA model can beused to identify the business operations andgeneral goals of an IT investment in mostgovernment agencies. Local governments,for example, do not typically have defenseand national security operations. Thesupporting documents that form part of theBusiness Reference Model containdescriptions of each of the components.These descriptions can be used to clarifymeaning and help identify whichcomponents are linked to a particular ITinvestment.

While comprehensive, this FEA model is notthe only enterprise architecture model thatmay be used. There are a number ofcomparable models or business processframeworks that could work as well, such asthose developed for the EU, the UK, HongKong, and Malaysia.7 The NationalAssociation of State Chief InformationOfficers (NASCIO) has an enterprisearchitecture model, adapted for US stategovernments, that may also be used. TheEuropean Commission also supportsadditional framework activity through theAthena Project, which has published similardocuments.8

Using this model, the analysis proceedsbyidentifying where the IT investment links togovernment business. This linking can bebased on the new technology’s statedgoals, organizational location, and intendedoperational profile. The detaileddescriptions of each Business ReferenceModel component are useful in identifyingthese links. If needed, those responsible forthe analysis can enlist a team of IT andoperational experts to help ensure allrelevant links are identified.


Figure 10. Elements in the FEA Business Reference Model

To illustrate how to use the model, wemarked several of the components in Figure10 with asterisks. The asterisks mark anexample of the business links we identifiedfor an IT investment project in the Merkavacase study used for this framework: anonline reverse auction system forprocurement. The agency conducting theonline auction sets the terms of aprocurement, publishes them, and qualifiesvendors as potential bidders. At apredetermined time, a Web site is opened forbidding. All qualified vendors can thensubmit authenticated price bids to a publicspace on the site, visible to all othervendors. The bidding continues until the lowbid remains unchanged for a predeterminedtime (e.g., five minutes), closing the auction.

The low price bidder wins the contract. Theentire auction takes place in public viewand is recorded in detail.

For an illustrative example, consider theprocurement of new police radios for a lawenforcement agency. Based on the nature ofthe e-procurement system and thecomponent descriptions, we see it as linkedto the five marked components in Figure 10:law enforcement, financial management,supply chain management, direct service toclients (vendors), and planning andresource allocation.

Services for CitizensCommunity and social services Correctional activitiesDefense and national security Disaster managementEconomic development EducationEnergy Environmental managementScience and innovation HealthIncome security International affairs & commerceLaw enforcement Judicial activitiesNatural resources TransportationWorkforce management Financial managementInformation& technology management

Management of Government ResourcesSupply chain management Human resource managementAdministrative management Financial managementInformation& technology management

Mode of Delivery Government Service Delivery Financial VehiclesDirect Services to Clients Financial assistanceKnowledge creation & Management Credit & insurancePublic Goods Creation Transfers to local governmentsRegulatory compliance & enforcement

Support Service DeliveryLegislative relations Public affairsRegulatory development Controls & oversightPlanning & resource allocation Revenue collectionInternal risk management & mitigation General government

Purpose of Government

Mechanisms used to achieve

purpose

Government Operations

Support Functions

Resource Management

Functions

**

*

*

** *


2. Links to Business Processes

Making further links to business processesmeans connecting the larger goal orfunction from the Business ReferenceModel to specific activities identifiable asbusiness processes. For this step, pickingthe appropriate level of detail is important.Choosing too general or too fine-grained aprocess analysis will tend to obscurestakeholder interactions that are necessaryto identify public value. For this step in theanalysis, we recommend another enterprisearchitecture tool: the Zachman Framework.9

The descriptions in the first two rows of theZachman Framework (Scope and Business

Model) are a good guide for the most usefullevel of detail.

This method identifies several businessprocesses for the e-procurement example.They include:

n determining communication needs for

the law enforcement agencies

n budgeting for the procurement

n recruiting and qualifying vendors

n conducting the auction

n paying the vendors

The list could be extended for fur theranalysis by adding deployment of theradios, training for users (human resourcemanagement), and evaluation of theimpacts of the new radios use in theagencies and business processes whereinstalled. To keep this examplemanageable, however, we are limiting theanalysis to the processes listed below. Thecomponents identified from Figure 10 arelinked to the business processes in the rowsof Table 1 below.

Identifying the business processes leads totwo key questions: (1) How does a businessprocess generate increased public value?and (2) For whom? A way of answering thefirst question is recorded in the third columnof Table 1. Our analysis of public valuegenerating mechanisms is shown in thediscussion of value impact mechanismsbelow.

E. What Kinds of PublicValue are Produced?

Identifying the linkages between businessprocesses and public value generatorsdescribed above carries the analysis acritical step farther: from how the IT changesa business process to the impact of thosechanges for a par ticular public or

msinahceMeulaVdna,ssecorP,tnenopmoCssenisuB.1elbaT

tnenopmoCssenisuB ssecorPssenisuB msinahceMeulaVcilbuP

tnemecrofnewaL noitcuatcudnoc 3epyt-nezitic

tnemeganamlaicnaniF tnemerucorproftegdub 3epyt-nezitic

tnemeganamniahcylppuS noitcuarofsrodnevyfilauq3epyt-nezitic1epyt-rodnev

noitacollaecruoser&gninnalP sdeennoitacinummocenimreted3epyt-nezitic

1epyt-ffatst'vog

stneilcotecivrestceriD noitcuatcudnoc 1epyt-rodnev

tnemeganamlaicnaniF srodnevyap 1epyt-rodnev


stakeholders. That step is accomplished bylooking to the value of the impact in terms ofthe interests of one or another stakeholder.So the framework includes an analysisscheme for taking into account interests thatcan apply to the full range of stakeholders.The framework employs these six basickinds of interests:

nnnnn Financial – impacts on current or

anticipated income, asset values,liabilities, entitlements, and otheraspects of wealth or risks to any of theabove

nnnnn Political – impacts on personal or

corporate influence on governmentactions or policy, role in political affairs,or influence in political parties orprospects for current or future publicoffice

nnnnn Social – impacts on family or community

relationships, social mobility, status, andidentity.

nnnnn Strategic – impacts on economic or

political advantage or opportunities,goals, and resources for innovation orplanning

nnnnn Ideological – impacts on beliefs, moral

or ethical commitments, alignment ofgovernment actions or policies or socialoutcomes with beliefs, or moral orethical positions

nnnnn Stewardship – impacts on the public’s

view of government officials as faithfulstewards or guardians of the value ofthe government itself in terms of publictrust, integrity, and legitimacy

This last interest—stewardship—isdeliberately not included in the ideologycategory, though it could logically fit there.Setting it off separately emphasizes itsimportance in the overall public valueframework. Just as corporate managers anddirectors are responsible for stewardship of

a corporation’s integrity and assets onbehalf of stockholders, public managersand elected officials have a parallelresponsibility for government on behalf ofthe public. Recent financial scandals in theUS have demonstrated how lack ofstewardship in major corporations (e.g.,Enron, WorldCom) can destroy theirfinancial value. The government officials inour cases expressed a similar stewardshipresponsibility in terms of, for example, usingIT to increase transparency andaccountability. Public surveys of trust ingovernment show that citizens have similarstewardship expectations of government.10

Clearly this list goes well beyond the internalefficiency or service quality impacts of an ITinvestment. Of course, attending to thisexpanded concept of value raises manyproblems of definition and measurement.However, our case research and results frommany other kinds of inquiry show theimportance of this more comprehensivetreatment of value and interests. Many of themeasurements and data implied by theseinterest types are found in existing ROImethods as well. However, none of themethods we reviewed for this white papercover the full range represented here.Working with this expanded range of valuetypes represents one important contributionof this framework.

It is also clear from this way of describingpublic value that it supplements rather thanreplaces methods that assess the efficiencygains or savings returns captured bygovernment agencies internally. Forexample, it seems safe to say that aDepartment of Motor Vehicle’s internal costsavings from putting license renewals onlineare invisible and largely irrelevant todrivers. These savings would have nodetectable impact on overall tax burdens orbenefits resulting from a shift of governmentresources to some other service. Theinformation about the savings, however, isanother matter. If government officials makesome political use of the cost savingsinformation, that would represent strategicvalue to those officials or to their political


allies. The public learning about the savingsmight also improve their opinion of thegovernment’s stewardship, fairness, or otherideological value.

F. Who Receives V alue?The Stakeholder Analysis.

Completing the analysis of value impactsrequires identifying those with an interest inthe value generating process: i.e., thestakeholders. The kind of stakeholderanalysis required for this task has threemain parts: (1) identifying the persons orgroups (including organizations) whoseinterests are potentially affected,(2) identifying what their specific interestsmay be, and (3) assessing their role andpotential influence in the delivery of publicvalue. The first two par ts are necessary forthe analysis of value impacts and possibleassessment strategies and so are discussedhere. The third part is more relevant to therisk analysis discussed in more detail insection III.H below.

Identifying stakeholders for the frameworkwill depend on in-depth knowledge of thecontext of the investment and the agenciesinvolved in its development and use.Typically, stakeholder analysis engages agroup of participants with extensiveknowledge of the political andorganizational setting of the investmentand, hopefully, with experience in this kindof analysis. Because the analysis is socontext-dependent, there are no standardprocesses to follow. There areconsistencies, however, among the manyversions of stakeholder analysis methodsfound in the management, planning, andassessment literature. These include:

n Involving multiple participants with wide

knowledge of the stakeholderenvironment

n Looking widely to identify all relevant

stakeholders through brainstorming andrelated methods to stimulate divergent

thinking and include multiple opinionsand information sources

n Identifying multiple stakeholder roles,

internal and external to the organizationsetting (e.g., internal user, customer,vendor, developer, manager, oversight,politician, taxpayer, analyst, advocate,etc.)

n Identifying stakeholder expectations,

influence potential, past and futureparticipation possibilities, and level ofinterest

In this kind of analysis, continuing referenceto the nature of the investment and impactmechanisms will help produce the neededfocus and detail.11

The results of a stakeholder analysis haveimplications for value assessment and forunderstanding their potential to influencethe investment project. These results can bepresented in ways that show the multipledimensions used in the analysis. Anexample of results from a hypotheticalstakeholder analysis is shown in Figure 11.

This figure combines data for 14stakeholders’ roles (S

A = Stakeholder A,

etc.) and times of possible influence in aproject’s lifecycle. This kind of displayillustrates some of the complexities ofstakeholder analysis and the possibilities forinteractions among different stakeholdergroups. A more complete analysis wouldinclude estimates of the stakeholder’sinfluence capabilities and specific interestsin the project development and outcomes.These issues are discussed in more detail inthe section on risk analysis (section III.H).

To focus on stakeholder interests, theillustration in Table 2 below shows how theresults of a simple stakeholder analysis canbe linked to the business process, valuetypes, and mechanisms.

The business processes identified in Table 2on page 20 link to a partial list ofstakeholder types in the columns: citizens at


large, vendors, and elected officials. Anactual assessment would include a moredetailed list of stakeholders. The valuetypes most likely to map onto eachstakeholder type are entered in the cells.The basis for this mapping is linking thenature of the value as understood in thatcontext with detailed knowledge of thestakeholder groups. This part of theassessment can include data collectionabout the interests and expectations ofstakeholders through interviews, surveys, orfocus groups if needed. The entries in thecolumn for the citizens at large, for example,are based on the assumption that theirinterests in government stewardship wouldbe served by the e-procurement system.While a reasonable assumption, this couldbe confirmed by collecting opinion data froma sample of citizens, or relying on previousopinion research. Similarly, the entries in thecolumn for government IT staff are based on

assumptions about what advances theinterests of those staff members. Asuccessful procurement system wouldprovide political value and some strategicadvantage to elected officials, as well asenhance their reputation as good stewardsof public resources.

G. How to Demonstrate theValue? Identifying V ariablesand Methods.

Using the framework up to this point hasidentified many kinds of data relevant toassessing public value. The next step is tochoose the specific variables or points ofobservation for collecting the assessmentdata. This is as much a practical problem asan analytical one. Many variables may berelevant for each combination of

mrofnI tlusnoC pihsrentraP lortnoC

noitaitinI

noitacifitnedI

gninnalP

noitucexE

noitatnemelpmI

gnillortnoC

dnagnirotinoM

noitulovE

gnisolC

Type of Participation

Sta

ge

in

Lif

ec

yc

le

SA

SA

SH

SC

SK

SM

SF

SB

SN

SI

SJ

SE

SD

SL

SG

SO

SD

Figure 11. Stakeholder Analysis Matrix12

Sx = Stakeholder X


stakeholder and value type. A few variablesthat would be relevant to this example areshown in Table 3.

This table illustrates two important aspectsof its role in the framework and assessmentprocess. First, it is highly unlikely thatapplying the framework to any IT investmentwould lead to variables in all, or even mostof the cells of this matrix. This matrix can bethought of as an heuristic device, promptingquestions about what might be useful andavailable variables for each row and columnwithout expecting to fill them. The second isthat identifying a specific variable relies oncombining information about stakeholderinterests, the value type, the impactmechanisms, and the context. This is acomplex and demanding process. Thissection offers additional guidance onchoosing the best public return variables fora given assessment. In addition, the otherROI methods described below include many

variables and additional methods that canbe helpful in that task. However, the morethe choice of variables can be tailored to thespecific public value context, the more validand persuasive the assessment is likely tobe.

An actual public value assessment shouldbe based, of course, on the best availableinformation. But actual assessments takeplace in practical settings of limitedresources and access to data, plus beingpart of the additional work needed forinternal returns and costs. The prioritysetting described in the risk analysis sectioncan narrow the field to only the mostimportant public value outcomes. Thesection below describes additionalstrategies for choosing the appropriatevariables and analysis methods.

xirtaMeulaVredlohekatS.2elbaT

sredlohekatS

ssenisuB

ssecorP

eulaV

rotareneG

tcapmI

epyT

tasnezitiC

egraLsrodneV

detcelE

slaiciffO

roftegduBtnemerucorp

cisnirtni 3 pihsdrawets

enimreteDsdeen

,cisnirtnissenevitceffe

3,ygoloedi

pihsdrawets,lacitilopcigetarts

yfilauQsrodnev

,ycneiciffecisnirtni

3,1,ygoloedi

pihsdrawets,laicnanif

cigetarts,lacitilopcigetarts

tcudnoCnoitcua

,ycneiciffe,cisnirtni

ssenevitceffe3,1

,ygoloedipihsdrawets

,laicnanifcigetarts

,lacitilop,cigetarts

pihsdrawets

srodnevyaP,ycneiciffe

cisnirtni3,1 pihsdrawets

,laicnanifcigetarts

,lacitiloppihsdrawets


1. Variables and AnalysisMethods

The choices of variables and analysismethods for the empirical par ts of a publicvalue assessment should be consideredtogether. In terms of basic measurement andanalytical methodology, what constitutes anappropriate analysis depends, in part, onthe types of data and variables involved.

The scope of this framework, as applied to aparticular IT project, could encompass avery wide range of data types. Many kindsof quantitative data from financial sources,operations research, and surveys areappropriate for statistical analysis,modeling, and simulations. Many of thesocial, political, and intrinsic value returnscan be expressed in normalized scales, ormay best be revealed in qualitative terms orin simple dimensions that are not suitable

epyTybselbairaVredlohekatS.3elbaT

selbairaVtcapmIredlohekatS

epyTeulaV egraLtasnezitiC srodneV slaiciffOdetcelE

laicnaniF cimonocelacoltnempoleved

euneverdesaercni

lacitiloP srodnevmorftroppus

aidemevitisopegarevoc

laicoS ninoitatuperdecnahnegniviecerseicnega

soidar

lacigoloedI tekramfolevelesaercninoititepmoc

noitatuperdecnahnetekramgnitomorprof

noititepmoc

neziticniesaercninoitamrofniotssecca

:tnemerucorptuobaycnerapsnartretaerg

cigetartS erahstekramdesaercni noitatuperdecnahne&nredomgniebrof

evissergorp

egdelwonkdesaercninoititepmoctuoba

pihsdrawetS nitsurtdesaercnitnemnrevog

ytilagurfrofnoitatupertnemeganamdoog&


for much quantitative manipulation. To helpguide the assessment, given the very largenumber of possible public return data typesand variables, the framework provides twokinds of resources. The first is a generalscheme suggesting variable types andsources for different value types (see Table4). The second consists of summaries of theapproaches and variables available in arange of existing ROI methods that can beof value in completing an assessment (seeTable 5).

A wide range of possible public value datacan be identified by the methods presentedhere. The framework approach is based onthe assumption that virtually any kind ofdata can be useful in describing publicvalue creation, from the most precisequantitative figures available from financialor physical measurements, to material asdiverse as the content of blogs orobservation of user or customer behavior. Aconclusion about public value creation

requires an inference, since value does notstalk about wearing a label. Valid inferencesabout value can be formed from qualitativeas well as quantitative data, contentanalysis as well as statistics. Taking thesefour principles into account, choices madeabout how a specific analysis proceedsshould be based on three criteria: 1) Whatconstitutes the best data? 2) What kind ofanalysis is appropriate to that type? 3) Whowill be the audience for the conclusionsreached? The best kind of data availablewill be specific to the operational andstakeholder context. The kinds of analysisappropriate to various data types are shownin summary form in Table 4.13

Beyond these general considerations, thechoice of variables and analyses for anassessment can draw on a volume ofexisting work on ROI methods for guidance.These methods, summarized in Table 5, varywidely in the number and type of variablesused, the scope of public value considered,

.4elbaT ataDeulaVcilbuProfepyTelbairaV

epyTelbairaV selpmaxE sdohteMsisylanA

elacsoitar,ega,emit,atadlaicnanif

noitcasnart,ycneuqerf,ecnatsidemulov

,secnereffid,slatot,segareva&cirtemarap,stnecrep

scitsitatscirtemarapnon,).cte,noisserger,noitalerroc(

,seiresemit,noitalumisnoitazimitpo

elacslavretni,selacsgnitar,selacsnoinipo

selacstrekiL,sedarg

,)snoitpmussahtiw(segareva,sisylanaelacs,seicneuqerf,2x(scitsitatscirtemarapnon

sisylanaretsulc,).cte

selacslanidro selacsgnitar,sgniknargnilacs,seicneuqerf,noitalerroc

sisylana

selbairavlanimon,noitacol,yticinhte,redneg

sroivahebresu,spihsnoitalerkrowten,gniretsulc,seicneuqerf

sisylana

atadesruocsid/txet stroperswen,sgolb,sweivretnidrow,sisylanatnetnoc

sisylanacitnames,ycneuqerf

atadlanoitavresbonoitavresbodleif,seidutsresu

oediv/mlif,sotohp,setonkrowten,spamssecorp

snrettaproivaheb,sisylana


sledoMdnasdohteMIORdetceleSfoyrammuS.5elbaT

sledoM/sdohteM

dohteM

srotpircseD

gnirusaeMeulaV

SU,ygolodohteM

OIClaredeF

licnuoC

nonruteRlaicoS

,tnemtsevnI

)SU(FDER

eulaVtseB

ecnamrofreP

srotacidnI

)KU,MPDO(

rentraG

tnemnrevoG

ecnamrofreP

krowemarF

noitatneirOemiT evitcepsorp evitcepsorter evitcepsorter evitcepsorter

emarFemiT lanoitcesssorc lanidutignol dexim *lanoitcesssorc

lanoitutitsnI

sucoFcilbup etavirp cilbup cilbup

tnemnrevoG

leveLlaredef/lanoitan a/n KUlacol lareneg

tcejorP

noitatneirOTI secivreslaicos

tnemnrevoglacolsmargorp

larenegecnamrofrep

noitatneiro

sepyTelbairaV

,deximyltnanimoderp

evitatitnauq

yltnanimoderpevitatitnauq

yltnanimoderpevitatitnauqevitatilauq+

yltnanimoderpevitatitnauqevitatilauq+

elbairaV

noitacificepS

seirogetaclareneg&selpmaxe+

snoitsegguscificepsylhgih cificepsylhgih

yhgih,evitatitnauq

gniyolpmesexedni

noitatneirOIORP,lanretniyliramirp

cilbupemos

deximlanretxe/lanretni

lanretxehtiwsisahpme

ynam;deximdnalanretni

lanretxeselbairav

&lanretnideximlanretxe

elbairaVIORP

teSdetimil,worran daorb esrevid&egral

detimil

lacitylanA

sdohteM

dohtememostroppus

yllufdohtemdeificeps

otdetimilelbairavnoitinifed

deificepsdohtem

lacitylanA

troppuSssecorPtroppustsedom troppuslaitrap dedulcniton

nwonknu

sisylanAksiR dedulcni dedulcniton dedulcniton noisulcnidetimil

troppuSnoisiceDlootnoisiced

dedulcnitroppusslairetam

enonslairetamtroppus

noitatneserP

slooTsnoitseggusronim

selpmaxededivorp

enonsecirtamstluser

*Includes some longitudinal variables


)deunitnoc(sledoMdnasdohteMIORdetceleSfoyrammuS.5elbaT

sledoM/sdohteM

dohteM

srotpircseD

dnadnameD

tnemssessAeulaV

)ailartsuA(ledoM

ecivreScilbuP

,dohteMeulaV

erutneccA

ecnamrofreP

ledoMecnerefeR

laredeFSU(

esirpretnE

)erutcetihcrA

AVERAM

fotnemnrevoG(

)ecnarF

noitatneirOemiT evitcepsorp denibmoc evitcepsorterevitcepsorter

emarFemiT lanoitcesssorc lanidutignol dexim*lanoitcesssorc

lanoitutitsnI

sucoFcilbup cilbup cilbup

cilbup

tnemnrevoG

leveLlareneg/lanoitan lareneg SU-lanoitan

ecnarF-lanoitan

tcejorP

noitatneirOlareneg lareneg

margorpecnamrofrep

TI

sepyTelbairaVylegral,dexim

evitatitnauqyltnanimoderp

evitatitnauqdexim

ylegral,deximevitatitnauq

elbairaV

noitacificepS

seirogetacdeximemosdna

deificeps

cificepsyltsom&ecnamrofrep

scirtemtsoc

yltnanimoderpelbairav

seirogetac

deificepsylegral

noitatneirOIORP

dnalanretnideximwef;lanretxe

lacitilop

yliramirpemos,lanretni

cilbupdexim

ylegral,deximlanretni

elbairaVIORP

teSesrevid,egral detimil,worran

&egralyrevesrevid

detimil

lacitylanA

sdohteM

&deificepsylhgihdeliated

dohtemsisylanadeificeps

dedulcnitondedulcni

lacitylanA

troppuSssecorP

&deificepsylhgihdeliated

dedulcni dedulcnitondedulcni

sisylanAksiR dedulcni dedulcniton a/ndedulcni

troppuSnoisiceDslairetamtroppus

senilediugdnatroppusslairetam

dedulcnilairetamdedulcnilairetam

noitatneserP

slooTdedulcni

dradnatssyalpsid

laitraplaitrap

*Includes some longitudinal variables


and the level of analytical detail andtechnique included. Some are intended foruse prospectively, in planning for andjustifying an investment. Others are aimedprimarily at showing impacts of investmentsafter the fact. They also vary in terms of thedegree to which they deal with both internaland public value results of the investment,and whether they are designed specificallyfor IT or government investments generally.The summary of these methods or models inTable 5 can help in the selection of variablesand analysis to fit the IT project.

The SROI (Social Return on Investment)model, the only private sector oriented onein the summary, is included for its specialfeatures. Its private orientation refers not tothe commercial sector but to a privatephilanthropic orientation. The method wasdeveloped by a San Francisco foundation toassess its social and economic developmentprograms. It illustrates not so much how toassess an IT investment but rather how todeal with personal and community impactsin a systematic and comprehensive way.


H. What Can Interfere Withor Prevent Public Returns?The Risk Analysis.

Government IT innovation is risky business.Evidence from many surveys and casessuggests that the threats and otherchallenges of IT projects often overwhelmthe capabilities of the developers andimplementers. Data from the private sectortells a similar story. For this paper, we focusour attention not on the full range of risks togovernment IT project success, butconcentrate on the public return aspects ofthat problem.

The distinction between the public returnand more generic aspects of risk analysis isnot, however, a simple one. The approach isthe same: identify and evaluate threats,develop and evaluate response methods,and produce a summary analysis and

mitigation strategy. To identify and evaluatethreats, we return to the overall valueproposition schematic presented earlier(Figure 12).

In this figure, we identified two kinds of risk:1) development risk and 2) benefit risk.Development risk, simply put, is risk that thedevelopment and implementation of the ITwill fail outright or will not perform asdesigned and intended. Benefit risk appliesto whether the IT investment will fail toproduce the envisioned benefits in spite ofbeing successfully developed andimplemented. For the public valueframework we focus on benefit risk. Ofcourse any threat to the development of theIT is an indirect threat to public returns.However, several of the ROI methodsdescribed below include adequate analysismethods for development risk. Thesemethods do not deal adequately, however,with the additional threats and issues inbenefit risk.

PublicCost

PrivateCost (?)

IT Investment




*

*

�

�

�

* *

*

**

***

**�

*


Gain

General Economic

Development









Figure 12. Public Return on Investment Value Chain


The benefit risks associated with thecreation and assessment of public returnscome from threats to the creation of thereturns and to their detection. And thereappear to be two main sources of thosethreats: one is what we will call “theoryfailure” and the other is “exogenous factors.”In theory failure, the underlying assumptionsor theory on which the project is based areflawed or simply wrong. One such theoryfailure caused the US Department ofEducation to abandon a multi-million dollarpilot project for online college studentfinancial aid administration. Developedwithout significant par ticipation from collegefinancial aid officers, the system did notattract supporters and generated muchstakeholder resistance.14 It is important touse both sources and types of benefitsthreats in a full risk analysis, as suggestedby the two-by-two array in Table 6.

The risk analysis process can then use thethreats identified in this way to estimate thepotential loss or cost that each threatrepresents and the likelihood of the threatmaterializing. These estimates should bebased to the extent possible on input fromstakeholders, analysts, users, developers,

managers, and policy makers. This willprovide a basis for accurate estimates andconcentrating attention on the threats withthe highest combined loss-likelihoodestimates. In some cases, the value, cost,and loss-likelihood estimates can bequantified to yield decision tools for movingahead with an investment. The displays inFigure 13 and Figure 14 on the next page,show such a result from the US Federal CIOCouncil’s Value Measurement Method,which employs voting and other methods toprioritize and estimate quantitative valuesfor cost and their related risks. The decisionboundaries shown in the figures come frommanagement decisions or some otherdeliberative process.15

The threats due to difficulty in detectingindirect or second-order effects deservesome added discussion. Public valueoutcomes can extend beyond those involvedin the direct interaction with government. Forexample, drivers are the direct beneficiariesof an online license renewal system.However, the scope of possible indirectvalue outcomes and beneficiaries can bevery broad. Residents in the neighborhoodsof the license renewal agencies will benefit

eulaVcilbuPotstaerhTfoselpmaxE.6elbaT

staerhTfosecruoS

epyTtaerhT eruliaFyroehT srotcaFsuonegoxE

snruterfonoitaercoT

dnamedcilbupfoetamitserevo;ecivresfoesu-nonotsdael

tcapmifognidnatsrednudewalfgnidnatsrednusim;smsinahcem

stseretniredlohekatsfo

gnillecnacrognidnuofnocymonoceninrut-nwod:stceffe;segatnavdalaicnanifslecnac

retnuoctnemnrevognisegnahclacitilop;semoctuopihsdrawets

aidemevitisopslecnacladnacsstroper

snruterfonoitcetedoT

enilesabsuoenorrerofokcalfonoitacificeps-sim;atad

etairporppani;selbairavemoctuo;sledomrosdohtemsisylananodesabsmetsysgnitnuocca

snoitinifedatadetairporppanisledomdna

redro-dnocesdnatceridniatad;elbatcetednustceffe

roycavirpoteudelbisseccaniytivitisnes


Figure 14. Example of a Cost-Risk Boundary Analysis

Figure 13. Example of a Value-Risk Boundary Analysis


to some degree from reduced trafficcongestion and pollution. Shopkeepers inthat neighborhood may lose business for thesame reasons. Second order effects may beeven more diffused and difficult to detect.Learning to trust the online process forlicense renewal, for example, can result inmore use and greater benefits from usingother online services.

Since risks are tied both to stakeholders andpossible value outcome variables, pursuingthis line of reasoning can lead to a verylarge, and likely infeasible, list of riskanalysis factors and tasks. To work withinresource constraints will require limiting therisk analysis to the most important valuevariables and stakeholders. Setting prioritiesfor this kind of analysis will therefore beessential, and must be based on the goals ofthe project and stakeholder value estimates.

I. Overview of Using theFramework.

The framework is a combination of individualanalysis steps combined into an overallprocess or plan for public ROI assessment. Asummary of that overall process is shown inFigure 15 on pages 30-31. As shown there,the process follows the arrows, starting witha clear understanding of the larger contextof public value generation. The investmentgoals that emerge from the investment plansand understanding of the context link to thebusiness processes and public value impactmechanisms. Analysis of the impactmechanisms and public return types leadsto specific impacts listed in the stakeholderinterest analysis. Identifying the interestsand impacts will then guide choosing publicvalue variables, analysis methods, riskanalysis, and ultimately, reporting tools andtechniques.

For simplicity, this process diagram issomewhat idealized. It shows only the basiclinear path through the various steps. Inpractice, the actual path is rather more likelyto have backtracking and loops. Learning at

one step can easily lead to revisions inconclusions or data collected at precedingsteps. This is par ticularly so for the riskanalysis steps in the process. It is notfeasible to complete the risk analysis forachieving any particular public value returnuntil the details about that return and how toassess it are known. Loops can occur in theprocess when a step produces unexpectedor challenging results. For example,identifying all related business processesfor a large, complex project may produce toomany results to carry forward in theanalysis. That would require looping back toreduce the scope. So the potential usershould view this process summary as arough guide rather than a lock-stepprocedure.

The process also represents acomprehensive approach to public valueassessment. We argue that all the steps areneeded to ensure that the full range ofpublic value returns will be included in anyassessment and report. Some small or verynarrow projects, of course, may have limitedpublic value potential and not require suchan extensive review. But the research andanalysis leading to this framework haveclearly demonstrated that the scope ofpublic value possible from government ITinvestments can far exceed what aninternally focused, financial assessment willreveal. Unless assessments use a muchbroader view, as represented in thisframework and other models cited above,the full value of government IT investmentswill not come to light. It also seems likelythat this same kind of analytical frameworkwould be useful for other forms ofgovernment investments. We did not,however, explore that possibility for thisreport.


Figure 15. The Public Value Framework Overview

tnenopmoCssenisuB ssecorPssenisuB msinahceMeulaVcilbuP

tnemecrofnewaL noitcuatcudnoc 3epyt-nezitic

tnemeganamlaicnaniF tnemerucorproftegdub 3epyt-nezitic

tnemeganamniahcylppuS noitcuarofsrodnevyfilauq3epyt-nezitic1epyt-rodnev

ecruoser&gninnalPnoitacolla

noitacinummocenimretedsdeen

3epyt-nezitic1epyt-ffatst'vog

stneilcotecivrestceriD noitcuatcudnoc 1epyt-rodnev

tnemeganamlaicnaniF srodnevyap 1epyt-rodnev

PublicCost

PrivateCost (?)

IT Investment




*

*

�

�

�

* *

*

**

***

**�

*


Gain

General Economic

Development









Public Return Types

F = FinancialP = PoliticalSo = SocialI = IdeologicalSt = StrategicSw = Stewardship

(Continued onnext page)

Business Reference Model Value Impact Types




laicnaniF cimonocelacoltnempoleved

euneverdesaercni

lacitiloP srodnevmorftroppus

aidemevitisopegarevoc

laicoS noitatuperdecnahnegniviecerseicnegani

erutinruf

lacigoloedI folevelesaercninoititepmoctekram

noitatuperdecnahnetekramgnitomorprof

noititepmoc

neziticniesaercninoitamrofniotssecca

:tnemerucorptuobaycnerapsnartretaerg

cigetartS tekramdesaercnierahs

noitatuperdecnahne&nredomgnieb

evissergorp

egdelwonkdesaercninoititepmoctuoba

pihsdrawetS nitsurtdesaercnitnemnrevog

ytilagurfrofnoitatupertnemeganamdoog&

sredlohekatS

ssenisuBssecorP

eulaVrotareneG

tcapmIepyT

tasnezitiCegraL

srodneVdetcelEslaiciffO

roftegduBtnemerucorp

cisnirtni 3 pihsdrawets

enimreteDsdeen

,cisnirtnissenevitceffe

3,ygoloedi

pihsdrawets,lacitilopcigetarts

yfilauQsrodnev

,ycneiciffecisnirtni

3,1,ygoloedi

pihsdrawets,laicnanif

cigetarts,lacitilopcigetarts

tcudnoCnoitcua

,ycneiciffe,cisnirtni

ssenevitceffe3,1

,ygoloedipihsdrawets

,laicnanifcigetarts

,lacitilop,cigetarts

pihsdrawets

srodnevyaP,ycneiciffe

cisnirtni3,1 pihsdrawets

,laicnanifcigetarts

,lacitiloppihsdrawets

Results &

Reporting

Identify Risk

Mitigation Strategies

Value

Analysis

Variable and Analysis

Selection

Risk

Analysis

(Continuedfrom previouspage)

Figure 15. The Public Value Framework Overview (continued)


J. How to Summarize andPresent Result s?

This framework for public value assessmentpresents both problems and opportunitiesfor summarizing and reporting. Theproblems arise from the large number andtypes of results that the assessment canproduce. For presentation to policy makersand non-technical audiences, the resultsshould be as simple and accessible aspossible. Simple char ts and summary tablesare best for this purpose. For multiplestakeholders and value variables, thenumber and complexity of charts maybecome a problem. This section discussessome of the specific issues and alternativemethods available.

For qualitative variables, the presentation ofresults can show the presence of a valueresult, and information about magnitude anddirection, if relevant and available. Usingthe information in Table 3 (page 21), asummary display similar to Table 7 belowcan present the types and direction ofresults, with the estimated relative

magnitudes as well. The down arrow in thestewardship row for vendors indicates apotential loss to vendors due totransparency increases that diminishopportunities for some forms of influencingprocurement. Whether this is a positive ornegative result overall may not be clear.

For the more quantitative results, where thepublic value variables lend themselves tocalculation and statistical analysis, manypresentation and summary methods areavailable. Clear and simple visuals aregenerally preferable to tables of quantitativedata for non-technical audiences. For crosssectional data, a column chart, such as inFigure 16, can be used to present the samecomparative public value data converted toa ten-point index or scale.

Such a multidimensional chart can bedifficult to interpret, however, and some ROImethods use the so-called radar char t forthe same data, shown in Figure 17. Thiskind of char t provides a clearer image of thepattern of results for each stakeholder typeand value dimension. However, this kind ofdisplay becomes much more difficult to

stcapmIredlohekatSfoyalpsiDevitatilauQ.7elbaT



laicnaniF

lacitiloP

laicoS

lacigoloedI

cigetartS

pihsdrawetS


0

1

2

3

4

5

6

7

8

9

10

Pub

lic V

alue

Inde

x

citizens at large vendors elected officials

Public Value Index by Stakeholder Groups

Financial

Political

Social

Ideological

Strategic

Stewardship

Figure 16. Stakeholder Impact Column Chart

0

5

10Financial

Political

Social

Ideological

Strategic

Stewardship

Citizens at Large Vendors Elected Officials

Figure 17. Stakeholder Impact Radar Chart


Figure 18. Portfolio Risk and Value Comparisons - DVAM

interpret if the number of axes orstakeholders is large.

The use of indexes can also be used tocompare the relative risk and value ofalternative investments on a value scale ifthe variables can be combined into a singleindex. The Demand and Value AssessmentModel (Australia) provides guidance on howto produce such a public value index anduse it for comparison purposes. An exampleof that kind of result is shown in Figure 18above.

For cross sectional assessments, moreelaborate presentations are available,par ticularly if the value variables areindexed or based on quantitative data. TheAccenture Public Service Value Model (PSV)can use historical data about governmentprogram effectiveness and costs to showchanges in performance over time.16 Themodel is based on the principle that publicvalue is created when both outcome resultsand cost effectiveness increases. Anexample of this form of results presentation

is shown in Figure 19 for the ArizonaDepartment of Revenue. Overall, theperformance shown is improving from 1999to 2001 and 2002 to 2003 when theorganization is creating increased outcomesand doing so more cost effectively.

Documentation of results through this kindof chart or visual device should includebackground material about methods andmeasurement issues. This frameworkadvocates the use of a wide range of dataand analytical styles, many of which areconsidered controversial or suspect in someenvironments. Decision makers and analystsoften have deep-seated biases about thevalidity of qualitative data or social statistics,or other non-traditional material for ROIanalysis. Those performing a public valueassessment must, therefore, be thorough inproviding rationale and supporting materialfor all results. They must be attentive to theissues of interpretation and validity that mayaffect how key members of their audiencerespond to the assessment results.

Fin

anci

al a

nd S

ocia

l Wor

th


Figure 19. Historical Performance Change Model (PSV – Accenture)

The principles for SROI analysis used by theREDF organization provide valuableguidance for the conduct of the kind ofassessment described here. Theseprinciples can be applied to traditional ROIanalysis as well, but seem particularly wellsuited to the public value issues involvedwith this framework and related methods.The principles make an appropriate bridgefrom the general ideas and methodspresented here to the difficult work ofcarrying out public value and ROIassessment in practice.

SROI Design Principles: 17

1. Feasible – A basic SROI Analysisshould be something any organizationcan afford to prepare itself.

2. Accessible – The process should beunderstandable and relevant toorganizations at various stages ofdevelopment.

3. Rigorous – The method should besubstantive and well-executed, andbased upon premises that are validatedby informed practitioners.

4. Replicable – The framework shouldresult in similar conclusions whenapplied by different practitioners whouse similar parameters (such as thescope and options). Thus, resultsshould also be comparable over timeand among organizations, at leastamong analyses that use similar optionsand where the options are clearly noted.

5. Transparent – The process by whichthe analysis was prepared, and thecontext in which results should be seen,should be transparent.

6. Credible – The results should becredible to investors, purchasers,mangers, and other users.

7. Integrative – The framework shouldrelate to, and where possible integratewith, other approaches to understandsocial value.

8. Avoids misuse – Proper application ofthe framework should reduce the risk ofmisuse of, or misleading, SROI numbersor analyses.

-10%

-8%

-6%

-4%

-2%

0%

2%

4%

6%

8%

10%

-20% -15% -10% -5% 0% 5% 10% 15% 20%

Lower HigherCost Effectiveness

Out

com

esH

ighe

rLo

wer

1996

PSV Agency Performance Matrix (FY 1996 – FY 2003)

1997

1998

1999 2000

2001

2002

2003AVERAGE SCORE

-10%

-8%

-6%

-4%

-2%

0%

2%

4%

6%

8%

10%

-20% -15% -10% -5% 0% 5% 10% 15% 20%

Lower HigherCost Effectiveness

Out

com

esH

ighe

rLo

wer

1996


1997

1998

1999 2000

2001

2002

2003

-10%

-8%

-6%

-4%

-2%

0%

2%

4%

6%

8%

10%

-20% -15% -10% -5% 0% 5% 10% 15% 20%

Lower HigherCost EffectivenessLower HigherCost Effectiveness

Out

com

esH

ighe

rLo

wer

Out

com

esH

ighe

rLo

wer

1996


1997

1998

1999 2000

2001

2002

2003AVERAGE SCORE


9. Open source – The framework shouldcontinuously be informed and improvedby the collective wisdom of practitionersin an inclusive, iterative process.

10. Useful – Applying the framework shouldresult in information that enables usersto make decisions or take actions thatfurther their goals.


This framework grows from the rathersimple principle: that the value of a

government’s investment in IT should beassessed from the point of view of thepublic it serves. That principle leads us toidentify two distinct but equally importanttypes of public value: delivering benefits

directly to citizens and enhancing the value

of government itself as a public asset.From this seemingly simple beginninggrows the rather high level of complexityinvolved in working through this frameworkand its potential application. Thatcomplexity derives in part from the waythese ideas radically expand the possiblescope of inquiry needed to identify anddocument public value creation. Thatexpanded scope brings with it a host ofmeasurement problems that emerge whenthe many social and political outcomescome into play. The cost of this expandedand more complex assessment can be quitehigh.

So government executives and IT plannerscan rightly question whether assessingpublic value is worth the effort. Part of theanswer may be that they will have nochoice. As we noted earlier, elected officialshave begun insisting on morecomprehensive cost and return analyses forIT investment proposals. Conventionalapproaches to ROI analysis may simply beinadequate for these increased demands.

More importantly, however, the desire for amore comprehensive and robustjustification for new IT investments reflectstheir greater complexity and ambition. Thelow-hanging fruit available from earlier ITinvestments, such as establishing Webpresence and automating simple servicetransactions, has typically been harvested.More substantial improvements ingovernment are now possible by exploitingthe integrative and transformative potentialof IT, but they require much largerinvestments. These projects require a way

IV. Reflections on the Framework

and the Value of Public Value

of assessing public value that matches theirgreater scope and complexity, a way that canbuild the needed public support and guidedevelopment. In this light, the cost of using aframework such as this should be easilyjustified. Moreover, the new knowledgeabout public value possible from such anassessment can help guide other forms ofinvestment and contribute to long-termgovernment improvement.

37


A. Consult ative W orkshop

On July 13, 2005, the Center for Technologyin Government, in collaboration with SAP,hosted a consultative workshop onassessing public return on governmentinvestments in IT. A select internationalgroup of 22 experts from government,academia, and the private sector wereinvited to meet with researchers from CTG todiscuss the core issues and themes thatmake assessing public ROI for ITinvestments such a complex and difficultproblem. The workshop results were used toinform CTG’s next steps for the project. Theworkshop results were used to help developa preliminary framework for conducting ROIanalysis in the government sector anddesign the case studies research approach.This approach was employed to identify andselect the five case studies, which wereused to refine the preliminary frameworkand develop the white paper. The lists belowsummarize the key issues that emergedfrom the workshop that characterize thecomplexity of the issue. Workshopparticipants provided their recommenda-tions in the form of key themes for CTG toexplore in the follow-on research.

1. Key Issues Characterizing theComplexity of Assessing PublicROI for Government ITInvestments

n Lack of incentives to assess public ROI.

There may be no consequences forabsence of ROI or other demonstrationof results.

n Lack of historical perspective and data.

Government tends to be prospective (notretrospective), so it tends to focus on

V. Appendix: The Research Basis

for the Public Value Framework

what should be done, but not on whathas already been done.

n Governments have trouble harvesting

savings, which often get moved aroundthe budget.

n There is no straightforward quantitative

bottom line value measure for ROI inpublic sector.

n Government is multidimensional. Non-

linear, complex interactions amongbenefits—hard to measure results andlink to specific programs ortechnologies, i.e., proving the causalrelationship between the two.

n ROI requires advanced project

management and portfolio managementskills that are often lacking.

n Comprehensive ROI analysis can

greatly increase transparency ofgovernment decisions and investmentresults. This level of transparency is a“risk” in and of itself, increasing theprobability for embarrassment andcriticism leading to loss of support.

n ROI is done in a vacuum. Not focused

on the strategic investment.

n Lack of longer-term tracking and

assessment makes it difficult to build ameasurement model.

n It is hard to evaluate IT ROI elsewhere in

the government enterprise because theoutcome frameworks (intersectors)aren’t established.

38


2. Themes to Explore in theResearch

n Value and impact measurements should

take into consideration the cost impactson other business processes, byelimination or changes in the way wework together.

n Look for efficiency and cost-reduction in

non-IT areas. IT is an enabler, and thepurpose of IT is to enable other things,including changing the culture oforganization, strategies, etc. Thus, it isimportant to see how IT is enablingreturns in other areas.

n Assessment should include attention to

how particular IT investment enables ITelsewhere. ROI analysis must getbeyond evaluating things on a moretraditional basis to include questions ofoutcomes for broader range ofbeneficiaries or stakeholders.

n Social cost-benefit analysis and political

considerations involve different peoplemaking different decisions. It isimportant to understand how this mayimpact the IT aspect of ROI calculations.

n Need more attention to risks as well as

benefits. Assessment should includeattention to par ticular governmentsensitivity to risk factors.

n Method development should include

system analysis that engages a broadscope of operations. Analysis shouldinclude questions about leadership,feasibility, political support as well asresults/outcomes, how constituents andopponents will react.

n Focus assessment on question of

getting ROI on the programs, not the IT;IT doesn’t deliver the outcomes.

n Portfolio management is a big theme in

current discussion of IT management.As applied to public ROI, implies lookingat the overall picture, not just a slice ofthe project.

n ROI methods must accommodate or

provide for shared outcomes that crossover existing stovepipes.

n While benefits are measured in terms of

traditional (financial), political and socialfactors, risks are measured based onother factors: (1) technical,(2) organizational, (3) time, and (4)political.

n Find relationships between inputs and

outcomes. Once indicators aredeveloped cause and effectrelationships can be explored usingeconometric methods.

n Value of IT investment may be enabling

change.

n Find ways to ensure follow-through in

delivery and assessment of governmentprograms.

n Explore what are the attributes of a

good public ROI model: measurable indifferent ways, creates expectations ofperformance and assessment at both ITand program levels, is integrated withthe budget process, and is included in ayearly review.

39


B. Case Study Summaries 18

1. The Government of Israel’sMerkava Project

Some government IT initiatives focusnarrowly on a specific technical problem,like enabling mobile data communication orWeb-based transactions. Others are drivenby a broader, more ambitious goal, to use ITas a tool to transform government. TheMerkava Project in the Government of Israel(GOI) is most cer tainly one of thesetransformative effor ts. It is transformative inits own right, as an effort to restructure thefinancial, logistics, and human resourcecomponents of governmentwideadministration into an integrated ERPsystem. Merkava is also part of acomprehensive e-government initiative thatincludes the ERP system as one of fivelayers of new technologies and operationalsystems for enhanced internal operationsand improved benefits and services tocitizens. These layers are part of a multi-year strategic plan, described in 2002 byPrime Minister Sharon: “The e-governmentproject is an expression of the governmentgoal to provide state-of-the-art strategicsystems, while affecting social andeconomic national targets. We believe thatthis infrastructure constitutes a bettertomorrow and that tomorrow is alreadyhere.”

From its beginnings in 1999 to the present,the Merkava ERP project has been a verylarge undertaking. It is described by theMinistry of Finance as the largest single ITproject in Israeli history. It has beenimplemented or is in the roll-out process in30 government units (out of 100), andcurrently supports approximately 2,300users. The 2005-2007 IT master plan callsfor the Merkava system to be implementedin 90% of the government’s offices duringthe planning period. Work on the other fourlayers in the overall eGov initiative has been

progressing in parallel with the Merkavaimplementation. The integration provided bythe Merkava ERP provides an importantpar t of the infrastructure for operations inthe three upper layers of the initiative andcontributes to the returns that they deliver aswell. The success of implementation effor tsto date and the growing record of returns,both internal to the government and thepublic, suggest that the remaining roll-outwill continue as planned and theaccumulation of returns will continue togrow.

2. The Austrian Federal Budgetingand Bookkeeping System

How a government obtains value from itscitizens and external public environmentand how it returns value to them are linkeddirectly to its financial management systems.These systems are crucial links in the flowsof revenues to the government and the flowof expenditures and services back to thepublic. Improving financial management,therefore, has the potential to producesignificant returns in terms of both greaterinternal efficiencies and enhanced publicreturns. These were the goals of theAustrian Federal Budgeting andBookkeeping System project initiated in1997 by the Minister of Finance andsupported by the Chancellor (Prime Minis-ter). The goal of the project was to redesignand integrate the electronic workflow of thefederal government’s budget andbookkeeping processes. The strategy theychose was to implement a single ERPsoftware standard throughout the federalgovernment, along with the adoption ofnecessary legal authority. The result wasthat, by 2005, the Ministry of Financesuccessfully consolidated 85 bookkeepingunits across the federal government into onefederally owned, but privately operated,agency.

The consolidation and integration producedimmediate and tangible benefits in terms of

40


internal efficiencies. These resulted from theimplementation of a new standardized workprocess for accounting and budgetingthroughout the federal government, withreduced work process steps and processingtime. As of 2005, the legal consolidation ofthe numerous bookkeeping departmentsinto one agency, along with the technicaland organizational implementation of theERP, has resulted in annual savings ofapproximately €30 million. In terms of thesereturns to the government itself, the projectis clearly a resounding success.

In terms of broader public returns, theproject goals went beyond internal financialmanagement efficiencies. The aim ofrestructuring the Austrian FederalAccounting and Bookkeeping system wasmuch more than simply an “IT investment”from the very beginning. The bookkeepingsystem was part of a larger effort to imple-ment the SAP ERP technology throughoutthe Austrian Federal Government as part ofa governmentwide public managementreform effort. The ERP technology was onepart of a comprehensive strategy thatincluded legislative reform, governmentwidereorganization and consolidation, andimplementing a modern accounting andbudgeting standard across the government.Though the specific ERP technology wasjust one element of a larger approach,involving legal and organizationalstrategies, the ERP technology played avery important enabling role vis-à-vis thelegal and organizational strategies.

3. The Washington State DigitalArchives

Washington State’s investment in digitalarchiving for government records provides ahighly focused and successful example ofpursuing public value through informationtechnology. The job of collecting,preserving, and providing access to therecords of government is central to themission of Washington’s Office of Secretaryof State. That mission recognizes thefundamental importance of government

record keeping in a democratic society. Thatis also the foundation of the public valueproposition guiding the Digital Archivesprogram: that the state has the constitutionaland statutory mandate to preserve andprovide access to records of enduring legaland historical significance.

The growth of electronic records ingovernment agencies in the 1990’s pre-sented a challenge to the State Archives’ability to fulfill its mission, since it lacked aneffective program and technology to dealwith records in these new formats. TheWashington State Digital Archives (WSDA),a program within the Office of Secretary ofState, is the response to that challenge. Itwas initiated by the Office of the Secretaryof State, with initial planning begun by thethen State Archivist in March of 2000. Theinitiative was taken up in 2001 as a priorityby the newly elected Secretary of State SamReed, and included in the Secretary ofState’s 2001-2007 Strategic Plan. Theproject was subsequently supported by thestate legislature and included in the State ofWashington’s 2001-2003 Capital Budget.Construction of the physical hub of theWSDA in Cheney, Washington, began inJanuary 2003.

Beginning in mid-2001, the WSDA teambegan exploring a wide range oftechnologies and techniques for collection,access, and preservation. The results led tothe custom development of a Web interfaceand database design that blended the latesttechnologies with traditional archival theoryto create a first-of-its-kind digital recordsrepository for state government. The goal ofthe program was to make the historicalelectronic records of Washington’s state andlocal governments easily accessible, fromanywhere, at anytime. The initial vision andvalue proposition were carried through acomplex political and technical process to afunctioning digital archiving program andfacility delivering the promised public value.The WSDA project team began with a clearvision of the expected value of the DigitalArchives to both the government andcitizens. In addition, it successfully identified

41


the benefits that it needed to communicateto the state and local government agenciesthat were keepers of public recorders inorder to mobilize their support andparticipation. The WSDA projectdemonstrates a strong connection betweenthe initial high-level public value propositionthat motivated the project and its realizationin the performance of WSDA itself.

4. The Commonwealth ofPennsylvania’s IntegratedEnterprise System

From the beginning, the Commonwealth ofPennsylvania’s Integrated EnterpriseSystem was much more than a technologyproject. The ERP implementation continuedthrough three gubernatorial administrationswith consistent top level executive support;eventually putting in place the technicalinfrastructure and enterprise standards forcore administrative functions with improvedpublic value. Between early 2001 andmid-2004, the ERP implementation for thefive business functions was completed for53 Commonwealth agencies including all 49of the agencies under the governor’sjurisdiction.

Immediate returns in the form of improvedgovernment operations were realized soonafter implementation and continue today.However, this infrastructure also providesthe Commonwealth with capability that canbe fur ther leveraged to support additionalimprovements in government operationsthat go well beyond direct improvements incore administrative functions.

The Commonwealth has begun such efforts.The IES infrastructure provides publicreturns in the form of direct improvements inthe efficiency and effectiveness of coreadministrative functions. This infrastructurealso provides the basis for improvements inthe back office operations of other serviceareas which, in turn, offer improved servicesto the public. The Commonwealth has

recently taken steps to move in this directionby implementing the necessary institutionalstructures and policies to take fulleradvantage of this enterprise-level asset.

5. Service New Brunswick

Service New Brunswick (SNB) is well-knowninternationally for its expertise in providingmulti-channel “single window” citizenaccess to government services as well as fordeveloping and maintaining geographicinformation databases. SNB’s award-winning approach provides one-stop-shopping for different government serviceson behalf of provincial and municipalgovernment agencies, but also provides alinkage to the Canadian FederalGovernment in a “joined up” governmentmodel. SNB was launched in a time of highpressure from citizens for improved servicedelivery; today it serves the provincethrough its award winning service deliverymodel, and also and maybe moreimportantly in the long run, through itsinnovations in economic development.

The numbers behind SNB speak forthemselves; customer satisfaction numbersare the highest in Canada going from about50% in late 1980’s to 92% in 2005. Thesenumbers require that responsibleexaminations of public value include SNB.But there is more to this story than customersatisfaction ratings. The essential par t of thepublic value story in this case is ServiceNew Brunswick (SNB) as a quasi-governmental organization and its evolutionas an integrated service provider andeconomic development innovation. SNB is apublic corporation with a single shareholder– the government of New Brunswick. Thisshareholder has guided SNB with thismodel throughout its existence: make publicservice good business.

42


Endnotes1 Expenditure data from: DG Information Society, European Commission; http://ec.europa.eu/idabc/en/document/4336/5860; China: CCID Advisory (2005, May) http://www.ccidadvisory.com/report_details.php?r_serial=229; India: Public Sector Technology and Management (2004, Jul 12) http://www.pstm.net/ar ticle/index.php?articleid=236; U.S. Office of Management and Budget Web site: http://www.whitehouse.gov/omb/egov/g-9-budget_highglights.html; Datamonitor. (2005, Jun). IT opportunities instate and local government: Five-year outlook for vendors. New York: Datamonitor.

http://www.datamonitor.com/~3d016a3bb9db4c6cb608be617baf36f5~/Products/Free/Report/DMTC1150/010DMTC1150.pdf; also http://zdnetasia.com/news/business/0,39044229,39371070,00.htm

http://www.publictechnology.net/modules.php?op=modload&name=News&file=ar ticle&sid=1920

2 Matthew Weigelt, “Senate committee questions e-gov ROI.” FCW (July 17, 2006), http://www.fcw.com/ar ticle95291-07-17-06-Web&newsletter%3Dyes

3 View and download copies of the white paper and case study reports at www.ctg.albany.edu/projects/proi.

4 This terminology is consistent with recent work on assessing government IT investment. See for example:Center for Digital Government, Prove IT: The Disciplines of Harvesting Value from Public Sector Information

Technology, 2006. (www.centerdigitalgovernment.com); and eGovernment Economics Project (eGEP):

Measurement Framework Final Version. Prepared for the eGovernment Unit, DG Information Society andMedia, European Commission, by Christiano Codagnone (RSO SPA) and Paulo Boccardelli (Luiss Manage-ment), May, 2006.

5 The Toxic Release Inventory is a publicly available data base maintained by the US EnvironmentalProtection Agency that contains information on toxic chemical releases and other waste managementactivities reported annually by certain covered industry groups as well as federal facilities (http://www.epa.gov/tri/).

6 These are taken from the Business Reference Model document, which contains considerable detail on thedefinitions and details of these components (http://www.whitehouse.gov/omb/egov/documents/CRM.PDF).

7 See the European Interoperability Framework for Pan-European eGovernment Services, http://europa.eu.int/scadplus/leg/en/lvb/l24226b.htm; The UK eGovernment Interoperability Framework, http://www.govtalk.gov.uk/documents/eGIF%20v6_1(1).pdf; Malaysian Government Interoperability Framework(MyGIF), http://unpan1.un.org/intradoc/groups/public/documents/APCITY/UNPAN016393.pdf;

Hong Kong Special Administrative Region (HKSARG) Interoperability Framework, http://www.bilgitoplumu.gov.tr/kdep/34/hongkongif_v1_0.pdf.

8 The NASCIO Enterprise Architecture materials are listed at http://www.nascio.org/publications/index.cfm.The Athena Project repor ts are available at http://www.athena-ip.org.

9 The Zachman Institute for Framework Advancement. http://www.zifa.com/

10 See, for example, polling results at Zogby International (http://www.zogby.com/index.cfm), or The PewResearch Center for People and the Press (http://people-press.org/).

11 Additional information and suggestions for conducting stakeholder analyses can be found in thesesources:

http://www.ittoolkit.com/workbooks/qt_stake.pdf; http://www.stsc.hill.af.mil/crosstalk/2000/12/smith.html;http://www.boxesandarrows.com/view/understanding_organizational_stakeholders_for_design_success;http://www.ctg.albany.edu/publications/guides/smar tit2?chapter=11&section=5; http://www.ctg.albany.edu/publications/guides/and_justice_for_all?chapter=8&section=2

12 Overseas Development Administration, Guidance Note on How to Do Stakeholder Analysis of Aid Projects

and Programmes, Social Development Department, July 1995.

13 More information on these data types and analysis methods can be found in social science researchmethod texts. See, for example, Earl Babbie, The Practice of Social Research, Tenth Ed. Belmont, CA:Wadswor th, 2004.

14 Stephen Burd. “Education Department Pulls the Plug on Novel Way to Deliver Financial Aid. Chronicle of

Higher Education. Vol 46, No. 29 (March 24, 2000).

15 See US Federal CIO Council Best Practices Committee. VMM How-To-Guide. http://www.cio.gov/docu-ments/ValueMeasuring_Methodology_HowToGuide_Oct_2002.pdf

16 Accenture Public Service Value (PSV) Model, http://www.accenture.com/Global/Services/By_Industry/Government/PublicGovernment.htm

17 A Framework for Approaches to SROI Analysis. http://www.redf.org/

18 View and download copies of the white paper and case study reports at www.ctg.albany.edu/projects/proi.

43

Center for Technology in Government

187 Wolf Road, Suite 301Albany, NY 12205

Phone: (518) 442-3892Fax: (518) 442-3886

E-mail: [email protected]

Documents

Advancing Return on Investment Analysis for Government IT · 2018-06-11 · CENTER FOR TECHNOLOGY IN GOVERNMENT —ADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT : A PUBLIC