Looking for Clues to Failures in Large-Scale, Public ... · PDF fileLooking for Clues to Failures in Large-Scale, Public Sector Projects: A ... cooperative work, networks require shared

Looking for Clues to Failures in Large-Scale, Public Sector Projects:

A Case Study

Sandeep Purao Penn State University [email protected]

Kevin Desouza University of Washington [email protected]

Abstract We describe results from historical analysis of a

large-scale, public sector effort: the IRS Modernization Project that has already spanned a decade and consumed more than 3 billion dollars. The results focus on analysis of Sentiments and Confidence expressed by different stakeholders, as found in various documents. We explore how such analyses may provide a window on project progress and potential early clues that may contribute to preventing undesirable outcomes in the future. 1. Introduction

The Internal Revenue Service (IRS) modernization project has been the poster child for academics, practitioners and consultants intent on showing the many things that can go wrong with large information systems and software implementation projects [6]. In this paper, our focus is different. We are interested in considering this project as an example of a public sector project that failed, and analyzing its progression during the decade that might provide us to clues that explain the failure.

The public sector, as the phrase suggests, is that part the federal, state, regional or local government machinery that deals with either production or delivery of infrastructure, goods and services for the government itself or its citizens. Contemporary definitions of the public sector broaden it to include community-led and citizen-led activities [27] that receive funding from government agencies, broadening the reach of the legal, moral and accounting canon. Examples of activity undertaken in the public sector, therefore, include delivery of societal benefits, building and maintaining urban and rural infrastructures, national defense and treasury.

Much activity in this sector is marked by public, instead of private, ownership. This is achieved via one of three types of entities: (a) direct involvement of a

government agency and funding via taxes; (b) governance via state-owned enterprises to ensure separation of daily activities from the longer-term goals of the government; (c) citizen or community-led organizations that rely on partial funding or access to public resources from government agencies [1, 2].

A public sector project, then, is a project that is undertaken, funded and managed by one or more entities in the public sector. Economic theory suggests that large infrastructure projects, physical or technological, are often undertaken by the public sector because of the heavy investments needed and because the business case often includes societal benefits. These large-scale projects tend to be complex undertakings. Flyvbjerg et al. [9] use the term megaprojects to describe these ultra-large projects ($1 billion+) that continue to have a non-optimal track record. Contemporary research describes these projects in terms of the number and variety of stakeholders, how they can benefit or harm multiple sections of the populace, and funding sources. Agranoff and Mcguire [1] describe several priorities for arriving at a greater understanding of these public sector projects, including the following.

First, a network-perspective suggests that alternatives to traditional planning-organizing-control cycles are needed to manage activities carried out in public sector projects. A possible alternative is found in behaviors such as activation, framing, mobilizing and synthesizing of participants and stakeholders in a network based on identification of skills, knowledge and resources needed to accomplish the tasks [3]. Second, based on analogies to groupware and cooperative work, networks require shared learning and shared commitment to succeed [19]. Third, flexibility and adaptation are critical for networks because changes can propagate across tasks, roles, responsibilities or even overall goals [31]. Fourth, accountability poses difficult challenges in public networks that are made up of government agencies as well as private contractors who may themselves create political ties with policymakers to outflank

Proceedings of the 44th Hawaii International Conference on System Sciences - 2011

11530-1605/11 $26.00 © 2011 IEEE

administrative overseers and even re-shape the project objectives [23]. Finally, legal-rational authority structures and trust may be seen as complements or competing [31].

The perception, attitude and sentiment of different stakeholders, therefore, can be an important indicator of project progress. The stakeholders can essentially act as distributed sensors through the project. It is, however, difficult to ascertain how these attitudes and sentiments, and understand how they might predict project progress. With hindsight though, one can make sense of why certain attitudes changed and based on these, speculate about possible clues that may be seen as early indicators or early warnings about project progress. This paper takes initial steps in this direction by outlining a technique to ascertain attitudes and sentiments of stakeholders, and speculating about how these can provide clues to failures in large-scale, public-sector projects. In doing so, we attempt to provide an alternative to the conventional wisdom and state-of-the-art about project management practices [24]. The specific case we study is the IRS Business Systems Modernization (BSM) project [29]. The research questions of interest to us is the following: • How can stakeholder sentiments be ascertained in a semi-automated manner? And • What clues may be discerned about project progress from an analysis of stakeholder sentiments?

The discovery of such clues may eventually allow us to predict impending project failures or identify project risks that can be mitigated before they threaten the project. The techniques we suggest for Sentiment and Confidence analysis include content analysis (see [35]), extended and refined for the purpose of this research. With these techniques, we analyze publicly available assessment documents and press reports to map the sentiments expressed by different stakeholders over time. The key contribution of this research is a demonstration of a plausible technique, based on publicly available documents, for sentiment analysis, and informed speculation about the use of such analyses for uncovering clues about project progress. 2. The IRS BSM Project

The IRS BSM project was undertaken in response to the bill, signed into law by President Clinton on July 22, 1998. The program was considered inherently important to IRS because it directly reflects and responds to the IRS’s stated mission: “provide America's taxpayers top quality service by helping them understand and meet their tax responsibilities and by applying the tax law with integrity and fairness to all” [21]. In response to this mandate for systems

improvement, the IRS entered into a contract with Computer Sciences Corporation (CSC), who was charged with assembling a team of contractors, known as the PRIME Alliance, to develop new systems for the IRS to achieve modernization [36, 22]. The IRS also created a BSM Office that was charged with leading the IRS’s acquisition and implementation of new technology [36]. Figure 1 shows the cumulative amount of funding spent on the program to date [29].

Since its inception, the BSM program was plagued by delays, scope reductions, funding reductions, and poor costing [10, 11, 12, 13, 14, 15, 16, 17, 18, 36] . As late as April 2007 [28] several systems considered a part of the BSM program were incomplete and behind schedule. Phillips [28] describes several of these. One module that is clearly illustrative of the problems the IRS has faced with the BSM program is the Customer Account Data Engine (CADE). This module, designed to process and store income tax returns, was supposed to be the ‘centerpiece throughout the life of the Modernization Program’ [29]. As late as 2009, only 30% of individual income tax returns were processed through this module [29]. In the early stages of the BSM program, the IRS had originally projected that this module would be complete by 2005. In 2007, the IRS changed its estimated completion date for this module to 2012 [15]. By June 2009, the IRS had suspended development of the module, declaring that ‘the immediate challenge is the future of the Customer Account Data Engine’ [29].

The importance of the project is underscored in the annual audits by the U.S. Treasury Inspector General for Tax Administration from 1999 to 2009, which identify it as the highest priority challenges facing IRS management in each year [10, 11, 12, 13, 14, 15, 16, 17, 18] except 1999, the onset of the BSM program, and 2002, when security became a higher priority issue given the 9/11 disaster.

2.1 Stakeholders in the BSM Project

Figure 1. BSM: Cumulative Expenditures

As a complex Public Sector Project, the BSM effort included not only the IRS but a number of other, external stakeholders such as the following:


2

The Treasury Department, which oversees the IRS,

responsible for activities including but not limited to managing federal finances, supervising national banks, designing currency, and advising on economic policy (U.S. Department of the Treasury 2008). The Restructuring Reform Act (RRA 98) required the Treasury Department to create two roles which have published frequent reports about the BSM program. These roles include: (a) a Treasury Inspector General for Tax Administration (TIGTA), reporting to the Treasury Secretary, to conduct audits, investigations, and evaluation of IRS programs and operations [34]; and (b) an IRS Oversight Board, consisting of nine members – six who are neither federal officers nor federal employees, one full-time federal employee, the Treasury Secretary (or Deputy Secretary if responsibility delegated), and the IRS Commissioner – to review strategic plans and operational functions of the IRS [34].

The US Congress, which passes tax laws that taxpayers are required to comply with. The IRS is responsible for incorporating new tax laws into its systems, communications, and procedures in time for taxpayers to comply with these new laws. After receiving a proposed budget from the US President’s office each year, the US Congress is also responsible for approving the level of funding for federal programs, including the IRS BSM program.

The General Accounting Office (GAO), who was asked to monitor the progress and performance of BSM and report on it because the US Congress deemed it sufficiently important. Similar to the TIGTA and the IRS Oversight Board, the GAO has published frequent reports addressing the BSM program, at the request of the US Congress.

Contractors:, to whom the work was outsourced by the IRS in addition to having its internal IT staff work on it. Computer Sciences Corporation (CSC) was contracted by the IRS to lead the PRIME Alliance. In the early stages of the BSM program, CSC was responsible for hiring other contractors and coordinating their work [13]. However, in early 2005, the IRS scaled back the role of CSC and took on the role of system integrator, leaving CSC largely in a systems development role [13]. In addition to CSC, numerous other contractors have also been involved with the BSM program in different capacities.

Taxpayers, as users of many of the BSM systems, clearly represented an important stakeholder group. TIGTA reports reflect that the IRS typically defines BSM program goals, at least partly, in terms of taxpayer benefits [11, 12, 13, 14, 15, 16, 17, 18, 36]. Taxpayers also indirectly fund BSM systems projects with taxes paid.

In addition several internal stakeholders could be

identified for the BSM program, including: Senior Management, with the IRS Commissioner

holding the most senior position within the IRS. Charles Rossotti, IRS Commissioner from 1997 to 2002, initialized the BSM project and championed it in his appearance before the Senate Finance Committee [30]. He handed the mantle to Mark Everson in 2003, who was succeeded in 2008 by Douglas Shulman. During this time, five CIOs assumed office, and in late 2008, an additional role of a CTO was filled [22]. Figure 2 shows changes in leadership for the Commissioner and CIO/CTO roles at the IRS during this time [29].

The bill also required the IRS to reorganize so that

it could better serve taxpayers [36]. In October 2000, the IRS created Taxpayer Divisions to serve specific segments [36] such as a Wage and Investment to serve taxpayers with wage and investment income only; Small Business / Self Employed to serves taxpayers that are self-employed individuals or small businesses; and others. These divisions continue in 2010 in the Services and Enforcement major division under the IRS Commissioner. They represent important

Figure 2. BSM: Changes in Leadership


3

stakeholders for the BSM program because they are highly involved in defining business requirements.

The Internal IT organization, which included the following [36]: (a) Information Systems organization, responsible for meeting the IRS’ information technology needs [36], (b) Business Systems Modernization Office, responsible for IRS’s acquisition and implementation of new technology and changes in business practices [36], the Modernization and Information Technology Services (MITS) organization, later restructured MITS to include: (a) Applications Development: responsible for building, testing, and maintaining IT application systems; and (b) Enterprise Services: responsible for system engineering, architecture, and program management, and Program Integration and Program Management Offices [29].

2.2 Challenges in the BSM Project

Reports from TIGTA provide a chronological view of challenges to the BSM project over the last decade (see Table 1).

Table 1. BSM Program: Key Challenges

Source Key Challenges 2000 TIGTA Report

• Inability of IRS to oversee a program as large as the BSM program.

• Working relationship between IRS and the PRIME contractor not established.

• Y2K compliance testing underway. • Waiting for major IRS reorganization before

IRS can proceed with program. 2001 TIGTA Report

• Some initiatives scaled back/delayed due to unstable program management.

• Program management staffing needs not yet determined.

2002 TIGTA Report

• Difficulty in implementing defined and repeatable processes.

• Steep learning curve for PRIME contractor. • Increasingly complex environment. • Human resources needed to ensure process

changes implemented. 2003 TIGTA Report

• Project teams not using defined project mgmt and development processes.

• Ineffective contract management • Established testing processes not being

followed. • Imbalance between the BSM projects and

available management capabilities. • CADE release delayed, revising schedules

for dependent projects. • Significant leadership changes, changes in

strategic direction. 2004 TIGTA Report

• Repeat of problems identified in 2003. • Delays due to reassessing strategic

direction.

2005 TIGTA Report

• Continued weaknesses in requirements management, contract management, software testing, and security controls.

• Measurement plan to determine progress on BSM plan not in place.

• Additional requirements, Delays in passing legislation by Congress.

• Budget reductions from Congress. • Specialized skills difficult to fill.

2006 TIGTA Report

• Continued weaknesses in process and contractor management.

• Adjustment period for reorganization. 2007 TIGTA Report

• Repeat of weakness identified in 2006. • Continual changes in management. • Changes in SEI’s CMM: IRS status needed

to be reassessed. 2008 TIGTA Report

• Struggles with the requirements management process.

• Federal Acquisition Regulation for Procurements not followed.

• Continued reductions in funding. 2009 TIGTA Report

• Development of long-term systems requirements challenging.

• Continued weaknesses in program and contract management.

• Problems identified with security controls. • IRS acknowledges that a “strategy

correction” is needed. • Difficulties in obtaining qualified personnel

and sufficient funding.

None of the challenges outlined appear to be unique, i.e., they reflect what has been known in prior research related to software engineering and project management elsewhere (see, e.g. [7]). Our efforts in this research were, therefore, aimed at understanding how different stakeholders perceived these challenges and their attitudes towards resolving these. The next section outlines the methodology we followed to uncover these concerns. 3. Methodology

The methodology followed for this research combined case study [37] along with historical analysis of documents (such as press releases, audit reports and the like) created by stakeholders in the BSM program. We acknowledge that these documents are unlikely to reveal the stakeholders’ personal agendas, the full nature of inter-personal communication, compromises incentives or secret caucus results. Others who study historical organizational processes express similar caveats (see, e.g. [25]).

This study does, however, allow us to go beyond the usual suspects of scale, complexity, number of stakeholders and other ongoing problems (see Table 1 above) that suggest that the IRS Business Systems


4

Modernization project may have been doomed from the beginning. The documents analyzed in the study came from the following sources.

Table 2. Sources of Documents

Source Documents TIGTA 48 GAO 43 CSC 9 Government Computer News 169 Accounting Web 6 GovExec / NextGov 28 IRS Press Release 10 Washington Technology 32

These are publicly available documents that

describe assessment of project progress by different stakeholders. The specific technique we used to analyze the documents was Sentiment and Confidence Analysis (see [35]). Sentiment analysis is a methodology that computes the strength of sentiment expressed in a document by analyzing the content in the document. The list of tags used for these analyses was obtained from the General Inquirer set of tags. These tags are commonly used words in English that, if present in a document, likely indicate the tone of the document. For example, the presence of certain words may suggest that the tone of the document is positive or negative. Similarly, another set of tags, if present in a document, likely indicate that the language in the document is understated or overstated. Table 3 shows a representative sample of these words. See http://www.wjh.harvard.edu/~inquirer/homecat.htm for a comprehensive list.

Table 3. Example words used for analysis

Positive Negative Understated Overstated Assurance Challenge Adequate Complete Better Concern Approximately Critical Complete Cost Despite Final Effective Delay Limit Full Improve Difficulty Unclear Important Progress Failure Small Necessary Success Lack Somewhat Overall Support Problem Suggest Significant

Enhancements to the list of words came mainly in

the form of discarding certain words. For example, based on the corpus of text from the documents, we found that certain words had specific meanings for the IRS BSM program. For example, the word ‘Tax’ was considered a negative keyword in the original set. However, its occurrence in the BSM documents was not tied to this meaning. It was, therefore, removed.

Other examples included the word ‘Oversight,’ because of the IRS Oversight Board as a stakeholder; the word ‘Return,’ which was removed as a positive keyword because of its connotation as a tax return; the word ‘Prime,’ which was removed as a positive keyword because it referred to the Prime contractor for the BSM program.

Based on the enhanced set of words, we devised two measures: one to calculate the relative Sentiment (Positive vs. Negative) in each document, the other to compute Confidence (Overstated vs. Understated). More specifically, for each document, the net percentage of positive less negative word occurrence was calculated to indicate Sentiment, and is the net percentage of overstated less understated word occurrences was computed to indicate Confidence. Sentiment = (# Net Tokens that are Positive) / (# Tokens) Confidence = (# Net Tokens that are Overstated) / (# Tokens)

The intended analysis for each document was, therefore, meant to produce a placement for each document in a two-dimensional space as shown in Figure 3. For example, we might expect that a Report from the Prime Contractor may appear as Positive Understated if the Prime Contractor follows the strategy of promising low and delivering high. On the other hand, an audit report may be cautious and may highlight negatives, suggesting a placement in the Negative Overstated quadrant. Over time, the placement of documents in the quadrants would reveal

the trajectory of progress. Seeing the trajectory from the perspective of different stakeholders would allow an analysis of how each saw its own participation in the project and assessed project progress.

A corpus of text was constructed from the documents by extracting text tokens with simple text

Figure 3. Framework for document analysis


5

processing mechanisms. The set of tokens produced was then used to compute the above measures. 4. Results We describe the results first, in terms of a longitudinal sentiment analysis that describes stakeholder perspectives (see Figure 4). Each data point in the figure corresponds to a specific document. For example, during the year 2000, immediately after the inception of the BSM program, the documents show

that the sentiment measure was largely positive with a lone document suggesting a negative sentiment. The figure shows that few documents betray a negative sentiment.

In fact, most documents show a positive sentiment although a creeping negative sentiment is evident as the project progresses. For example, the years 2001 and 2002 continue to point to just one or two documents with a negative sentiment albeit with increasing levels of negative sentiment.

The figure also shows a trendline that suggests a

very gentle downward slope, that is, a slow deterioration in sentiment. Two other shorter-duration trends are visible in the figure. One appears as deterioration during the years 2000 to 2002 (note the three data points at the bottom for these years); another mirrors this during the years 2004 to 2006 (once again, as three data points at the bottom for these years). The two short-duration trends are followed by a more balanced and positive outlook during the years 2007 to 2009.

This, general outlook on the BSM project, without distinguishing the perceptions from different stakeholders suggests the following interpretation for the project. The project followed a cyclical pattern with a significant deterioration in stakeholder

sentiment over a time-window of about every three years. It may be possible to link the revival in the second time window to the changes in leadership (see Figure 2 earlier). The third time window (years 2007-2009) may be seen as that corresponding to the leadership of Art Gonzales as the CIO. With a stable set of leaders in the IRS Commissioner and the CIO, the pattern of deterioration in the first two cycles is being averted in the third cycle.

One possible interpretation of the data is that the scale and complexity of the project is reflected in the cycles: the stakeholder network is fully informed about the project and is able to factor in the information available in its own assessments of the project As the information trickles down to the stakeholders and influences stakeholder sentiments,

Figure 4. Longitudinal Sentiment Analysis


6

the leadership is able to react and institute new practices to address stakeholder concerns. This cyclical view may provide a clue to recognizing temporal patterns in large public sector projects, which may be managed to reduce the risks. To further understand how different stakeholders may perceive the project and whether these differences in

perspectives may provide additional clues to project progress, the data was also analyzed and mapped to assess whether stakeholder sentiments were different across the different stakeholder groups. Figure 5 shows these results.

The figure shows a preponderance of reports from

the Government Computer News that are in the top left quadrant, that is, Positive and Overstated (shown with triangles). However, reports from the same source also appear in the other three quadrants. In contrast, the few reports from the Prime Contractor (shown with dark squares) all appear in the top left corner, highly Positive as well as Overstated. A comparison with the reports from the Government Accountability Office (shown with circles) points to the almost uniform toning down of Positive sentiment. The press releases from IRS (shown with lighter squares) also appear as Positive and Overstated but less so compared those by the Prime Contractor. The analysis suggests that there are clear differences across stakeholder groups.

One possible interpretation of the data is that assessing project health may require not just recognizing that different stakeholders may view it differently but also appreciating how their roles

influence the positive or negative assessment. Further, the views expressed by three stakeholders – TIGTA, GAO and the Government Computer News – all appear to take on the omniscient observer role, that is, each of these stakeholders represent a perspective that is external to the project. The TIGTA acts as entity responsible for overseeing the project, the GAO is the entity responsible for ensuring that the resources are being expended appropriately on the project, and the Government Computer News fulfills the journalistic mission. In spite of the apparent similarities across these three stakeholders, their assessment, as seen in the sentiment analysis differs considerably.

Differences across the assessments from these stakeholders will need to be reconciled to understand project progress. The differences may also point to the differences in the extent of information available to each stakeholder and also how the information is interpreted.

Figure 5. Sentiment and Confidence Analyis based on Stakeholder Perspectives


7

In contrast, the assessments from Press Releases from the IRS and the Prime Contractor appears to dovetail much more, which may suggest that the internal stakeholders are more in sync with the contractors who are working on the project. The analysis we have conducted does not account for the distance across the sentiments expressed by these stakeholders. This can provide further clues about how the sentiment analysis can be used to assess project progress. Table 4 summarizes interpretations from the analyses above.

Table 4. Possible Clues to Project Progress Analysis Interpretation Longitudinal analysis of sentiments

Assessment of overall project progress

Shorter cycles of stakeholder sentiments

Changes in stakeholder roles

Differences in stakeholder sentiments

Understanding formation of stakeholder networks

Differences in stakeholder sentiments

Triangulation of information available to different stakeholders

These interpretations will need to be confirmed

by triangulating against additional evidence. However, they suggest alternatives that may be used to understand project progress and predict emerging risks based on the techniques we have outlined.

5. Discussion

The central motivation of this research was to explore the possibility that Sentiment and Confidence analysis techniques based on historical analysis of documents can provide clues to project progress and along with it, to impending risks. The preliminary results we have shown suggests that longitudinal analysis of these properties as well as comparison of these properties across stakeholder groups can provide clues for understanding project risks.

It is important to juxtapose these results against conventional wisdom related to the study of risks in general and those related to information systems development. Beyond the basic ideas related to risks such as the probability of loss, more recent work points to notions such as systemic risks [5] and risk society [4] that expand our understanding of risk. Much work related to information systems development risks does recognize that risk perception and mitigation must address how different stakeholders may perceive risk. However, much of this literature positions the source of risk as external and continues to negate the endogenous nature of risk variables [26]. The ideas presented in this work

suggest that it may be possible to characterize risks that may arise from stakeholder actions by following their expressions of sentiment and confidence. These expressions can provide clues to emerging risks. With the techniques we have outlined, it is possible to derive clues that are more predictive instead of post-fact. A number of additional analyses (without sophisticated text processing) are possible following the sets available as part of the General Inquirer (see http://www.wjh.harvard.edu/~inquirer/homecat.htm). Additional analyses are also possible by combining and extending the results presented in this paper. For example, combining and comparing different stakeholders by examining their sentiments over time can provide clues about how stakeholder incumbent and role changes may result in stakeholder networks.

The research presented in this paper has important implications for the management of public sector projects and the viability of public sector networks. The IRS BSM project shares several similarities with another public sector project that we examined, the Seattle Monorail Project (SMP) [38]. The SMP never got off the ground due to lack of cohesion among stakeholders. The central authority, the SMP Board, failed to create an environment where the various stakeholders could collaborate fruitfully towards achieving the goal of the project. In the SMP, just like the BSM project, there were a number of opportunities for the SMP Board to analyze the sentiments of the various stakeholders as evidenced through press releases, blogs, testimonies, etc to understand the challenges faced. Monorail supporters and activist praised the project and touted its strengths, while the anti-monorail pressure groups focused on the negatives and overstated the limitations. At the end, as noted in [38], the failure of the SMP Board to manage interactions between the stakeholders, and not between the stakeholders and the focal organization (i.e. the SMP Board) eventually led to the downfall of the project.

The study of sentiments may provide early warnings about such events because they may tell us more about the formation of and changes in public sector networks, i.e. the configuration of organizations that must collaborate on multiple projects. Consider the case of the US Intelligence Community (USIC) [8]. The USIC has been plagued by a number of major fiascos from the inability to assess the terror threat prior to 9/11 to the faulty assessments regarding Iraq’s WMDs and most recently, the underassessment of Iran’s nuclear capability. The Office of the Director of National Intelligence (ODNI) was stood up after 9/11 to coordinate the sixteen agencies and instill a sense of collaboration. However, to date, the USIC is far from


8

an integrated community. Since its creation, the ODNI has witnessed the departures of several Directors. Turf battles have also persisted. Being an information-laden network, one avenue where battles among the various agencies play out is in ‘reporting.’ Reports are a critical ingredient within the USIC. As noted by [8], agencies often express sentiments of agreement/disagreement and either exaggerate/understate threats based on their analysis of local information (i.e. information available to each agency). However, it is the job of the ODNI, and the President, to make sense of these sentiments to arrive at actionable knowledge that informs policy decision. To date, there has been a conspicuous absence of sentiment analysis when examining the classic cases of intelligence failures. The study of sentiments and the identification of early warning signs (e.g. agreements, conflicts, etc) can enable policymakers to better process information because they will have the necessary critical context for sense-making. In addition, if we take the study of sentiments seriously, we may be able to develop ‘education’ and ‘training’ tools to bridge communities and build collaboration, while avoiding the downfalls of collusion and groupthink.

One caveat about the work presented in the paper is the concern about generalizability of results. As we continue this stream of research, we hope that we will be able to uncover more specific clues, and confirm them based on comparative studies of multiple confirmatory as well as competing cases. The addition of cases that exhibit varying degrees of success will allow us not only the ability to generalize but also fine tune our interpretations. These remain on our future research agenda.

6. Acknowledgements

We acknowledge the able assistance of Karin Lamb McConnell who worked on the data collection and preliminary analysis of the BSM case. 7. References [1] Agranoff, R. and M. McGuire, 2001. “American

federalism and the search for models of management,” Public Administration Review 16, 18-41.

[2] Agranoff, R., and M. McGuire. 1998. Multi-Network Management: Collaboration and the Hollow State. Journal of Public Administration Research and Theory 8(1): 67–91.

[3] Agranoff, R., and M. McGuire. 1999. Managing in Network Settings, Policy Studies Review, 8(1): 67–91.

[4] Beck, U. 1992. “From Industrial Society to the Risk Society: Questions of Survival, Social Structure and

Ecological Enlightenment,” Theory, Culture & Society 9, no. 1: 97-123.

[5] Carlo, J.L., Lyytinen, K., Boland, R.J. 2004. "Systemic Risk, IT Artifacts, and High Reliability Organizations: A Case of Constructing a Radical Architecture," Case Western Reserve University, USA . Sprouts: Working Papers on Information Systems, 4(4). http://sprouts.aisnet.org/4-4

[6] Charette, R.N., 2005. Why Software Fails. IEEE Spectrum.

[7] Curtis, B., Krasner, H. & Iscoe, N. 1988. A field study of the software design process for large systems. Communications of the ACM, 31, no. 11, 1268–1287.

[8] Desouza, K.C. Information and Knowledge Management in Public Sector Networks: The Case of the US Intelligence Community, International Journal of Public Administration, 32 (14), 2009, 1219–1267.

[9] Flyvbjerg, B. et al. 2003. Megaprojects and Risk: An Anatomy of Ambition. Cambridge University Press.

[10] Gardiner, P. (2002, October 15). Management performance challenges facing IRS FY2003. http://www.treas.gov/tigta/management/management_fy2003.htm

[11] Gardiner, P. (2003, October 15). Management performance challenges facing IRS FY2004. http://www.treas.gov/tigta/management/management_fy2004.htm

[12] Gardiner, P. (2004, October 26). Management performance challenges facing IRS FY2005. http://www.treas.gov/tigta/management/management_fy2005-long.htm

[13] Gardiner, P. (2005, August 10). Annual assessment of the Internal Revenue Service’s Business Systems Modernization Program. http://www.treas.gov/tigta/auditreports/2005reports/200520102fr.pdf

[14] George, J.R. (2005, October 27). Management performance challenges facing IRS FY2006. http://www.treas.gov/tigta/management/management_fy2006.htm





[19] Innes, J. and D. Booher, 1999. “Consensus building and complex adaptive systems: A framework for evaluating collaborative planning,” American Planning Association. Journal of the American Planning Association 65, no. 4 (Autumn 1999): 412.


9

[20] Internal Revenue Service. (1998, December 9). Award/Contract between the Internal Revenue Service and Computer Sciences Corporation. http://www.irs.gov/pub/irs-procure/prime-contract.pdf

[21] Internal Revenue Service. (2007, November 5). The agency, its mission and statutory authority. http://www.irs.gov/irs/article/0id=98141,00.html

[22] Internal Revenue Service. (2008, November 15). Terence V. Milholland to serve as IRS Chief Technology Officer. http://www.irs.gov/pub/irs-news/ir-08-129.pdf

[23] Kettl, D. 2006. “Managing Boundaries in American Administration: The Collaboration Imperative,” Public Administration Review 66 (December 2006): 10.

[24] Kerzner, R. 2010. Project Management: Best Practices: Achieving Global Excellence. Wiley.

[25] Langley, A. 1999. Strategies for Theorizing from Process Data. The Academy of Management Review, Vol. 24, No. 4 (Oct., 1999), pp. 691-710.

[26] Lyytinen, K., Mathiassen, L., and Ropponen, J. 1998. Attention Shaping and Software Risk – A Categorical Analysis of Four Classical Risk Management Approaches, Information System Research (9:3), March, pp. 233-255.

[27] Nowell, B. 2010. “Out of Sync and Unaware? Exploring the Effects of Problem Frame Alignment and Discordance in Community Collaboratives,” Journal of Public Administration Research and Theory 20, no. 1 (January 2010): 91.

[28] Phillips, M. (2007, August 24). Annual assessment of the Internal Revenue Service’s Business Systems Modernization Program. http://www.treas.gov/tigta/auditreports/2007reports/200720121fr.pdf

[29] Phillips, M. (2009, September 14). Annual assessment of the Internal Revenue Service’s Business Systems Modernization Program. http://www.treas.gov/tigta/auditreports/2009reports/200920136fr.pdf

[30] Rossotti, C. (1998, January 28). Statement of Charles O. Rossotti, Commissioner of Internal Revenue Service, before the Senate Finance Committee, January 28, 1998. http://www.irs.gov/pub/irs-news/ir-98-3.pdf

[31] Sahin, B. 2009. “Factors influencing effectiveness of interorganizational networks among crisis management organizations: A comparative perspective” (University of Central Florida, 2009).

[32] U.S. Department of the Treasury. (2008, October 23). Duties & functions of the U.S. Department of the Treasury. http://www.ustreas.gov/education/duties/

[33] Watts, L., Groen, Y., Matsubara, K., Hovey, J., & Whittle, M. (2005). Internal Revenue Service Restructuring and Reform Act of 1998. http://www.pmstax.com/gen/bull9808.shtml

[34] Watts, L., Groen, Y., Matsubara, K., Hovey, J., & Whittle, M. (2005). Internal Revenue Service Restructuring and Reform Act of 1998. http://www.pmstax.com/gen/bull9808.shtml

[35] Weber, R. 1996. Basic content analysis (SAGE, 1996).

[36] Williams, D. (1999, December 1). Management performance challenges facing IRS FY2000. http://www.treas.gov/tigta/management/management_fy2000.htm

[37] Yin, R. K. 2003. Case Study Research. Sage Publications

[38] Yuttapongsontorn, N., Desouza, K.C., and Braganza, A. Complexities of Large-Scale Technology Project Failure: A Forensic Analysis of the Seattle Popular Monorail Authority, Public Performance & Management Review, 31 (3), 2008, 443-478.


10