Journal of High Technology Management Research 25 (2014) 21–35

Contents lists available at ScienceDirect

Journal of High Technology Management Research

Managing large-scale IS/IT projects in the public sector:Problems and causes leading to poor performance

Peerasit Patanakul 1

Black School of Business, The Pennsylvania State University, 5101 Jordan Road, Erie, PA 16563, United States

a r t i c l e i n f o

E-mail address: pxp25@psu.edu.1 Tel.: +1 814 898 6534.

1047-8310/$ – see front matter © 2013 Elsevier Inc. Ahttp://dx.doi.org/10.1016/j.hitech.2013.12.004

a b s t r a c t

Available online xxxx

With a large scope and high degree of complexity, managing large-scale projects is a challenge toproject managers. The challenge is even greater when it comes to public sector projects due tothe involvement of many stakeholders and the need to manage various relationships. For thesereasons, many projects ended up with poor performance. Research has shown that success inmanaging large-scale projects requires a great deal of coordination and collaboration which canbe done through established processes, strong teams, and involvement of stakeholders. Eventhough these processes and approaches are known, effectively implementing them is verydifficult. The objective of this study is to investigate the management of selected large-scale IS/ITprojects in the public sector in order to identify common problems and causes leading to poorperformance. Fourteen projects from the US, UK, and Australia were studied, making thisresearch among the few studies to investigate large-scale IS/IT projects in the public sector fromdifferent countries. The research results indicate common problems related to system design andimplementation, project management and governance, and contract management. Theoreticalcontributions and implications for practitioners are also discussed.

© 2013 Elsevier Inc. All rights reserved.

Keywords:MegaprojectsLarge-scale government projectsIS/IT projectsPublic sectorProject performance

1. Introduction

It has been known in the communities of researchers and practitioners that many IS/IT projects failed (Nelson, 2007). To theorganization, these failures often lead to financial loss, including significant losses in opportunity, competition, productivity, andemployee morale (Williams, 2005). The losses are even more significant in the case of the large-scale IS/IT projects in the publicsector (GAO, 2005a,b; Nelson, 2007). In general, a large-scale public project is often characterized as uncertain, complex,politically-sensitive and involving a large number of partners. Typically, these projects are commissioned by governments anddelivered by private enterprises (Clegg, Pitsis, et al., 2002). Many of these projects attract public attention because of theirsubstantial impacts on communities, the environment, and budgets (Flyvbjerg, Bruzelius, et al., 2003). Sometimes, such projectsare called major programs, or megaprojects. However, the word ‘megaproject’ often refers to an extremely large scale project. Itscost is usually more than one billion dollars (Flyvbjerg et al., 2003; Koppenjan, 2005).

The fact that many IS/IT projects failed have motivated researchers and practitioners to investigate the underlined problems ofpoor performance. While practitioners conducted retrospectives—that is project postmortems, performance reviews, or lessonslearned sessions (Nelson, 2005), several researchers conducted studies to identify the causes of failure, critical success/failurefactors, and approaches contributing to project success (Yeo, 2002). However, these studies emphasized on projects with asmaller scale. Few studies were investigated such issues in the context of large-scale projects (Nelson, 2007). With the largerscopes and more parties involved, the management of large-scale IS/IT projects is much more challenging than the managementof typical internal IS/IT projects. This leads to a question whether or not the findings from the studies of the smaller scale IS/IT projects

ll rights reserved.

22 P. Patanakul / Journal of High Technology Management Research 25 (2014) 21–35

are applicable to the larger scale projects. On the other hand, while research on poor performance of large-scale projects is quiteextensive (Clegg et al., 2002; Flyvbjerg et al., 2003; Han, Yun, et al., 2009; Koppenjan, 2005; Marrewijk, 2007), those studies putmore emphasis on the context of public infrastructure projects. Limited studies were conducted on IS/IT projects (Mora, Gelman, etal., 2008). Since many large-scale IS/IT projects still failed (GAO, 2005a,b; Nelson, 2007), both researchers and practitioners cannotignore the fact thatwemay need different managerial approaches, including tools and techniques, for managing large-scale IS/IT projects.Prior to the development of the new approaches, the underlined problems of poor performance must be investigated. Lack ofresearch on this issue motivated the author to conduct this study.

The objective of this study was to investigate the management of large-scale IS/IT projects in the public sector to identify theproblems and causes leading to poor performance. Fourteen projects from the US, UK, and Australia were randomly selected as a partof this study. The projects are large-scale multi-million dollar IS/IT projects commissioned by government agencies and delivered byprivate enterprises. The projects are complex and involve multiple partners. The anticipated contribution of this research centers onthe identification of the problems in managing such projects that can be a foundation for future research, including the developmentof new managerial approach for managing large-scale IS/IT projects. Also, the research findings provide practitioners with a betterunderstanding of common challenges in managing large-scale IS/IT projects so that practitioners are aware of these problems andtake appropriate measures to address them.

2. Background

In this section, the literature regarding the performance of large-scale projects in general and the performance of IS/IT projects isreviewed.

2.1. Performance of large-scale projects

Management of large-scale projects, including megaprojects, has been of interest to many researchers for decades. Studies haveshown that a majority of megaprojects have ended up with poor project performance and many researchers have analyzed thecauses of this performance such as cost overrun and schedule delay. Flyvbjerg et al. (2003), for example, suggest that the maincauses of cost overruns stem from a lack of realism in initial cost estimates, low level of contingencies, insufficient consideration ofchanges in project specifications, designs, and exchange rates, and undervaluation of price changes, expropriation cost, and safetyand environmental demands. Analyzing the schedule delay of the Korea Express Train project, Han et al. (2009) indicated that majorcauses of schedule delay arise from lack of owner's abilities and strategies to manage hi-tech megaprojects, lack of appropriatescheduling tools, underestimating the technical requirements, and public resistance due to environmental concerns.

Project complexity is also identified as a root cause of poor performance of large-scale projects. Researchers agree that the level ofcomplexity of large-scale projects is very high since the level of complexity involved in any project has a positive correlation with itssize (Jolivet & Navarre, 1996). Besides possible technical difficulties, the complexity of megaprojects stems from a range ofambiguous and uncertain external and internal forces. Analyzing twomegaprojects in the Netherlands and Australia, Marrewijket al.(2008) observed that, in fact, the project teams managed both projects to the best of their abilities in the context of very complexoperations, paradoxes, uncertainties, influences, and ambiguities that surrounded these projects. Different project outcomes wererealized through different approaches that the project team pursued to manage complexity (Marrewijk et al., 2008).

Researchers have proposed different strategies to address the project complexity of large scale projects. For example, Jolivet andNavarre (1996) recommend a project management approach, referred to as self-organization and meta-rules, for a large-scaleproject that involves a large element of novelty or must be developed at an accelerated pace. Miller and Lessard (2001) proposemanagerial strategies and processes for risk management in large engineering projects. With multiple partners involved inmegaprojects, several researchers suggest the creation of public–private partnerships (PPPs) (Koppenjan, 2005; Marrewijk et al.,2008). Others propose the creation of appropriate project cultures. In particular, Clegg et al. (2002) suggest that the project teamshould design an alliance culture of inter-organizational collaboration. Marrewijk (2007) observed functional and dysfunctionalproject cultures throughout the life cycle of one megaproject and suggested that a project teammay need to create a distinct culturefor each phase of a megaproject. Several researchers discuss the governance issues in large public projects (Miller & Hobbs, 2005;Miller & Lessard, 2000). Recently Miller and Hobbs (2009) discuss the complexity of decision making with multiple partners andKlakegg, Williams, et al. (2009) propose a governance framework for public project development and estimation. Wirick (2009)discusses the challenges in public-sector project management. Among them is the shortage of good project managers in the publicsector. He, then, propose project management methods and tools that should be useful for public-sector project managers.

2.2. Performance of IS/IT projects

Several studies have been conducted to better understand the poor performance of IS/IT projects (Jiang, Klein, et al., 2009). Thereare the studies on, for example, the definition of failure, the critical failure factors, and approaches contributing to project success. Interms of the definition of failure, Lyytinen and Hirschheim (1987) have defined four categories of IS failures—Correspondence failure(when system design objectives are not met), Process failure (when an IS cannot be developed within an allocated budget, and/ortime schedule), Interaction failure (when the level of end-user usage of the information system is suggested as a surrogate) andExpectation failure (the inability of a system to meet its stakeholders' requirements, expectations, or values). Sauer (1993) proposedthat IS projects should be considered as failures only if there is development or operation termination.

23P. Patanakul / Journal of High Technology Management Research 25 (2014) 21–35

As for the factors leading to the failure of IS/IT projects and approaches contributing to project success, Yeo (2002) proposes alist of factors related to the project planning process, organizational context, and IS content. He suggested, for example, that weakdefinition of requirement and scope, inadequate project risk analysis, and unclear vision are the critical factors related to projectplanning; lack of user involvement, poor internal communication, and absence of an influential champion are the factors underthe organizational context category; and underestimation of the project scope and complexity, incomplete specification, andchanges in design specifications are the factors related to IS content. With the focus on requirement uncertainty, Jiang et al.(2009) argue that the perception differences among stakeholders (e.g. users and IS professionals) can be the cause of poorperformance. From a study of 99 retrospectives of IT projects, Nelson (2007) suggests a list of classic mistakes in IT projectmanagement. The top five mistakes are poor estimation and/or scheduling, ineffective stakeholder management, insufficient riskmanagement, insufficient planning, and shortchanged quality assurance. Escalation of commitment to a failing course of action isalso suggested as a cause of poor project performance (Keil, Mann, et al., 2000; Montealegre & Keil, 2000).

To successfully lead IS/IT projects, Yeo (2002) proposes a framework based on a basis of soft system thinking for the IS projectplanning and delivery processes. Nelson (2007) suggests some best practices that have been found across projects in his study. Theyare the use of agile development, establishment of a programmanagement office, comprehensive project charter, and joint applicationdevelopment, for example. Other researchers have stressed approaches including, e.g., risk management (de Bakker, Boonstra, et al.,2010; Iversen, Mathiassen, et al., 2004; Jiang & Klein, 1999), configuration management (Wateridge, 1999), project control (Kirsch,Sambamurthy, et al., 2002), and relationship management (Smyth & Edkins, 2007) for successful implementation of IS/IT projects.

3. Research method

The main objective of this exploratory research is to investigate the management of large-scale IS/IT projects in the public sectorin order to identify the common problems and causes leading to poor performance. Themain research questions are: 1) what are thecommon problems in managing the large-scale IS/IT projects in the public sector, and 2) what are the causes of these problems? Toanswer these questions, the researcher must study multiple projects with sufficient depth. With the availability of the governmentreports on the management of the large-scale IS/IT projects and the inability to conduct in-depth case study research with manyprojects, qualitative research based on content analysis was selected as a methodology. Since, the reports produced by the nationalauditing agency of the US, UK, and Australian government do exist and are available to public, the analysis of these reportsshould provide some insight regarding the problems or challenges facing by the project teams.

3.1. Sample

Based on the availability of the reports, fourteen projects were selected as a part of this study. Out of these fourteen projects,five projects are from the US; five projects are from the UK; and four projects are from Australia (see Table 1 and Appendix TablesA1–3 for more details). The five US projects had an estimated duration between 2 and 9 years and the initial budgets ranged from$60 million to $2.5 billion. The UK projects had durations of 3–11 years and initial budgets between £184 million and £1 billion.The Australian projects were smaller in size. The duration ranged from 3 to 7 years and the budgets were between $30 millionand $312 million Australian dollars. With the selection of projects from three different countries provided the researcher with theability to investigate whether or not the problems in managing the large-scale IS/IT project are common across these countries.

Table 1The fourteen projects in this study.

Project name Year ofimplementation

Country Owner

1 Customer Account Data Engine-Release 1: CADE 1999–2004 US Internal Revenue Service: IRS2 Trilogy 2001–2005 US Federal Bureau of Investigation: FBI3 Logistics Systems Modernization: LSM 1980–1994 US US Air Force4 ERP Pilot Projects: ERP 1998–2004 US US Navy5 Weather Systems Modernization: WSM 1980–1995 US National Weather Service: NWS6 National Offender Management Information Systems: NOMIS 2004–2011 UK National Offender Management Service7 LIBRA 1998–2007 UK Lord Chancellor's Department8 Benefits Payment Card: BPC 1996–2001 UK Department of Social Security and Post Office Counters Ltd.9 National Insurance Recording System – 2: NIRS-2 1995–1998 UK Contributions Agency, Inland Revenue Service10 National Probation Service Information System: NPSIS 1994–2001 UK Home Office's Probation Unit11 The Edge Project: EDGE 2000–2003 AUS Department of Family and Community Services12 Personnel Management Key Solution: PMKeyS 1997–2002 AUS Department of Defense13 Cargo Management Re-engineering: CMR 1998–2006 AUS Customs Service14 High Frequency Communication Systems Modernization: HFCSM 1997–2010 AUS Department of Defense, Defense material Organization

Table 2Common problems in managing large-scale IS/IT projects in the public-sector.

System design andimplementation

Project management and governance Contract management

Problems inrequirementidentification: Poorlydefined initialrequirements, unclearrequirements

Problems in systemintegration:Incompatiblesubsystems, setbackin system integrationand interoperability

Problems inmanaging projectrisks: Underestimation of risks,ineffectiveresponse plans

Problems in monitoring&control and managingchanges: Insufficienttimely review, nooversight of projectactivities

Problems in projectgovernance:Insufficient andincompletemanagement reviews,insufficient escalationto management

Problems in managingcontractualarrangements: Poorrelationship, contractualdisputes, poor contractorperformance

USA CADE X X X X XTrilogy X X X X XLSM X X X XERP X X X X XWSM X X X X X

UK NOMIS X X X XLibra X X X X X XBPC X X X X XNIRS-2 X X X XNPSIS X X X X

AUS EDGE X X X X X XPMKeyS X X X XCMR X X X X XHFCSM X X X X X

Total 11 8 11 14 12 11

24 P. Patanakul / Journal of High Technology Management Research 25 (2014) 21–35

3.2. Data sources

For the projects from the US, project information was gathered from audit reports generated by the GovernmentAccountability Office (www.gao.gov). Other independent reviews and reports were also used depending on the availability andthe need on a case-by-case basis. For the projects from the UK, the audit reports of the National Audit Office (www.nao.org.uk)were used for reference. Finally for the cases from Australia, information from the audit reports generated by the Australian

Table 3Problems and their potential causes found in this study.

Problems Potential causes

Problems in requirement identification: Poorly defined initial requirements, unclearrequirement left for developer's interpretation, etc.

- Lack of the understanding of system complexity- Difficulties in requirement elicitation due to:- Ineffective and inconsistent elicitation process- Contentious relationship between the owner and contractor- Lack of user involvement- Inadequate time to gather requirement

- Inability to refine elicited data into high-quality requirements due to:- Lack of effective requirement documentation and traceability

Problems in system integration: Incompatible subsystems, setback in systemintegration and interoperability

- Lack of understanding of system complexity- Deficiencies in system architecture due to:- Misperception of project as a collection of autonomous subsystems

- Lack of coordinated management oversightProblems in risk management: Underestimation of risks, unidentified risks, andineffective response plan

- Lack of risk management process- Poor execution of risk management process due to:- Limited understanding of system complexity

Problems in project monitoring & control and managing changes: In sufficienttimely review and limited oversight of project activities

- Lack of standard, procedure, and metrics for monitoring and control- Lack of change management process- Lack of experienced project managers- Unclear roles and responsibilities of project managers

Problems in project governance: Insufficient and incomplete management reviewsand insufficient escalation to management

- Failure to recognize project as a major initiative- Poor governance and report structure due to:- Lack of coordinated management oversight- Lack of Senior Responsibility Owner (SRO)

- In appropriate Internal Verification and Validation process- Ineffective Internal review process

Problems in managing contractual arrangement: Poor relationship, contractualdisputes, and poor contractor performance

- Insufficient bidding and tender evaluation processes- Misperception of contractors and suppliers by owner- Poor communication- Inappropriate managerial approach used- Lack of contractor oversight

25P. Patanakul / Journal of High Technology Management Research 25 (2014) 21–35

National Audit Office (www.anao.gov.au) was analyzed. Over 30 reports and related articles were reviewed. All of these reportsare available to the public.

3.3. Content analysis

Using reports from the government auditing agencies of the respective countries, content analysis was conducted (Bauer,2000) to identify the problems in managing large-scale IS/IT projects. For each project, multiple reports were reviewed. Tworesearchers started reading each report with a blank frame of mind. Once the researchers identified any problem from the texts ofthe report, they developed a code to represent each problem. In many cases, the researchers were able to identify the causes ofthese problems. They manually recorded and coded such relationships. Diagrams were also developed to illustrate the problemsand their causes, including their impact on the performance of the project if identified. The researchers performed such contentcoding in order to extract the most information from each report and also develop chains of evidence. Based on the list ofproblems that were identified for each project, the researchers were able to preliminarily categorize them into groups to betterunderstand their relevance and hierarchy. Since the researchers performed content analysis from the audit reports where theproblems and some of the causes, including their impact on project performance were already identified by the auditors, thecontent analysis process involved little subjectivity. The findings from two researchers were consistent.

After analysis within each case, a cross-case analysis was performed to identify common problems and their causes. Tableswere also created to record such information and diagrams were developed to represent such relationships. In addition, contentgrouping was performed based on these common problems. This grouping step was done in order to present the findings in ameaningful way. Based on this bottom-up approach, three major categories of problems were created. They are systems designand implementation, project management and governance, and contract management. To ensure the research validity, thefindings were compared and contrasted with the literature on large-scale projects and IS/IT projects.

4. Research findings

Based on the data analysis, three major categories of problemswere identified across the cases (see Table 2). The system design &implementation category includes problems related to requirement identification and system integration. The project managementcategory includes problems related to project risk management, project monitoring & control and change management, and projectgovernance. Problems inmanaging contractual arrangements were identified in the contractmanagement category. Common causesof these problems were also identified, see Table 3. Note that the problems and their causes are presented in this section. Thediscussion of the findings and their implications is presented in the subsequent section.

4.1. System design and implementation

The large-scale projects in this study mostly involved the development of systems to support the functioning of governmentagencies. The development process of most projects was plagued by technical issues ranging from requirement management tosystem development and integration. The development of systems architecture at the systems level and enterprise architecture atan organizational level was also one of the major issues in these projects.

4.1.1. Problems in requirement identificationRequirement identification was a problem in eleven out of fourteen cases. Requirement identification is a process of eliciting,

documenting, analyzing, prioritizing and agreeing on requirements and then controlling change and communicating it to relevantstakeholders. Being among the earliest project steps, poorly defined requirements can have a cascading negative effect on projectperformance throughout the project life cycle. In fact, identifying user requirements for a typical IS/IT project is a challenge in itself, letalone when one considers the size and the complexity of a project. Without clearly defined and documented requirements, there is ahigh possibility of new requirements being added to the project and existing requirements being discarded. It was found in this studythat lacking clear definition of requirements negatively impacts the project performance. In the Trilogy case, some of the requirementswere left to a trial and error process. As the software evolved in a spiral approach, there were numerous requirement additions andchanges, as well as placeswhere the requirements were left open for future specification or for developer interpretation. Therewas clearevidence in the IRS's Customer Data Account Engine (CADE) project that poorly defined initial requirements and deferring requirementsled to cost over-runs and schedule delays (Anonymous, 2004a,b; GAO, 2001). A similar pattern was seen in the Libra project, whichinvolved building new IT systems for the Magistrates' Courts (NAO, 2003) and the Cargo Management re-engineering (CMR) project(NAO, 2001). In some other cases, poorly defined initial requirements led to the termination of the projects as was seen in the case of theFBI Trilogy project (Anonymous, 2004a,b, 2005; OIG, 2002, 2003, 2005), and the Navy ERP pilots (GAO, 2005a,b). Poor definition of therequirements either led to infeasible technical solutions or to the development of systems that were not serviceable.

Why is obtaining clearly defined requirements challenging? Several reasons were found. First, the complexity of the system was notfully understood or underestimated such that the project teams were not able to fully develop quality requirement (e.g. the CMR case(ANAO, 2007a)). The second reason was the difficulty in requirement elicitation. In some cases (the LSM project (GAO, 1989a,b,c), forexample), the difficulty came from the process that the project teams used to gather requirements was ineffective and inconsistent. Inthe CADE project, the contentious relationship between the owner and the prime contractor hindered communication, which maderequirement elicitation difficult (GAO, 2004). Lack of user involvementwas found in the National Probation Services Information System

26 P. Patanakul / Journal of High Technology Management Research 25 (2014) 21–35

(NPSIS) case. In the Benefit Payment Card (BPC) case, inadequate time was devoted to requirement gathering and piloting (NAO, 2000,2002a). Third, although efforts were made to elicit requirements, the challenge in refining the data into high-quality requirements alsoexists. This means that the data was not always refined into high-quality requirements—some areas were left open, while others wereover specified. In addition, lack of effective requirement documentation and requirement traceability such as in the CADE project (GAO,2001) lead to an inability of the project team to describe and follow the life of a requirement through all periods of on-going refinementand iteration in both forward and backward direction. Table 3 summarizes problems and their causes found in this study.

4.1.2. Problems in system integrationSystem integration was another challenge in managing the IS/IT projects. Development issues related to system integration

are generally due to the multiple platforms on which systems within the same organization operate. Integration issues oftenresult in degraded performance and cost overruns as seen in the CADE, ERP, WSM, Libra, NIRS-2, and HFCSM projects (ANAO,2007a; GAO, 1994, 2001, 2005a,b; NAO, 1998, 2003).

Why is having systemic integration challenging? Several reasons were found. First, many problems in system integration arise dueto the lack of understanding of system complexity. This can in turn be attributed to a lack of clearly defined requirements as discussedearlier. Such issues were evident in cases such as CADE, WSM, LSM, NOMIS, BPC, NIRS-2, CMR, EDGE, and HFCSM (ANAO, 2005, 2007a,b; GAO, 1989a,b,c, 1994, 2001; NAO, 2000, 2002a,b, 2009). Second, deficiencies in the system architecture were found as another causeleading to difficulties in system integration. Typically, a system architecture or systems architecture is the conceptual design thatdefines the structure and/or behavior of a system. It is the fundamental organization of a system, embodied in its components, theirrelationships to each other and the environment, and the principles governing its design and evolution. Lack of system architecturecan have an adverse effect on system development as was seen in cases such as CADE, WSM, and ERP (GAO, 1994; GAO, 2001; GAO,2005a). In the WSM case the team did not develop system architecture. Modernization subsystems were developed independently. Amajor reason for this was that the project was perceived by management as a collection of autonomous subsystem. As a result, thesubsystems were incompatible and required additional resources to interconnect them. The third reason for integration challenges isthe lack of coordinated management oversight. This problem is very obvious in the ERP case (GAO, 2005a), where four separate Navyorganizations began their ERP Pilot programs independent of each other, at different times, and with separate funding. Even thougheach pilot implemented the same ERP software and each pilot was small in scale relative to the entire program, software configurationwas done differently, leading to serious problems of integration and interoperability.

4.2. Project management and governance

Besides problems in system design & implementation, somemajor problems related to project management were found in mostcases. These problems were associated with the lack of standard project management processes such as risk management, projectmonitoring and control, change management, and project governance. The phenomenon of systems being delivered behindschedule, costing far more to produce than original budget estimates and failing to meet user requirements can be attributed notonly to the non-application of principles and methods, but also to inadequate project management (Ratcliff, 1987).

4.2.1. Problems in managing project risksLack of effective risk management was evident in eleven cases. They are ERP, LSM, NOMIS, Libra, BPC, NIRS-2, NPSIS, PMKeyS,

EDGE, CMR, and HFCSM (ANAO, 2006; ANAO, 2007a,b; GAO, 1989a,b,c, 2005a; NAO, 1999, 2000, 2001, 2003, 2009).With the scale andcomplexity of megaprojects, effectively managing risk is among the important activities leading to the project success (Miller &Lessard, 2001).

Why is risk management challenging? The common issues that were found across these cases were the lack of an effective riskmanagement process and effective execution of the process. In the ERP project (GAO, 2005a), risks were not effectively managed in thesense that disciplined processes were not implemented, increasing the susceptibility of the system to risks, which directly or indirectlyaffected the schedule and cost objectives of the project. Lack of a sound riskmanagement process was also found in other cases such asLSM, NOMIS, BPC, CMR, and HFCSM. In the BPC project (NAO, 2002a), the initial business case did not adequately assess risks and theirconsequences.Without risk assessment, soundmonitoring and control practices could not be implemented. In the NPSIS project (NAO,2002b), an underestimation of the risks associated with transferring an existing system onto the NPSIS network led to inaccurateestimation of cost for implementation and maintenance of the effort. The EDGE project (ANAO, 2005) was a risky effort as multiplesystems were developed concurrently. Although the team developed mitigation strategies to address project risks, these strategieswere ineffective. It is found that the complexity of the system impacts the team's ability to effectively execute the process.

4.2.2. Problems in project monitoring & control and managing changesOne of the underlying problems of project management in the cases was insufficient project monitoring and control, including

managing changes. In many cases, lack of project monitoring and control led to missed opportunities in identifying problemsearly. It was found from the cases that the monitoring and control of changes in requirements are especially important in themanagement of the large-scale projects. Many of the projects did not effectively monitor and control those changes.

Why is project monitoring and control challenging? One of the reasons is the lack of standard processes, procedures, and metrics forproject monitoring and control. In some cases, projects were set up without significant milestones, checkpoints, incentives, or otherways of assessing and rewarding project progress. For example, in the PMKeyS (ANAO, 2006) project, no formal project managementmethodologies were used to implement and monitor the project. The Project Office, as part of its planning processes, did not develop,

27P. Patanakul / Journal of High Technology Management Research 25 (2014) 21–35

complete or update a number of the key project planning and management documents. As a result, the project lacked timely reviewsof milestones and validation of objectives. There was a lack of metrics for monitoring performance in key areas. Second, themajority ofprojects in this study experienced changes in requirements without having a good process to monitor and control those changes. In thecase of the HFCSM project (ANAO, 2007b), the contractor pointed out that delays on the part of the owner in finalizing contractchange proposals and engineering change proposals resulted in schedule slippages.

Third, in the majority of the projects, lack of experienced project managers and lack of clear roles and responsibilities for projectmanagers caused problems in project monitoring and control. Without experienced project managers and their clear roles andresponsibilities, the projects suffered from lack of oversight of project activities leading to development of deficient systems, costand schedule escalations, etc. In the Trilogy case (Anonymous, 2005), inexperienced IT programmanagers and contract managersmade it difficult for the upper management to deal aggressively or effectively with its contractors. Inexperienced managers oftenlack the ability to assume proactive management roles and are often held hostage to the perspectives of the contractor. In theWSM case (GAO, 1994), the lack of a single/central project manager for individual systems led to a lack of coordination andintegration. With multiple parties involved in the projects, defining clear roles and responsibilities for project managers can bedifficult, especially when communication between parties is ineffective.

4.2.3. Problems in project governanceIn managing megaprojects, it is sensible to anticipate project governance as a major challenge. According to Miller and Hobbs

(2005), when undertaking a large project without an adequate governance regime, most organizations are exposed to a highprobability of failure since megaprojects are qualitatively more complex and riskier, and therefore require governance regimes thatare different from those of more routine and less risky endeavors. Twelve out of the fourteen cases faced a major problem in projectgovernance. In general, governance can be defined as the decision accountability structure to leverage the behaviors and empowerteams to drive value. Governance relates to consistent management, cohesive policies, processes and decision-rights for a given areaof responsibility. Governance is an important factor in determining project outcomes as timely and effective management decisionsare critical for project success. Many of the cases showed problems at the management level as well as lack of oversight andreviews resulted in cost overruns, schedule delays and missed opportunities to identify problems early.

Why is project governance challenging? Several reasons were found in the cases. The first reason is the failure of managementto recognize the project as a major initiative that requires their close attention. In the NOMIS case (NAO, 2009), for example,management treated NOMIS as an IT project rather than a major IT-enabled business change program. For this reason, themanagement review board did not ask for or receive any other more specific reports on project progress, although the board metat least once every two months—nor did it actively monitor delivery of the project. A similar issue was found in the PMKeySproject where we found a failure on the management's part to manage the project as a strategic procurement activity or a majorproject (ANAO, 2006). Thus, there was a lack of monitoring, control, and reviews that should be a part of major capital initiatives.

Second, it was found inmany cases that the problems in project governance are caused by a poor governance structure. In particular,some of the projects in this study lacked coordinated management oversight and did not have a senior individual who had totalresponsibility for the project, referred to as the Senior Responsible Owner (SRO). When multiple partners/agencies are involved in aproject, the lack of SRO can lead to conflicts between partners and, in some cases, lack of involvement from any of the partners. Thiscan, in turn, result in a poor governance structure affecting a variety of other project activities along with costs and schedule. In theWSM case (GAO, 1994), there was no individual who had a total responsibility for the modernization effort. Management viewed themodernization project as a collection of autonomous subsystems. Individual subsystem changes were managed through changecontrol groups that brought together representatives from multiple subsystems. However, this process, which focuses on individualsubsystem changes, was not an effective alternative to having a system architecture, a chief engineer or a single manager, thus limitingthe ability to enforce a coherent architecture. Lack of coordinated management oversight was also found in the ERP project (GAO,2005a). The project was funded by different major commands within the Navy. This allowed the project to be developedindependently but it caused stove-piped systems that could not operate with each other. In some cases such as NOMIS, although theproject had an SRO, the SRO had little experience in major project delivery. With other full-time roles, the SRO had insufficient time toundertake the SRO responsibilities. In the CMR and EDGE cases, lack of an SRO cascaded into a number of problems, for example, lackof clarity on roles and responsibilities, unclear project objectives, and weak business cases (ANAO, 2005; ANAO, 2007a).

The third reason for problems in project governance stems from inappropriate internal verification & validation (IV&V), andineffective internal reviews. In many cases, such reviews were not completed. Thus any shortcomings in the project were notescalated to upper management for immediate attention. In the ERP case (GAO, 2005a), there was no established independentverification and validation function to provide an assessment of the ERP project to the Department of Defense and themanagement of the Navy. In addition to the lack of IV&V, there was a lack of reporting structure to capture quantitative data thatcould be used to assess the effectiveness of the overall effort. Similar patterns were found in the LSM, EDGE and PMKeyS projects(ANAO, 2005, 2006; GAO, 1989a,b,c). On the other hand, there is a limit to the amount of IV&V that should be used as too manyaudits and reviews during the course of the projects can hamper the progress of the project and the performance of the projectpersonnel, as was found in the Trilogy and CADE cases (Anonymous, 2004a,; GAO, 2001).

4.3. Contract management

Effectively managed contracts can deliver significant benefits to an organization. Often, the effective management of contractsseparates successful projects from unsuccessful ones. Therefore, to ensure that a project's implementation is successful, it is absolutely

28 P. Patanakul / Journal of High Technology Management Research 25 (2014) 21–35

necessary to manage contracts effectively from the bidding process through the operational support after a project's completion.Withmultiple stakeholders involve in large-scale projects, managing contractual relationships becomes highly challenging.

4.3.1. Problems in managing contractual arrangementsContract management issues surfaced in eleven out of the fourteen cases. Since the performance of the contractors directly links

to the performance of a project, it is imperative that the project teammanages contractual relationships very well. Poor performanceby the contractor, resulting in missed milestones, delayed payments, and contract modifications, often leads to a contentiousrelationship between the contractor and the owner, affecting future performance both of the contractor and the project.

Why is managing contractual arrangements challenging? Several reasons were found from the cases. First, several contractualproblems stem from an inefficient bidding and tender evaluation process. Since the bidding process is critical to a project as itsoutcome decides the contractor(s) whowill be responsible for the deliverables of the project, it is very important for an organizationto have a competitive bidding process for the projects with clearly defined and detailed requirements. In some cases, the lack oftechnical expertise of the contractor and a lack of experience inmanaging similar projects have damaging effects not only on the costand schedule of the project but also on the relationship between the contractor and the owner. In the WSM project (GAO, 2000a,b),the contractor lacked experience or technical expertise in developing weather systems that were as technologically advanced asneeded by the National Weather Service. In Libra project (NAO, 2003), management did not predetermine the bidders that would beinterested in the project through a market survey. This resulted in only one contractor bidding for the project with resultantdisastrous consequences as the contractor turned out to be neither competitive nor technically competent for a project of this scale.Without proper competition, it is difficult to demonstrate that any offer from a single bidder represents good value for the taxpayer'smoney. In addition, an inefficient tender evaluation process leads to a contract that lacks detail and leads to issues andinconsistencies during the implementation of the project. In the PMKeyS project (ANAO, 2006), there was a lack of adherence to theacceptance criteria and limited evidence from the Certificates of Acceptance indicating that the acceptance criteria had beencompromised. In the CMR project, Customs seemed to lack documentation outlining the method of procurement or approval for theexpenditure of public money in several instances. Without these core documents, there is no way to determine compliance toprocurement guidelines or the financial framework. Other best practices such as well defined performance metrics and contractprices and payment schedules were not defined in advance, leading to subpar deliverables.

The second reason that leads to problems inmanaging contractual arrangements is the lack of management skills on the part of theowner to manage the contractual arrangement. This includes the perception of the owner of the contractors, the managerial approachused, communication between stakeholders, and contractor oversight. In the NOMIS project (NAO, 2009), although managementexecuted a Time and Materials contract that required strong interpersonal relationships between the contractor and the owner foreffective implementation, the owner (National Offender Management Service: NOMS) perceived the contractor (EDS) as just asupplier of skilled labor rather than a strategic partner. There were negative tensions and strong polarizations between NOMS andEDS aggravated by confusing management structures and the poor performance of the EDS staff. Similarly, missed milestones andthreats to exercise contract termination by the owner led to deteriorating relationships between the Prime contractor and the owner(Defense Material Organization: DMO) in the case of the HFCSM project (ANAO, 2007b) although the contractor pointed out thatdelays on the part of the DMO in finalizing contract change proposals and engineering change proposals resulted in scheduleslippages. Insufficient communication between stakeholders, especially between the owner and the contractors was anotherchallenge in managing relationships. In fact, insufficient communication between the stakeholders, i.e., the owner and the contractorin most cases, is the leading cause of contractual disputes. In addition, the owner must have the ability to oversee the contractualactivities not only to ensure that the contractor's performance is on par with expectations but also tomake sure that the contractor isadhering to the terms of the contract. Moreover, poor contractor oversight may easily lead to communication gaps between theowner and the contractor, over optimistic expectations of the owner and the contractor losing direction of the project. In the Trilogycase (GAO, 2006b), the FBI lacked an effective review and approval process for contractor invoices, strong internal or physicalcontrols over procurement activities, a written policy to govern its tracking and oversight activities and a contractormonitoring plan.This left the FBI vulnerable to questionable contractor costs and missing equipment.

5. Discussion and implications

5.1. What can we learn from these findings?

The findings indicate six common problems in managing mega IS/IT projects in public sectors. These problems are in thecategories related to system design & implementation, project management and governance, and contract management. Inconcert with the literature on large-scale projects and IS/IT projects, the causes of IS/IT project failures as presented in theliterature are also to the causes of poor performance of the large-scale IS/IT projects in this study. This seems to imply thatregardless of the scope and scale of the IS/IT projects, the common causes of poor project performance persist. They are, forexample, the underestimation of project complexity, lack of user involvement, poor communication, absence of experiencedproject manager, insufficient management controls, insufficient risk management, ineffective stakeholder management, lack ofclearly defined requirement, and changes in design specifications (Jiang & Klein, 1999; Jiang et al., 2009; Keil et al., 2000; Nelson,2007; Yeo, 2002). Even though the large-scale projects in this study share the common causes of poor performance with IS/ITprojects in general, the level of impact of these causes on the project performance increases as the scale of the projects increases.This may be because of the large number of interactions and relationships among various stakeholders. In addition, since the

29P. Patanakul / Journal of High Technology Management Research 25 (2014) 21–35

projects in this study are from the public sector, these IS/IT projects also share common challenges with typical large scalepublic-sector projects. They are the challenges regarding system complexity, governance, project management competency, andcontract management. It can then be concluded that the combination of the unique characteristics of an IS/IT project and thecharacteristics of large scale public-sector projects makes the management of large-scale IS/IT projects extremely challenging.

The common problems found in this study seem to point out to the areas that have been well researched and well specified inpractice. Methodologies and approaches have been developed and suggested to overcome such problems. Why do these problemsstill exist? This is a challenging question that requires additional analysis beyond what was done in this study. However, thefindings from this study may provide some initial insights that address the aforementioned question. When investigating theseproblems and their causes in depth, the research evidence indicates some common themes of the causes underlining theseproblems, including the interactions among them. They are 1) the use of processes, 2) the capability in managing large-scale projects,and 3) understanding and recognition of system complexity.

Lack of, ineffective use of, and inconsistent use of the appropriate processes are the causes of many problems identified in thisstudy. This includes the ineffective and inconsistent use of requirement elicitation process, leading to problems in requirementidentification; lack of risk management process, process for monitoring and control, change management process, and internalverification and validation and internal review processes, leading to problems in project management and governance; andinsufficient biding and tender evaluation process, leading to problems in contract management. This finding is rather surprisingsince these processes exist in practice and have been developed and extensively studied by many researchers. Why do theseissues still exist? Several reasons can be drawn from the research evidence. First, even though these processes exist, practitionersin some cases were not aware of such processes and therefore did not use them. This issue indicates the problems in knowledgeacquisition and dissemination. Second, the processes have been used; however, the practitioners in some cases were not able touse them effectively and consistently. This issue may stem from the fact that some practitioners lack general management andproject management capability. These two reasons indicate that the problems do not stem from the lack of the processes. Theycome from the lack of the awareness and the proficient use of the processes. However, with the increase in the level of systemcomplexity, it may be a time for researchers and practitioners to revisit, discuss, and improve the aforementioned processes toensure their applicability to the management of large-scale public IS/IT projects.

Besides the use of the process, the capability (both managerial and technical) of practitioners in managing large-scale projects isanother common theme underlying the problems found in this study. The research evidences also indicate that the lack ofmanagerial and technical capability is among the causes of the ineffective and inconsistent use of the process. The managerialcapability includes the capability in the areas of owner–contractor relationship management, management oversight, projectmanagement, communication, for examples. The technical capability includes the capabilities necessary for addressing the technicalaspects of the project such as the ability to elicit and refine requirements, the development of system architecture, and the ability todevelop appropriate risk response plans. Perhaps, the existence of these issues despite the extensive research has been conductedare that every project is unique and these projects have high level of complexity. Given that the knowledge and lessons learned exist,more focus should be put toward the dissemination and adaptation of the knowledge to the government personnel. This, however,cannot rule out the fact that new knowledge and managerial approaches may be needed in managing large-scale IS/IT projects.

Overall system complexity was found as another underlying cause that makes the management of large-scale IS/IT projectschallenging. Sources of complexity come not only from the technical aspect of the system but also from the large numbers ofinteractions and relationships among various stakeholders. The under estimation or misunderstanding of system complexity canlead to several problems along the project life cycle that can eventually lead to poor project performance. Among those are theproblems in system design & implementation and project management & governance. Management misperception of the projectand its complexity is also a common cause of several problems. If management perceives a large-scale project as a collection ofautonomous subsystems instead of a major initiative, the system integration and project governance can suffer. To successfullymanage such a large scale project, a concept of system of systems may need to be applied to cope with project complexity(Keating, Rogers, et al., 2003). An interdisciplinary engineering andmanagement body of knowledge using the system approach isneeded for the successful management of such large-scale IS/IT projects (Mora et al., 2008; Wirick, 2009).

It is important to note that while this research focuses on internal issues related to the management of large-scale IS/ITprojects, there might be other internal and external factors that impact the performance of the projects that were not identified.Factors such as organizational culture, organizational processes, policies, politics, and structure may contribute to the systemcomplexity and post challenges to project management. In addition, external factors such as national policies and politics mayalso hinder the successful implementation of large-scale projects especially in the public sector.

5.2. What can practitioners do to increase the level of project success?

Based on the problems found in this study and the discussions above, this section summarizes the practices that may contributeto the success of large-scale projects. They are presented under the categories of system design and implementation, projectmanagement and governance, and contract management. These implications were supported by research evidences and literature.

5.2.1. System design and implementationThe first step to a successful IS/IT project is to understand the requirements and to effectively manage them. The information from

this study consistently suggests that without clear requirements, the system being developed will have limited benefits. Whilemarginal improvement may be realized, it is typically difficult to ascertain the sustained, long-term benefits. Practitioners should also

30 P. Patanakul / Journal of High Technology Management Research 25 (2014) 21–35

be aware that the requirement uncertainty and instability could also intensify interpersonal conflicts among project stakeholders,which can then magnify requirement diversity, and become a major source of project failure (Jiang et al., 2009; Liu, Chen, et al., 2010).Since a large-scale IS/IT project is very complex in nature, practitioners must follow disciplined processes to identify and managerequirements. Requirement identification must be done with the involvement of project stakeholders (Chatzoglou & Macaulay, 1997)and appropriate techniques should be used with regards to different stages of the identification process and the users' cognitive abilityand behaviors (Browne & Ramesh, 2002; Chiu, 2005). Besides requirement identification, a process of documenting the requirementsmust be put in place and requirement traceability must be practiced. It is possible that requirement identification is an on-goingcontinuous process. In such cases, management must be certain that a change management process is established.

As already stated, it is inevitable that large-scale IS/IT projects will be very complex. To address this issue, practitioners have tofirst understand the project complexity and determine whether or not it is too ambitious to undertake the large-scale project atonce. Is it possible to break up the project into manageable pieces that can be delivered incrementally?Whether or not the projectis pursued at once or incrementally, management must ensure the existence of system architecture. The evidence from the casessuggests that a lack of system architecture leads to problems in system integration, and eventually poor project performance. It istherefore important that system architecture is in place for any large-scale IS/IT project (Philbin, 2008).

It is imperative that the formulation of architecture must have the highest priority in the IT efforts. To do so, upper managementmust make sure that the creation and communication of a complete architecture is a top priority. Although the formulation ofsystem architecture can be done by the project team, managementmust be personally involved in this process and participate in keydecisions. It is critical that management is well versed in and is comfortable with the operational aspects of the architecture and itsoverall linkage to the high-level system design. Management involvement in the formulation of system architecture will improvetop-level decision making of the system being developed and will ensure that IT investments realize their full potential. Even thoughit is possible that a contractor might assist the system owner in developing the system architecture, the contractor should not besolely responsible for this effort. The contractor will not fully understand the operational issues that must be reflected in thearchitecture. In addition to the development of system architecture, management must ensure the validity of the architecture.Independent and regular reviews of the system architecture by external experts may help.

5.2.2. Project management and governanceManagement and oversight of large-scale projects by experienced project managers are essential for ensuring project success.

The successful implementation of IS/IT systems also calls for sound project management processes and methodologies. As alreadyindicated, processes for project risk management, changemanagement, and monitoring and control are necessary. Although it is notindicated explicitly, project management processes related to project planning are also important (Nelson, 2007). The priordiscussions on requirement management and system architecture already addressed some of these project planning processes.

In managing large-scale projects, it is important that a formal and standard risk management process should be established(Aloini, Dulmin, et al., 2007; Miller & Lessard, 2001; Patanakul, 2008). The process should include risk identification, riskassessment (qualification and quantification), risk handling (response planning), risk surveillance (monitoring and control), andrisk closure. Since a large-scale project consists of many programs and projects, risk management should be conducted onmultiple levels. For example, the implementation of a team-level board encourages the management of risks at the lowestpossible level. Risks that require management attention may be escalated to a higher-level board, such as a program level board.Literature suggests that the formation of public–private partnerships can also help manage risk in project delivery (Abednego &Ogunlana, 2006; Shen, Platten, et al., 2006). Besides standard risk management processes, a formal and standard changemanagement process should also be established. Because of the complexity of a large-scale IS/IT project, it is likely that manychanges will occur as the project progresses. The information from the cases indicates that requirement changes can havenegative impacts on project performance and changes occurring in the later phases of the project can be detrimental to theproject objectives. Change decisions must be implemented with careful consideration. A formal change management process, tiedto the change management board discussed earlier, can facilitate the decision to implement changes and may help assuremanagement that all the changes implemented in the project are beneficial to the overall project objectives.

The success of large-scale projects is also associated with project monitoring and control. Earned Value Management (EVM) is aproven concept for project monitoring that integrates scope, schedule, and cost measures for gauging the project progress. EVMshould be properly applied as a technique for monitoring the progress of large-scale projects. The project management competencypossessed by an organization is also significant to the success of large-scale projects. As indicated by the cases, many organizationsthat undertook large-scale projects without having necessary resources or competency related to project management ended upwith poor project performance. Depending on how project work is allocated, the majority of the work could be done by thecontractors. However, the project owner must assign key personnel from its staff to the project and these personnel should workhand-in-hand with the contractors. A project manager is among the key personnel that represents the owner. The project managerassigned to the project must be experienced and work closely with the contractor's project manager. The project manager must beproactive and be on top of the project issues such that s/he can make project decisions or escalate the issues when needed.

Governance is also an important issue in managing a mega IS/IT project. Based on evidence in this research, the lack of managementcontrol and oversight resulted in poor performance of most of the projects. It is obvious that appropriate management controls andprocesses must be in place to have reasonable assurance that an enterprise effort will be successful. As it is a high-level organizationalactivity, an organization embarking on large-scale projects must insure that it has established procedures and policies specifically forproject governance (Miller & Lessard, 2000; Williams, Klakegg, et al., 2010). Many cases in this study indicate that the lack of anexperienced senior responsible owner (SRO) leads to poor project governance. In implementing a large-scale project, the owner's

31P. Patanakul / Journal of High Technology Management Research 25 (2014) 21–35

management must assign an SRO for the project. Management must also make sure that the authority of the SRO should becommensurate with his or her roles and responsibilities. It is important that the SRO is committed to the project. Research shows thatthe capabilities of the SRO have an important impact on the dynamics of a project and its performance (Miller & Hobbs, 2005;Sutterfield, Friday-Stroud, et al., 2006). The SRO should have an integrative business perspective, ability to evaluate complex systemsfrom multiple perspectives, relational and coalition-building competencies, and political and negotiation skills (Miller & Hobbs, 2005).The SROmay establish a project office to provide support to the project teams and become a formal channel to relay project informationto the SRO and vice versa (Nelson, 2007).

To ensure the effective governance of large-scale projects, besides assigning an SRO for the projects, management can establishan enterprise project management office to oversee all high priority projects. The office can set up standards and ensure thatmanagement of large-scale IS/IT projects follows the standards. These standards may range from financial controls to governanceand senior management oversight. In addition, management must establish an independent verification and validation (IV&V)function. Formal procedures and policies for IV&V and internal reviewsmust be established. In general, management can establishreview boards to carry out the review activities. For example, management can establish: (1) an IT investment/project reviewboard to ensure that the investments in mega IS/IT projects are beneficial; (2) a technical review board to assess the technicalfeasibilities of the project; (3) a change management board can also be established to review the changes associated with theproject; (4) an IT policy review board to review all IT related policies of an organization, and (5) an enterprise architecture reviewboard to ensure the alignment of IT capabilities with the organization's strategic goals (GAO, 2004).

5.2.3. Contract managementContract management is a significant issue in managing large-scale projects since the majority of the project work is

contracted out. A rigorous selection process must ensure that the awarded contractor is capable of delivering the expectedoutcomes. Being engaged in the project and keeping open communication lines with the contractor is essential for the owner tofacilitate a good relationship with the contractor.

For contract selection, it is very important that the owner organization has a competitive bidding process. The project ownermust have a set of criteria for contract selection. It is important that the owner chooses the type of contract that best suits theneeds of the organization. Different types of contract suit different types of organizations and pose different levels of risks to theowner and the contractor (Turner & Simister, 2001). The financial standing, the risk-taking capacity of the organization, and themanagement controls and oversight ability are some of the factors to be considered when deciding on the types of contracts. Forexample, a cost plus award fee contract needs tighter management controls and oversight than a fixed price contract. Similarlymanagement bears all the risk in a Time and Materials contract whereas in a Private Finance Initiative, which was found in someof the UK projects we studied, all the risk is transferred to the contractor (Smyth & Edkins, 2007).

Forming partnerships with the contractors seems to be an effective approach for contract management. Public–private partnership isconsidered to be a form of structured cooperation between public and private parties in which they share or reallocate risks, costs,benefits, resources, and responsibilities (Koppenjan, 2005). The partnership creates a sense of involvement and ownership both for theproject owner and the contractors. The partnership should also facilitate the oversight of the project by the owner. Without thepartnership, negative tensions and strong polarization could occur. Such tensions and polarization could jeopardize the projectperformance as we have learned from some of the cases (NAO, 2001). Creating an alliance culture is also an approach to promote inter-organizational collaborations (Clegg et al., 2002). To do so, management must build a collaborative commitment, create transparencyand seek to constitute each stakeholder in such a way that they have something to gain from greater collaboration within the project.Alliance contracts seem to work best where there is uncertainty in the product and process, the owner has some knowledge of thesituation and looks for a breakthrough performance, and the alliance partners have the competence and financial backing to share therisk (Turner & Simister, 2001).

6. Limitations and contributions

Potential limitations of this study center on its research methodology. First, the sample size is small since only 14 cases werestudied. Generalization of the research findings must be done with caution. Second, content analysis was conducted based on theaudit reports that were available. Instead of analyzing raw information as in the case study research methodology, the informationthat was analyzed had already been processed. However, the information was from the audit reports generated by governmentagencies. These sources of information should be considered reliable since these audit reports tend to provide factual information.The author is aware of the problem that the information in the report is based on the interpretation of the auditors. However, theauditors were believed to be capable and could havewell interest of the situation. Third, although the information from the reports isfactual, there is a possibility that emphasis had been selectively set forth on certain areas depending on the focus of the audit agencyat the time of auditing. This may lead to the appearance or disappearance of certain issues in the report. For example, the GAO mayput more emphasis on enterprise architecture/system architecture, whereas NAO or ANAO may focus on the issues of contractmanagement. IV&V is not an issue in any projects from the UK that we studied. This can either be due to effective implementation ofthese processes or the focus of the auditing agency at that time. For a future study, researchers may consider implementing a casestudy methodology in order to study each case in-depth. However, to be able to study fourteen projects from three countries, suchresearch would require a tremendous amount of time and resources. Fourth, this study focused on projects from the US, UK,and Australia. As a result, the findings may not include other issues or problems that practitioners face when managing large-scale

32 P. Patanakul / Journal of High Technology Management Research 25 (2014) 21–35

projects in other countries. This includes the roles that corruption and cronyism play during, e.g., bidding for government contract,sub-contractor selection, filing for permits, execution of the project work, or closing.2 These topics warrant future studies.

Despite its limitations, this study provides significant contribution to the literature. First, this research investigated themanagement of large-scale IS/IT projects, the area that has limited research. It shed light on the major problems and their causes. Eventhough some of the research findings are not new to researchers and practitioners, they may help confirm that regardless of the scopeand scale of the IS/IT projects, the common causes of poor project performance persist. However, the level of impact of these causes onthe problems and the impact of the problems on the project performance increase as the scale of the projects increases.

Second, by recognizing the problems and their causes found in this study, a more focus study can be conducted. Such a studycan focus on one specific problem to investigate what the real issue is and what can be done to advance the practices of projectmanagement. The findings may suggest the needs for new approaches or the adaptation of the current approaches that can helppractitioners address a particular problem in management of large-scale IS/IT projects. As discussed earlier, such an approachmayneed to integrate the concepts of system approach and system of systems such that they are applicable to the management oflarge-scale IS/IT projects (Mora et al., 2008; Wirick, 2009).

Third, as the information in Table 3 illustrates the relationships between the problems and their causes drawn from theresearch evidence, they can be used as a basis for future empirical studies. In other words, research hypotheses for a large-samplestudy can be developed based on the relationships presented in the table. Future studies can investigate the direct relationshipsamong those variables or the interactions among them. For example, studies can be conducted to investigate the impact of theinteraction between the ineffective use of a process and lack of coordinated management oversight on the performance of thelarge-scale IS/IT projects. In addition, research opportunity also exists to examine the impact of external factors, such as change inpolicies and procedure, on those relationships.

Appendix A

2 The author thanks the reviewer for pointing this out.

Table A1Descriptions of US projects.

Project name(year start–finish)

Description Duration(original/actual)

Budget(original/actual)

Country(owner)

Customer Account DataEngine-Release 1:CADE(1999–2004)

The goal of this project was to replace the IRS's 30 year old outdated andinefficient data management system that relied on outdated storagetechnology (magnetic tapes) with a single, authoritative corporate datasource for enabling future customer service and financial managementapplications (Anonymous, 2004b; GAO, 2001, 2004, 2006a).

3 yrs/5 yrs $60 M/$98 M US (InternalRevenueService: IRS)

Trilogy (2001–2005) The goal of this project was to replace FBI's existing investigativeapplication —Automated Case Support (ACS) system with a VirtualCase File (VCF) system to enhance the ability of agents to organizeaccess and analyze information, modernize IT infrastructure byproviding a high-speed network linking the offices of the FBI, modernworkstations and software within each office for every FBI employee(Anonymous, 2005; GAO, 2005b, 2006b; OIG, 2002, 2003, 2005).

2 yrs/4 yrs $379.8M/$581.1 MVCF terminated.

US (FederalBureau ofInvestigation:FBI)

: VCF terminated

Logistics SystemsModernization:LSM (1980–1994)

This was a project to modernize the Air Force's Logistics Systems(GAO, 1989a,b,c, 1992). Three out of ten projects were investigated.

9 yrs/14 yrs $359.4M/$564 M

US(US Air Force)

- Requirements Data Bank (RDB): To modernize the Air Force'sautomated and manual requirements functions.- The Contracting Data Management System (CDMS): To provide adata base management system to manage contracting information.- Depot Maintenance Management Information System (DMMIS):To improve the Air Force's depot management and maintenance Functions.

ERP Pilot Projects:ERP (1998–2004)

These pilots were a part of the Navy's program for operation,maintenance, and modernization of its business systems and relatedinfrastructure. These consisted of the CABRILLO focusing on FinancialManagement, SMART with a focus on Supply Management, NEMAISfocusing on Regional Management and SIGMA with a focus onProgram management (GAO, 2005a).

2 yrs/6 yrs $2.5 B/$1 BTerminated

US (US Navy): discontinued

Weather SystemsModernization:WSM (1980–1995)

These projects were a part of a program to modernize systems toimprove the accuracy, timeliness, and efficiency of NWS's weatherforecasts and warnings (Anonymous, 1991; GAO, 1994, 2000a,b).The four major systems were: Next Generation Weather Radar(NEXRAD), the Next Generation Geostationary OperationalEnvironmental Satellite (GOES-Next), the Automated SurfaceObserving System (ASOS), and the Advanced Weather InteractiveProcessing System (AWIPS).

NEXRAD:9/16

$340 M/$1.5 B US (NationalWeatherService: NWS)GOES-Next: 3/8 $640 M/$2 B

ASOS: 4/9 $72 M/$120 MAWIPS: 9/12 $350 M/$465 M

Table A2Descriptions of UK projects.

Project name(year start–finish)

Description Duration(original/actual)

Budget(original/actual)

Country (owner)

National OffenderManagementInformation Systems:NOMIS (2004–2011)

The goal of this project was to support a new way of working,known as end-to-end offender management, and to replaceexisting prison inmate and local probation area offender casemanagement systems with one integrated system, allowingprison and probation officers and others to access sharedoffender records in real time (NAO, 2009).

4 yrs/7 yrs £234 M/£690 M UK (National OffenderManagement Service): re-scoped

LIBRA (1998–2007) The goal of this project was to provide a national standardIT system across all 42 Magistrates' Courts Committees. Itconsisted of IT infrastructure and office automation facilities,together with a core application to support court work(NAO, 2003).

11 yrs/9 yrs £184 M/£390 M UK (Lord Chancellor'sDepartment): partly

implemented

Benefits Payment Card:BPC (1996–2001)

This was a project to replace the existing paper-basedmethods of paying social security benefits with a magneticstripe payment card, and to automate the national networkof post offices through which most benefits are paid acrossGreat Britain and Northern Ireland (NAO, 2000, 2002a).

3 yrs/5 yrs £1 B UK (Department ofSocial Security andPost OfficeCounters Ltd.)

: partlyimplemented.

National InsuranceRecordingSystem —2: NIRS-2(1995–1998)

This project was initiated under the Private Finance Initiativeto replace the existing Insurance Recording System (NIRS-1)by a large computer system designed to support the InlandRevenue's administration of the national insurance scheme,transfer of data, development of the system to implementlegislative changes arising from the Pensions Act 1995, andthe operation of the new system until 2004 (NAO, 1998,1999, 2002b).

3 yrs/9 yrs £76 M/£146 M UK (ContributionsAgency, InlandRevenue Service)

: contract extended

National ProbationServiceInformation System:NPSIS (1994–2001)

This was a project to provide a national computer infrastructure(comprising personal computers, operating software and acommunications network supported by common servers) anda case recording and management system (CRAMS) across allthe probation services in England and Wales (NAO, 2001).

5 yrs/7 yrs £97 M/£118 M UK (Home Office'sProbation Unit)

Table A3Descriptions of Australian project.

Project name(Year Start–Finish)

Description Duration(original/actual)

Budgeta

(original/actual)Country (owner)

The Edge Project:EDGE (2000–2003)

The goal of this project was to build an expert systemfor the Family Assistance Office for the administrationof claims and payments for people applying forentitlement to family related payments (ANAO, 2005).

3 yrs/3 yrs $59.26 M/$64.4 M AUS (Dep. of Familyand Community Services)Discontinued

Personnel ManagementKey Solution: PMKeyS(1997–2002)

The project was a significant and complex human resourcebusiness process change in Defense, and involved movingmilitary and civilian staff off purpose-built long runninghuman resource legacy systems to a common platform.PMKeyS is the authoritative management record for allDefense personnel in the areas of: administration andleave, development and training, career management,organizational structure, workforce planning, andrecruitment (ANAO, 2006).

3 yrs/5 yrs $103.5 M/$131 M AUS (Departmentof Defense)De-scoped

Cargo ManagementRe-engineering: CMR(1998–2006)

A large and complex Information CommunicationTechnology project for: re-engineering Customs' businessprocesses, increase cargo management efficiency forindustry by developing the Integrated Cargo System (ICS)to replace Customs' Transaction processing systems and toimprove targeting of high risk cargo (ANAO, 2007a).

5 yrs/8 yrs $30 M/$205 M AUS (Customs Service)

High FrequencyCommunication SystemsModernization: HFCSM(1997–2010)

Project to provide an alternative means of long rangecommunication capability for Australian Defense Forcemobile platforms not fitted for satellite communication.The Project involved the development and implementationof a modernized, High Frequency communication fixednetwork and the upgrade of communications equipment onselected mobile platforms (ANAO, 2007b).

7 yrs/13 yrs $505 M/$616 M AUS (Dep. of Defense,Defense materialOrganization)

Re-baselined,de-scoped

a Australian Dollar.

33P. Patanakul / Journal of High Technology Management Research 25 (2014) 21–35

34 P. Patanakul / Journal of High Technology Management Research 25 (2014) 21–35

References

Abednego, M. P., & Ogunlana, S. O. (2006). Good project governance for proper risk allocation in public–private partnerships in Indonesia. International Journal ofProject Management, 24(7), 622–634.

Aloini, D., Dulmin, R., & Mininno, V. (2007). Risk management in ERP project introduction: Review of the literature. Information and Management, 44, 547–567.ANAO (2005). The auditor general audit report: The Edge Project. Australian National Audit Office.ANAO (2006). The auditor general audit report: Management of the Personnel Management Key Solution (PMKeyS) implementation project. Australian National Audit Office.ANAO (2007a). The auditor general audit report: Customs' cargo management re-engineering project. Australian National Audit Office.ANAO (2007b). The auditor general audit report: High frequency communication system modernisation project. Australian National Audit Office.Anonymous (1991). Weather forecasting: Cost growth and delays in billion-dollar weather service modernization. General Accounting Office, US Congress.Anonymous (2004a). A review of the FBI's trilogy information technology program. The National Research Council of the National Academies.Anonymous (2004b). To ensure the customer account data engine's success, prescribed management practices need to be followed. Treasury Inspector General for Tax

Administration.Anonymous (2005). A report to the committee on appropriations, U.S. Congress, House of Representatives, House Surveys and Investigations.Bauer, M. W. (2000). Classical content analysis: A review. In M. W. Bauer, & G. Gaskell (Eds.), Qualitative researching with text, image, and sound: A practical

handbook (pp. 131–151). London: Sage Publications.Browne, G. J., & Ramesh, V. (2002). Improving information requirements determination: A cognitive perspective. Information and Management, 39, 625–645.Chatzoglou, P. D., & Macaulay, L. A. (1997). The importance of human factors in planning the requirements capture stage of a project. International Journal of

Project Management, 15(1), 39–53.Chiu, C. -M. (2005). Applying mean-end chain theory to eliciting system requirements and understanding users perceptual orientations. Information and

Management, 42, 455–468.Clegg, S. R., Pitsis, T. S., Rura-Polley, T., & Marosszeky, M. (2002). Govern mentality matters: Designing an alliance culture of inter-organizational collaboration for

managing projects. Organization Studies, 23(3), 317–337.de Bakker, K., Boonstra, A., & Wortmann, H. (2010). Does risk management contribute to IT project success? A meta-analysis of empirical evidence. International

Journal of Project Management, 28, 493–503.Flyvbjerg, B., Bruzelius, N., & Rothengatter, W. (2003). Megaprojects and risk: An anatomy of ambition. Cambridge: Cambridge University Press.GAO (1989a). ADP acquisition: Air Force logistics system modernization projects: report to the Honorable John Conyers, Jr., chairman. Legislation and National Security

Subcommittee, Committee on Government Operations, House of Representatives, United States Government Accountability Office.GAO (1989b). Air force ADP: Evaluations needed to substantiate modernization program benefits. United States Government Accountability Office.GAO (1989c). Automated information systems: Schedule delays and cost overruns plague DOD systems. Report to the Chairman. Legislation and National Security

Subcommittee, Committee on Government Operations, House of Representatives, United States Government Accountability Office.GAO (1992). Air force ADP: Status of logistics modernization projects and CIM impacts. United States Government Accountability Office.GAO (1994). Weather forecasting- systems architecture needed for national weather service modernization—National weather service modernization. United States

Government Accountability Office (Report Number:).GAO (2000a). Report to the Chairman, Committee on Science, Space, and Technology, House of Representatives.Weather satellites: Action required to resolve status of

U.S. geostationary satellite program. United States Government Accountability Office.GAO (2000b). Testimony before the Subcommittee on Energy and Environment, Committee on Science, House of Representatives, National Oceanic and Atmospheric

Administration, National Weather Service Modernization and Weather Satellite Program, Statement of Joel C. Willemssen Director, Civil Agencies InformationServices Accounting and Information Management Division. United States Government Accountability Office.

GAO (2001). Business systems modernization: Results of review of IRS' customer account data engine project. United States Government Accountability Office.GAO (2004). Business systems modernization: Internal revenue service needs to further strengthen program management. United States Government Accountability

Office.GAO (2005a). DOD business systems modernization: Navy ERP adherence to best business practices critical to avoid past failures. United States Government

Accountability Office.GAO (2005b). Post-hearing questions on FBI management capabilities. United States Government Accountability Office.GAO (2006a). Business systems modernization: IRS needs to complete recent efforts to develop policies and procedures to guide requirements development and

management. United States Government Accountability Office.GAO (2006b). Federal Bureau of Investigation: Weak controls over trilogy project led to payment of questionable contractor costs and missing assets. United States

Government Accountability Office.Han, S. H., Yun, S., Kim, H., Kwak, Y. H., Park, H. K., & Lee, S. H. (2009). Analyzing schedule delay of mega project: Lessons learned from Korea Train Express. IEEE

Transactions on Engineering Management, 56(2), 243–256.Iversen, J. H., Mathiassen, L., & Nielsen, P. A. (2004). Managing risk in software process improvement: An action research approach.MIS Quarterly, 28(3), 395–433.Jiang, J. J., & Klein, G. (1999). Risks to different aspects of system success. Information and Management, 36, 263–272.Jiang, J. J., Klein, G., Wu, S. P. J., & Liang, T. P. (2009). The relation of requirements uncertainty and stakeholder perception gaps to project management

performance. Journal of Systems and Software, 82, 801–808.Jolivet, F., & Navarre, C. (1996). Large-scale projects, self organizing and meta-rule: Towards new forms of management. International Journal of Project

Management, 14(5), 265–271.Keating, C., Rogers, R., Unal, R., Dryer, D., Sousa-Poza, A., Safford, R., et al. (2003). System of systems engineering. Engineering Management Journal, 15(3), 36–45.Keil, M., Mann, J., & Rai, A. (2000). Why software projects escalate: An empirical analysis and test of four theoretical models. MIS Quarterly, 24(4), 631–664.Klakegg, O. J., Williams, T., & Magnussen, O. M. (2002). Controlling information system development projects: The view from the client. Management Science,

48(4), 484–498.Klakegg, O. J., Williams, T., & Magnussen, O. M. (2009). Governance frameworks for public project development and estimation. Newtown Square: Project

Management Institute.Koppenjan, J. F. M. (2005). The formation of public–private partnerships: Lessons from nine transport infrastructure projects in The Netherlands. Public

Administration, 83, 135–157.Liu, Y. Y. -C., Chen, H. -G., Chen, C., & Sheud, T. S. (2010). Relationships among interpersonal conflict, requirements uncertainty, and software project performance.

International Journal of Project Management, 29(5), 547–556.Lyytinen, K., & Hirschheim, R. (1987). Information failures—A survey and classification of the empirical literature.Oxford Surveys in Information Technology, 4, 257–309.Marrewijk, A. v (2007). Managing project culture: The case of Environ Megaproject. International Journal of Project Management, 25(3), 290–299.Marrewijk, A. v., Clegg, S. R., Pitsis, T. S., & Veenswijk, M. (2008). Managing public–private megaprojects: Paradoxes, complexity, and project design. International

Journal of Project Management, 26, 591–600.Miller, R., & Hobbs, B. (2005). Governance regimes for large complex projects. Project Management Journal, 36(3), 42–50.Miller, R., & Hobbs, B. (2009). The complexity of decision-making in large projects with multiple partners: Be prepared to change. In T. M. Willaims, K. Samset, & K.

J. Sunnevag (Eds.), Making essential choices with scant information (pp. 375–389). Basingstoke: Palgrave Macmillan.Miller, R., & Lessard, D. R. (2000). The strategic management of large engineering projects: Shaping institutions, risks, and governance. Cambridge: Massachusetts

Institute of Technology.Miller, R., & Lessard, D. (2001). Understanding and managing risks in large engineering projects. International Journal of Project Management, 19, 437–443.Montealegre, R., & Keil, M. (2000). De-escalating information technology projects: Lessons from the Denver International Airport. MIS Quarterly, 24(3), 417–437.Mora, M., Gelman, O., Frank, M., Paradice, D. B., Cervantes, F., & Forgionne, G. A. (2008). Toward an interdisciplinary engineering and management of complex

IT-intensive organizational systems: A systems view. International Journal of Information Technologies and the Systems Approach, 1(1), 1–24.

35P. Patanakul / Journal of High Technology Management Research 25 (2014) 21–35

NAO (1998). The contract to develop and update the replacement national insurance recording system. UK National Audit Office, Committee of Public Accounts.NAO (1999). Delays to the new National Insurance Recording System. 22nd Report. UK National Audit Office.NAO (2000). The cancellation of the benefits payment card project. UK National Audit Office.NAO (2001). The implementation of the national probation service information systems strategy. Report by the comptroller and auditor general. UK National Audit

Office.NAO (2002a). The cancellation of the benefits payment card project. UK National Audit Office, Committee of Public Accounts.NAO (2002b). NIRS 2: Contract extension. : UK National Audit Office.NAO (2003). New IT systems for Magistrates' courts: The Libra project. Report by the comptroller and auditor general. UK National Audit Office.NAO (2009). The National Offender Management Information System. Report by the comptroller and auditor general. UK National Audit Office.Nelson, R. R. (2005). Project retrospectives: Evaluating project success, failure, and everything in between. MIS Quarterly Executive, 4(3), 362–372.Nelson, R. R. (2007). IT project management: Infamous failure, classic mistakes, and best practices. MIS Quarterly Executive, 6(2), 67–78.OIG (2002). Federal Bureau of Investigation's management of information technology investments. Office of the Inspector General.OIG (2003). Federal Bureau of Investigation's implementation of information technology recommendations. Office of the Inspector General.OIG (2005). The Federal Bureau of Investigation's management of the trilogy information technology modernization project. Office of the Inspector General.Patanakul, P. (2008). Program risk management: How it is done in major defense programs. Project Management Research Conference. Warsaw, Poland: Project

Management Institute.Philbin, S. P. (2008). Managing complex technology projects. Research Technology Management, 51(2), 32–40.Ratcliff, B. (1987). Software engineering principles and methods. Oxford: Blackwell.Sauer, C. (1993). Why information systems fail: A case study approach. Henley-on-Thames: Alfred Waller.Shen, L. -Y., Platten, A., & Deng, X. P. (2006). Role of public private partnerships to manage risks in public sector projects in Hong Kong. International Journal of

Project Management, 24(7), 587–594.Smyth, H., & Edkins, A. (2007). Relationship management in the management of PFI/PPP projects in the UK. International Journal of Project Management, 25,

232–240.Sutterfield, J. S., Friday-Stroud, S. S., & Shivers-Blackwell, S. L. (2006). A case study of project and stakeholder management failures: Lessons learned. Project

Management Journal, 37(5), 26–35.Turner, R. J., & Simister, S. J. (2001). Project contract management and a theory of organization. International Journal of Project Management, 19(8), 457–464.Wateridge, J. (1999). The role of configuration management in the development and management of Information Systems/Technology (IS/IT) projects.

International Journal of Project Management, 17(4), 237–241.Williams, T. (2005). Assessing and moving on from the dominant project management discourse in the light of project overruns. IEEE Transactions on Engineering

Management, 52(4), 497–508.Williams, T., Klakegg, O. J., Magnussen, O. M., & Glasspool, H. (2010). An investigation of governance frameworks for public projects in Norway and the UK.

International Journal of Project Management, 28(1), 40–50.Wirick, D. W. (2009). Public-sector project management: Meeting the challenges and achieving results. Wiley: Hoboken.Yeo, K. T. (2002). Critical failure factors in information system projects. International Journal of Project Management, 20, 241–246.