Politics in Datawarehouse Projects

7/31/2019 Politics in Datawarehouse Projects

1/13

The Politics of Data Warehousing

Marc Demarest

[email protected]

June 1997

ABSTRACT

Data warehousing projects are frequently side-tracked or derailed completely bynon-technical factors, in particular the political treaty lines within the firm, and thepoliticized nature of data itself. Because data warehouses are infrastructure forsociotechnical systems (STSs) within the firm, politics and the exercise of power

are inherent in data warehousing projects, and data warehouse designers have toadopt work practices and methods from non-technical disciplines, think ofthemselves in new ways, and employ some fairly sophisticated qualitativelysociological methods in order to optimize the chances for successful deploymentof data warehouses.

Where Are The Failed Data Warehousing Projects?

The signal-to-noise ratio in the data warehousing market is the lowest it has everbeen. As more and more vendors enter the marketplace, as more and moreprofessional conference-making companies create more and more professionalconferences, as the number of articles that cross our desks each week climbs and

climbs, I cant help feeling that we are missing something in the loud and oftenpointless public discourse on data warehousing: something fairly important.

Where are the failed data warehousing projects?

I believe -- although I have only my own experience as a consultant, and quiet talkwith some of my peers, to support this belief -- that the data warehousingmarketplace woefully underreports both actual cases of failed data warehouse

projects, and normative and prescriptive information for warehousingpractitioners on techniques for avoiding project failure.

Why actual cases of failure are underreported will be obvious to anyone whos

ever written or approved a capital appropriation request for a million dollars:misspending that much money is not something you talk about with PC Weekifyou intend to remain employed or get another job in the IT business.

Whats not so obvious to me is why the normative or prescriptive information

of 13 11/18/99 11:5

he Politics of Data Warehousing http://www.hevanet.com/demarest/marc/dwpo


2/13

about failure-avoidance in data warehousing projects is so scanty. If we mark thefounding of the data warehousing discipline from Inmons seminal definition in1990 (and surely the discipline is older than that by any practical standard,particularly if we give credit to Commander and CommandCenter as theprogenitors of OLAP), we have at least a decade of collective experience in thediscipline, multiplied by, say, 10,000 practitioners worldwide, each working anominal 8 hour day for 220 days: 176,000,000 person-hours of experience. That

ought to be more that enough collective guild practice for some clear prescriptivemodels for failure avoidance to have emerged by now.

Yet the information in the IT trade press -- our disciplinary best-practicesexchange mechanism -- is almost nonexistent. For example, using a commonCD-ROM based data base of about 70,000 IT journal articles from 110periodicals covering the period July 1995 to July 1996, the query:

("data warehouse" OR "data warehousing") AND ("fail" or

"failure")

returned less than 100 articles, while the query

("data warehouse" OR "data warehousing") AND ("success" or

"successful")

returned 279 articles, and the query

("data warehouse" OR "data warehousing")

returned 1283 articles.

Assuming I wasnt being tricked by the disclosure of failure avoidance tips under

the headings of "how to succeed" articles, I looked into the 100 articles from thefirst query, and found some of what I was looking for. Most articles containedthrow-away, unimplementable advice:

But organizations considering this project must also consider thedeployment or data administration issues. Without a well-thought-outstrategy, the data warehouse will fail. At the very least, a warehousedeployment strategy must have three key goals: consistent andsynchronized data, data quality, and adequate end-user query and access

tools.[1]

Such incisive analysis...

Some of the articles were more substantive, however. Larry Greenfield, one of themore astute observers on the data warehousing scene, lists these ten warnings:

1. You may need to spend 80% of your time extracting, cleaning, and loadingdata. Consequently, you may not have enough time to build applicationsagainst the data.

of 13 11/18/99 11:5



3/13

2. You are going to find hidden problems in the systems feeding the datawarehouse.

3. Warehouse projects often turn up the need for data not being captured byexisting systems.

4. After end users receive query and reporting tools, requests for IS written

reports may increase rather than decrease, contrary to your ROI forecast.

5. Your warehouse users will develop conflicting business rules. Many query,reporting, and OLAP tools allow users to perform calculations, and there isnear certainty that users will perform the same calculation differently.

6. Large-scale data warehousing can become an exercise in data homogenizingthat lessens the value of data.

7. Overhead can eat up great amounts of disk space.

8. You can't assign security when using a transaction-processing systemmindset.

9. Data warehouses are high-maintenance systems.

10. You will fail if you concentrate on resource optimization to the neglect ofproject, data, and customer management issues and an understanding of

what adds value for the customer.[2]

All true, I think, and all important considerations. But articles of this type werethe exception, not the norm, which looked more like this:

1. Pick a careful target. A data warehouse is a complex undertaking. It canalso have a dramatic impact on the bottom line. Thus, you should pick arelevant pilot project that can demonstrate the technical considerations andprove that such a warehouse can be productive for the organization.

2. Define consistent business rules. The most important aspect of setting up adata warehouse is to define business rules consistent with the data that will

be going into the warehouse.

3. Create a metadata model. Politics often impact IT projects. The worst way

to start a warehousing project is to develop a corporate-wide datawarehouse and define all the business rules across the company.

4. Demonstrate bottom-line results. [3]

"A data warehouse is a complex undertaking." Theres rocket science-classanalysis for you. "Create a metadata model" -- you can guess what this consultantis selling as the cure-all for data warehousing. All in all, we dont seem to betelling one another very much about what kinds of complexity were dealing with,

of 13 11/18/99 11:5



4/13

why data warehousing projects fail, and what we can do, as practitioners, toprevent project failures.

Modeling Data Warehousing Project Failure

It seems to me that all the advice in the trade press, all my personal experience,and what I know of other practitioners experiences, leads to a pretty simple

four-category model of warehousing project failures. Data warehousing, datamarting and other kinds of decision support systems (DSS) projects fail becauseof:

1. Design Factors: the architecture of the system is in some way deficient.Metadata is ignored as a structural concern, data engineering complexityand time factors are minimized, schema are designed inappropriately or in avaccum, client-side tools are either neglected as components or allowed todominate the design. No consistent business-oriented design method isused, or no method at all is used.

2. Technical Factors: the technology components selected for integrationinto a warehouse ensemble are in some way deficient. The wrongcomponents are chosen, the wrong components are allowed to drive designor implementation, vendor claims are not tested, scalability with respect todata set sizes, query volume or network traffic goes unexamined.

3. Procedural Factors: the way in which the warehouse is built and deployedis in some way deficient. The project is scoped improperly; scope creepisnt countered effectively; proof-of-concept implementations are not usedor not used appropriately; user communities are not involved in the designphase; operations and management procedures in the data center are not

tested against the new and different requirements of warehouseenvironments; pilots do not precede full-scale implementation; designers,

implementors, DBAs, operators and end-users arent trained.

4. Sociotechnical Factors: people and politics arent considered explicitlywithin the project scope.

Of these four categories, only the last -- the sociotechnical factors -- are

significantly new for most IT organizations. Design factors have always been inplay in complex IT projects; data warehousing introduces new design models anddisciplines, but doesnt change the fundamental playing field. Similarly,

technology factors have always played a role in IT project success and failure; alldata warehousing projects do is aggravate this historical problem area by (a)

increasing the number of separately-purchased components involved in the projectand (b) raise significantly the integration burden, in terms of development andtesting activities, imposed on the IS organization. Procedural factors are alsofamiliar territory, though the presence of a large data warehouse platform on thedata center floor frequently means new database technologies, different backupand recovery processes, and sometimes foreign systems management and tuningtools and processes.

of 13 11/18/99 11:5



5/13

But the sociotechnical factors are largely new; this area was pretty muchsubmerged in classical OLTP design. People and politics were less of an issue, orno issue at all. The target user community held little organizational power, wasrarely seriously consulted during design, and could in most cases be compelled touse the system once it was deployed (though there are those interestingly complexunionized situations). And the projects sponsor was almost invariably after a

rock-solid, measurable business objective: process or task routinization, costcontainment, workforce reduction.

Its this last area -- people and politics -- that Im really interested in, because themost painful, indirect and ultimately revelatory discussions I have with my clientsare centered around the politics of data warehousing. Sometimes theseconversations begin under the heading of "engaging with the business unit"; othertimes they are framed by the question "How do we cost-justify a datawarehouse?" or "How do we get management buy-in on a data warehouse?"Infrequently, they are frank enough that other, more central and more overtlypolitical topics surface: how do we get the business units IT organizations (or

end-user communities) to trust us?

I have had these conversations quite often in the last couple of years, particularly

when my firm is rebuilding one of our customers data warehousing projects afteranother firm has cratered the customers project and beat a hasty retreat.Although I cant prove it, I am convinced that the most common set of factorscontributing to data warehousing project failure are not design factors ortechnology factors or procedural factors, but sociotechnical factors: people andpolitics.

Talking About Politics

If we were honest with ourselves, as professionals, we would admit whatRosabeth Moss Kanter suggested in 1979 in a famous Harvard Business Review

article: that

Power is Americas last dirty word. It is easier to talk about money --and much easier to talk about sex -- than it is to talk about power.People who have deny it; people who want to it do not want toappear to hunger for it; and people who engage in its machinationsdo so secretly.

Power is also the last dirty word in data warehousing. And its crippling thediscipline, in my view. Anyone whos actually done a warehousing projectknows very well that most of their organizational energy was spent dealingwith political issues of a few particular sorts, and yet the query

("data warehouse" OR "data warehousing") AND

("political" or "politics")

against the article database I mentioned earlier returned only 29 articlesfrom the database described above. The vast majority of these articles are

of 13 11/18/99 11:5



6/13

notable for the vacuity of their comments on the political factors in datawarehousing, resorting to pat formulae like:

IS managers are realizing that the road to data warehousing is litteredwith as many people and political land mines as it is with technologyobstacles. Turf battles, user resistance, and power struggles can be ascritical to the success of a data warehouse as choosing the right

back-end database or design schema. IS managers need to beproperly prepared to effectively manage people and finesse

organizational issues.[4]

and this gem, found in a value-added reseller trade publication in an articleon how "hot" the data warehousing market is for VARs:

Another factor you need to consider before rushing headlong intodata warehousing can be summed up in one word: Politics. Since theprojects are multiyear deals with costs frequently running into sevenfigures, organizational politics become a key factor in the success of

these endeavors.[5]

Apparently at least one vendor community recognizes the sociotechnicalfactors in the market, if only as an obstacle to quick and easy revenue.

A couple of well-put, pointed articles did emerge, particularly two by Julia

Vowler[6]. But there was no systematic discussion of the politics of datawarehousing to be found anywhere in the almost 1300 articles in thedatabase.

This fact, and a couple of recent conversations with clients, led me to askmyself a few questions:what does "politics" mean in the context of data warehousing?

how would a project team know, before it even started analysis anddesign work on a warehouse, that they were embarking on a political

odyssey? what are the top 10 signs of a politicized data warehousingproject?

what are the 10 practical things a project team can do to minimize theimpact of the sociotechnical factors -- people and politics -- on a datawarehousing project?

Politics In Data Warehousing

Data warehousing projects are always potentially political because:

they cross organizational treaty lines

they change both the terms of data ownership and data access, andexpose the often-checkered history of data management in the IT

of 13 11/18/99 11:5



7/13

organization

they affect the work practices of highly autonomous and powerfuluser communities in the firm.

Politics, Part One: Treaty Line Violations

Any sensible data warehouse design is a part of a larger architectural modeldesigned to deliver data from the points of capture (inside or outside the

firm) to the points of use, probably transforming the data elements in theprocess. A warehouse, in other words, is the key data consolidation andpumping station in a complex data distribution system that begins with the

firms production applications and external data syndicates[7],

wholesalers[8] and enrichers[9] of various sorts, and ends on the intelligentdesktops of managers, analysts, customer care personnel and the like. Thatnetwork always crosses treaty lines: invisible boundaries within the firm thatmark both "turf" and "domains of control".

Some of these treaty lines are functional (with the inevitable tensions that

are an in-built feature of the functional organization as an organizationalform), but the most insidious treaty lines are almost never functional.Consider, for example, the near-invisible treaty line that is drawn across thebackplane of every PC in the modern corporation. Much like the residencejack in the American telephone system, the desktop treaty line marks aboundary between the "provider" and the "consumer" and, just as we arefree to choose our telephones, PC users feel themselves imperatively to be

in rightful possession ofpersonal computers[10], the tool configuration ofwhich is a private affair. When a warehousing project mandates toolsets or

impinges on the desktop in other ways, the treaty line is crossed, andwarning bells often ring out.

Politics, Part Two: Data Ownership and Data Access

If there is any rule that does apply across organizations regardless of theirmarket focus or structure, it is this: power accrues to those who:

gather data

control access to that data.

The irony in data warehouse projects is that, all too often, these laws areworking against the very organization they used to work for: the corporateIS organization. As often as not, the data we cant get for a data warehouseis the data controlled by semi-autonomous divisional or departmental ISfunctions that came into being because the historically stingy policies of thec orporate IS organization with respect to data access hamstrung a division,department or business unit so badly that they built their own IS function to

regain the autonomy they needed for marketplace effectiveness. Thedangers of centralized data control constitute, at some fundamental level,

of 13 11/18/99 11:5



8/13

the raison detre of too many distributed IS organizations, and as a result

their willingness to collaborate with the adversary is minimal at best[11].

But the real political problem with data warehousing is not the loss of dataownership that such projects imply, for every organization asked tocontribute to the warehouse, a loss of control over access to the raw dataitself, something frightening for any group with:

dirty data

ambiguous data

unflattering data.

which, in my experience, is pretty much every commercial organizationoperating today. Inasmuch as divisions, departments and business unitshave gently cooked their dirty, ambiguous and unflattering data for years inthe interest of keeping things clear and simple (and in the legitimate interestof not allowing bad or ambivalent data to get in the way of clarity about thereal state of the business at whatever level), these organizations areunderstandably leery about exposing the uncooked data to inspection byother groups that may have a vested interest or historical reason to pointout the data quality issues.

Politics, Part Three: Work Practice Integration

We are comfortable with the notion that production and service workersare obliged, by the terms and conditions of their employment, to submit tothe discipline of information technology. When I was an accounts receivable

data entry clerk for IBM in the early 1980s, I did my work as the systemtold me to do it: I stepped through tasks the IT presented to me, in theorder it presented them to me, on a 327x terminal screen, for six hours aday, every day. That was my job: to be disciplined by the machine.

As Ive walked up the hierarchy over the last fifteen years, from productionworker to service worker to knowledge worker, my IT has backed off, andthen come back not as a disciplining force, but a facilitating force. That is

so in part because the firm is not permitted, by the terms and conditions ofemployment, to inspect or discipline the work practices of knowledgeworkers. Attempts to do this even relatively benign ones such as TQM

process definition initiatives or restatements of the obvious fact thatelectronic mail is company property are met with resistance on a grandscale, muttering about invasion of privacy, and the magic word:"professionalism." Professionals, it seems, are above needing externaldiscipline: because they practice a profession, whatever discipline they needwith respect to what they do and how they do it is supplied by theprofession, not by IT.

One of the things knowledge workers do, early and often, in a mostly

of 13 11/18/99 11:5



9/13

unscrutinized way, is make decisions. Small and large, important andinconsequential, decisions get made every day about all kinds of things. Andthe plain fact seems to be that most of these decisions are data-free ordata-poor, made based on "experience" or "gut feel" or some otherintangible, unmeasurable quality that is decidely human, decidely part ofthese magic professional disciplines and decidely not something a computercan frame, direct or do (see Treaty Lines above).

The work practice of decision-making has been done historically outside the

IT infrastructure of the firm. Data warehousing projects threaten thislong-standing practice. And they create, in knowledge workers, whatThorsten Veblen called, in another context,"the conscious withdrawal ofefficiency": passive-aggressive behavior on the part of knowledge workercommunities that includes

an unwillingness to participate in requirements gathering, schemadesign activities, and pilots

failure to use deployed warehouses and marts

endless, and pointless, micro-analysis of the "quality" or "real

meaning" of the data provided by warehouses and marts.

Sensing The Political: 10 Warning Signs

How can a project team, comprised mostly of IS personnel, know beforethey start that their data warehousing project is, or is likely to become,politicized? While your mileage may vary, there are common, clear signsthat your project is or will become politicized:

1. a strong but poorly-defined sense of urgency, often bordering on thefrenetic, that cannot be clearly linked to changes in the firms marketposition or financial health is driving the project.

2. demands made by IT management or senior business management fora "cost justification" or "business case" without clear indications ofthe targets for that justification or case

3. lack of a strong, vocal business constituency advocating for theproject

4. project sponsors that do not include both IT and businessrepresentation

5. a stated project objective focused on "data" rather than on "revenue,"

"sales," "customer satisfaction," "marketing," or some other businessobjective

6. people inside and outside the project team talking about the data

of 13 11/18/99 11:5



10/13

warehouse primarily or exclusively in terms of its technologycomponents.

7. warehouse project team members who cannot explain to themselveshow the deployment and subsequent use of the technology ensemblewill materially (a) contain or reduce costs, (b) enhance revenue or (c)manage risk.

8. end-user organizations who do not consistently devote time, at the

individual level, to warehouse or mart analysis, design and pilotactivities

9. senior business management that does not take a proactive interest inthe status of the project, but instead has to be compelled to take aninterest

10. nay-sayers and doubters in end-user constituencies who becomeincreasingly vocal as the project progresses.

Mastering The Political: 10 Countermeasures

1. Change your mindset. As a warehouse designer, you are asociologist, marketeer, diplomat and technologist, probably in thatorder. Spend more time thinking about your constituencies and theirneeds, how youll market to those constituencies, and how youllestablish treaties and working relationships with those constituenciesthan you spend drawing data models and technology architecturediagrams.

2. Get comfortable with being frank, particularly with data set owners.Expect, and say that you expect, to find dirty, inconsistent,incomplete data. Help the data owners get comfortable with this as

state-of-nature, rather than some failing on their part. Help themclean it up; in fact, point out to them that the warehousing or martingproject is an ideal opportunity for them to get out from under theirdata burden, to get clean.

3. Do the sociological analysis first. Know all your constituencies, dataand access, technical and business, what their risk/reward profilesare, what they have to give up for the project to succeed, and what

they gain if the project is successful.

4. Develop the internal marketing plan before you do the first design.Know who you have to sell the project to, what the value proposition

for each target audience is, and how youll know when theyre onboard.

5. Establish clear bilateral treaties or contracts with all the data-ownerconstituencies involved in the project before the design phase of the

0 of 13 11/18/99 11:5



11/13

project is complete. Each treaty should specify who gets what fromwhom when and under what circumstances.

6. Regularly repeat and reset expectations with each constituency, faceto face. Youll know youre doing your job when the constituenciesbegin to notice that youre repeating yourselves.

7. Spend at least 10 minutes of every project meeting discussing thepolitical climate surrounding the project, changes in the

constituencies, status of internal marketing efforts, and late-breakinggossip and hearsay. When team members are uncomfortablediscussing the politics of the project in company, help them to getover that discomfort.

8. Spend at least 15 minutes of every meeting reviewing the work inprogress in terms of (a) the business objectives for the project (whichmust be measurable in terms of the firms income statement) and (b)the user constituencies needs and their work environment.

9. Have a formal process for logging, reviewing and either rejecting oraccepting (with cost and time changes formally noted) all changes to

project scope.

10. Spend a lot of time in the face of the senior technical and businesssponsors for the project. Make sure they know, every week, what thestatus of the project is, what the organizational roadblocks are, andwhat the project team expects the sponsors to do about thoseroadblocks. Use your sponsors like air cover to break down, viaorganizational fiat, resistances in the organization that the project

team cannot remove through diplomacy.

Getting Into Politics

Information technology is, for better or for worse, social these days. Thegood old days of batch and online transaction processing systems designand deployment are gone; we buy those things now, from independentsoftware vendors. The systems we have to build decision supportsystems, computer-supported collaborative work environments, workflowsystems, intranets, extranets, whathaveyou-nets are all deeply andinextricably social applications of IT: computing applied to groups of

people with power, status and a network of relationships.

That means, for better or for worse, that politics is an integral part of ITprojects from here on in. Or out, depending on your perspective. And that,

in turn, means IT professionals have no choice but to get intoorganizational politics, understand the forms, shapes and pathsorganizational politics takes, and become astute at navigating in a politicalenvironment. Not because politics is cool, or fun, but because politics is afeature of the landscape: the beast standing between us and the gate marked

1 of 13 11/18/99 11:5



12/13

"successful project conclusion."

Footnotes1. Judith Hurwitz, "Preparing for the warehouse."DBMS April 1996.

2. Larry Greenfield, "Don't let data warehousing gotchas getcha."Datamation, March 1, 1996.

3. Judith Hurwitz, "A pragmatic approach to data warehousing",DBMSOctober 1995.

4. Rose Cafasso. "Data minefields." PC WeekMarch 18, 1996.

5. "Building a warehouse: more data and more disparate systems makeit a great VAR market." VARBusiness, January 1, 1996.

6. Julia Vowler. "When people come first" in Computer WeeklyDecember 14, 1995, and "Problems in Store", in Computer Weekly,

May 30, 1996.

7.8.9. Data syndicates are member organizations in which members

contribute their data to a syndicator and get back their data and thoseof other members of the group as well, in some unified form. TRW,and A.C. Nielsen are syndicators. Data wholesalers purchase datafrom a suppliuer and resell it down value chains at a mark up, with or

without enriching it first. Dun and Bradstreet is a data wholesaler.Data enrichers typically take a firm's internal data, merge it with datafrom other sources and return it to the firm in question. Harte-Hanks

is a data enricher.

10. This phenomenon - "this is my computer" - has little to do with thefirm's explicit policies about control over the desktop. The firm mayvery well tell its employees that the firm owns the desktop, butMicrosoft and Novell and other desktop vendors repeat incessantly,in every advertisement and on every web page, the same basicstatement: "this is your tool. Where do you want to go today?" That

"you" is not the firm or the IS organization - it is the individual.

11. Although I didn't have the guts to say it in public at the time, one ofthe fundamental reasons I proposed a two-tiered warehouse/mart

enterprise DSS architectural model in 1994 was to partition thecorporate IS organization's legitimate need for a warehouseinfrastructure from the divisional or business unit IS organization'sequally legitimate need for control over their own components of theDSS infrastructure: the marts. The treaty line, in the real world,

2 of 13 11/18/99 11:5



13/13

between CPG wholesalers and warehousers, and CPG retail outlets,seemed to me a pattern we needed to adopt in large-scale DSSenvironments, in the name of getting on with the business ofinformating the business.

Last updated on 06-26-97 by Marc Demarest ([email protected])

The authoritative source of this document is

http://www.hevanet.com/demarest/marc/dwpol.html


Documents

Politics in Datawarehouse Projects