Wicked Problems

Solving Complex Business Problems Wicked Problems

At the risk of gross simplification we can state that business Problems as well as other problems fall into two classes Complex (Wicked) Problems and Technical Problems. Of the two Wicked problems are by their name and their nature most difficult. The following pages contain the following some extracts on Social Complexity as a contributory factor to Wicked Problems, A discussion of Wicked Problems in the Context of Software, and finally a 1 page graphical representation of Technical versus Wicked Problesm

Extract 1 is from the paper Wicked Problems and Social Complexity" by Jeff Conklin http://cognexus.org/id42.htmContributory causes to wicked problems are Technical and Social Complexity.

Social complexity, refers to the number and diversity of players who are involved in a project. The more parties involved in a collaboration, the more socially complex. The more different those parties are, the more diverse, the more socially complex. The fragmenting force of social complexity can make effective communication very difficult. Social complexity requires new understandings, processes, and tools that are attuned to the fundamentally social and conversational nature of work.

For example, in a joint project involving several companies and government agencies, there was a prolonged struggle over the mission statement simply because of a terminology difference: each sponsoring agency had their own term for the core concept2, and to pick one term meant disenfranchising one of the agencies. This is a very simple example of fragmentation of meaning. Social complexity means that a project team works in a social network, a network of controllers and influencers including individual stakeholders, other project teams, and other organizations. These relationships, whether they are with direct stakeholders or those more peripherally involved, must be included in the project. For it is not whether the project team comes up with the right answer, but whose buy-in they have that really matters. To put it more starkly, without being included in the thinking and decisionmaking process, members of the social network may seek to undermine or even sabotage the project if their needs are not considered. Social complexity can be can be understood and used effectively, but it can be ignored only at great peril.

Wicked problems fragment the process of project work, especially when the problem is misdiagnosed as tame. Wicked problems also fragment direction and mission if you cant agree on what the problem is, how can you be aligned on a solution? Social complexity fragments team identity the ideal of team unity is compromised by the dynamics of competing interests and hidden agendas. The duality of design tends to divide allegiances between requirements and implementation. Social complexity also fragments meaning key terms and concepts are used in different ways by the different stakeholders. Project teams are often geographically distributed, further fragmenting relationships and communications. Participants in a modern project team are pulled in a thousand different directions by the centrifugal forces of wicked problems, social complexity, and technical complexity (see Figure 6).

Virtually all creative work is a process of design9. All problems call for designing a

solution. All projects are essentially designing something. Design, in both the technical and artistic sense, is the process of creating something new e.g., a new car, a strategic plan, a software program, a stickier web site, next years budget, a new environmental policy.The antidote for fragmentation is coherence. How, then, do we create coherence? In

organizations and project teams in situations where collaboration is the life blood of

success coherence amounts to shared understanding and shared commitment. Shared understanding of meaning and context, and of the dimensions and issues in the problem.Shared commitment to the processes of project work and to the emergent solution matrix. Coherence means that stakeholders have shared meaning for key terms and concepts, that they are clear about their role in the effort, that together they have a shared understanding

Wicked Problems

Our infrastructure is essentially developed. The easy problems have been solved. Designing systems today is difficult because there is no consensus on what the problems are, let alone how to resolve them. The year is 1973 and the topic is public policy. In the landmark article Dilemmas in a General Theory of Planning[1], Horst Rittel and Melvin Webber observed that there are a set of problems that cannot be resolved with traditional analytical approaches. They labeled such problems Wicked Problems.

It seems that wicked problems are not unique to public policy. In a wonderful book published in 1990 called Wicked Problems, Righteous Solutions,[2] Peter DeGrace and Leslie Hulet Stahl pointed out that many of the systems problems facing software developers have all the characteristics of wicked problems. Judge for yourself.

Wicked ProblemsWicked problems, according to Horst and Webber, have ten characteristics:[1]

1. There is no definitive formulation of a wicked problem.Formulating the problem and the solution are essentially the same thing. Each attempt at creating a solution changes the understanding of the problem. 2. Wicked problems have no stopping rule.Since you cannot define the problem, it is difficult to tell when it is resolved. The problem solving process ends when resources are depleted, stakeholders loose interest or political realities change.3. Solutions to wicked problems are not true-or-false but good-or-bad.Since there are no unambiguous criteria for deciding if the problem is resolved, getting all stakeholders to agree that a resolution is good enough can be a challenge.4. There is no immediate and no ultimate test of a solution to a wicked problem.Solutions to wicked problems generate waves of consequences, and it is impossible to know how all of the consequences will eventually play out.5. Every implemented solution to a wicked problem has consequences.Once the web site is published or the new customer service package goes live, you cant take back what was on-line or revert to the former customer database.6. Wicked problems do not have a well-described set of potential solutions.Various stakeholders will have differing views of acceptable solutions. It is a matter of judgment as to when enough potential solutions have emerged and which should be pursued.7. Every wicked problem is essentially unique.There are no classes of solutions that can be applied to a specific case. Part of the art of dealing with wicked problems is the art of not knowing too early what type of solution to apply.18. Every wicked problem can be considered a symptom of another problem.A wicked problem is a set of interlocking issues and constraints which change over time, embedded in a dynamic social context.9. The causes of a wicked problem can be explained in numerous ways. There are many stakeholders who will have various and changing ideas about what might be a problem, what might be causing it, and how to resolve it.10. The planner (designer) has no right to be wrong.A scientist is expected to formulate hypothesis, which may or may not be supportable by evidence. A designer doesnt have such a luxury, they are expected to get things right.Recognizing Wicked ProblemsWhat kind of problems are wicked problems? Here are some examples:

1. Locating a new freeway or homeless shelter. 2. Optimizing all the features on a new model car. 3. Deciding on the best way to re-engineer a business process. Wicked problems arise when an organization must deal with something new, with change, and when multiple stakeholders have different ideas about how the change should take place.

How might you identify a wicked problem? The thing to look for is divergence. If requirements are volatile, constraints keep changing, stakeholders cant agree and the target is constantly moving, in all likelihood, you are dealing with a wicked problem. If considerable time and effort has been spent, but there isnt much to show for it, there is probably a wicked problem lurking somewhere. If business process reengineering is involved, there is a good chance of encountering a wicked problem.

Tame ProblemsAccording to Rittel and Webber, the opposite of a wicked problem is a tame problem. Tame problems may be quite complex, but the lend themselves to analysis and solution by known techniques. A traditional linear processes is sufficient to produce a workable solution to a tame problem in an acceptable period time, and it is clear when a solution has been reached.

It is possible, but not advisable, to pretend that a wicked problem is a tame problem. This makes it easy to address the well-formulated problem with standard techniques. In time, however, the wicked problem will surface as changed constraints, volatile requirements, or stakeholder resistance. If a problem was truly a wicked problem in the first place, treating it like a tame problem before it is actually tamed is a recipe for disaster.

How to Handle Wicked ProblemsThe most fundamental rule for handling wicked problems is that they must not be treated like tame problems. To quote Rittel and Webber: The classical systems approach is based on the assumption that a project can be organized into distinct phases: understand the problems, gather information, synthesize information, work out solutions and the like. For wicked problems, however, this type of scheme does not work. One cannot understand the problem without knowing about its context; one cannot meaningfully search for information without the orientation of a solution concept, one cannot first understand, then solve.[1]

The appropriate way to tackle wicked problems is to discuss them. Consensus emerges through the process of laying out alternative understandings of the problem, competing interests, priorities and constraints. The application of more formal analysis tools is impossible before the problem can be articulated in a concise, agreed upon, well-bounded manner. In other words, the problem must first be tamed.

Wicked ProjectsWicked projects arise when a project is organized to tackle a wicked problems as if it were a tame problem. Another candidate for a wicked project is a Death March Project (as defined by Yourdon in [3], this means a mission-critical project with less than half the time and resources necessary to do the job). Think about it. Death March Projects are often caused when volatile requirements, changing constraints and stakeholder disagreements meet up against immovable deadlines. They are frequently a symptoms of management's unwillingness to tackle wicked problems.

Failure to recognize wicked projects has given Software Development a bad name. A 1994 Standish Group Report[4] found, for example, that about a third of software development projects get canceled and half do not meet their original cost projections. Some have taken this to indicate that the state of software development is in disarray. However, it can also be read as strong evidence that there are a large number of wicked software development projects out there, trying to address wicked problems with the wrong approach.

A typical management reaction to a failed software development project is to conclude that the organization is immature and to aim for more maturity. This usually means imposing more requirements documentation, more analysis, more planning and tracking against the plan. Managers feel that more use of the classic project management processes will avert future disasters. If the failed project was addressing a tame problem, this approach will probably be beneficial.

However, classic project management practices simply do not work for wicked projects. In fact, referring again to Rittel and Webber, attempting to baseline requirements and then use an analytical approach to reach a solution is a recipe for disaster with wicked problems. These problems are resolved through discussion, consensus, iterations, and accepting change as a normal part of the process.

How to Deal with a Wicked ProjectStep 1: Recognize that this is a Wicked Project.The first step to combating most systemic problems is to recognize them admit that they are there. If the project has to satisfy stakeholders who do not agree on fundamental issues surrounding the project, then it is definitely a wicked project. Its fine to say that there should be an executive sponsor or that the customers should speak with one voice. What if they dont? What if they pretend to agree, but deep down, they really dont? This is a wicked project and everyone would be better off if it was recognized as such.

Step 2: See if the Project can be Tamed.In recent Harvard Business Review[5], Robert Herbold gave an excellent example of taming a wicked project. As new COO of Microsoft in 1994, he realized that he had to impose centralized financial, purchasing and human resource systems on staunchly independent Microsoft business units. Heres how he tamed the problem:

1. Executive Support. Bill Gates made it clear to all divisions that anyone not reporting results through the new system would be deemed to have no results at all. Steve Ballmer told balking general managers: We define the measures. You put all your creative juices into growing them.

2. Clear problem Definition. All central systems were implemented with off-the-shelf software which dumped data into a data warehouse. There were few system decisions to be made beyond which package to use and how to configure it. This was accomplished by a small group of technical and business experts because, according to Herbold, you can get the job done in the time it takes a cross-functional committee to decide on its goals.

3. Separation of Concerns / Reduction of Stakeholders. Business units had web-based access to the data warehouse for each system, and they could get at the data however they wanted to, but they did not have access to the packaged software. Thus business unit managers had no stake in the packaged software, and where they did have an interest, which was data access, they had a high degree of control and flexibility. This had the effect of reducing the number of stakeholders for the central system to a very few.

A project is not easily tamed at a low level, although a component-based architecture helps to reduce stakeholders and thus tame the problem.

The main approach for taming a wicked project is to obtain appropriate executive support. This means that the sponsor must recognize and resolve wicked problems. If the sponsor cannot resolve conflicts and the customer does not speak with one voice, the problem has not been tamed.

Step 3. Use Adaptive ProcessesWicked problems are resolved through discussion, consensus, iterations, and accepting change as a normal part of the process. There is nothing that difficult about an adaptive process. It has been used successfully in World Class new product development organizations for decades. Most public policy is developed this way. Most business policy is developed this way. Most marketing is done with adaptive processes. All sophisticated process control systems use adaptive control techniques.

There is nothing wrong with using adaptive processes to solve wicked software development problems. In fact, it if the problem cannot be tamed, this is the only good choice.

There is intense pressure not to use adaptive processes for software development, because they are not considered mature or predictable or controllable. In fact, empirical control techniques are just a successful at controlling processes as defined control techniques, when used for the right kind of problems. Since classic processes have such a poor track record in controlling wicked projects, it is time to give adaptive processes a chance. If the project is a wicked one, the chances of success will be significantly improved.

ScrumPerhaps the best adaptive project management approach for wicked software development projects is Scrum. This Rugby term was first used by Takeuchi and Nonaka in 1986[6] to describe the way in which world class companies develop new products. Considering that a new product development project is by its nature a wicked problem, the best new product development processes present a good model for solving other wicked problems.

Ken Schwaber and Mike Beedle describe the use of Scrum for software development projects in their book Agile Software Development with Scurm[7]. The important thing about Scrum is that it forces incremental action which creates a basis for stakeholder dialog and project feedback. A scrum project collects stakeholder input in a feature list called the Backlog. Each month, the development team starts at the top of the Backlog and selects as many of the top priority features as they think can develop in a month. The team is then left alone for a month, when the result is demonstrated to the stakeholders. This provides a basis for rethinking the backlog features and priorities. The stakeholders are allowed to modify and reprioritize the backlog, after which the team selects its next months work from the top of the list .

Scrum provides a way for the development team to make regular progress even if the problem is not well understood. At the same time, it provides a method for stake holders to discuss the problem and reach consensus. It provides a way for solutions to emerge, as is necessary for the resolution of wicked problems.

At the same time, the Scrum process provides a high degree of predictability. Each line on the Backlog is estimated, and the estimates are added to create an overall project completion estimate. After three months, a graph of the Backlog estimate against time is a highly reliable indicator of the projects progress and estimated completion date. Features may be added or subtracted from the Backlog monthly to adjust for constraints as well as changing stakeholder interests. The trend in the Backlog estimate is a reliable indication of whether the project is converging or diverging.

ConclusionIf software development projects have not responded well to traditional project management practices, the fault may lie in the attempt to apply inappropriate methodologies to software development projects. When projects must deal with conflicts in stakeholder requirements and changes in management constraints, an adaptive process is far more likely to succeed than traditional methodologies. At minimum, adaptive project management practices will quickly determine if a project will converge on a solution, and thus provides actionable data for project portfolio management.

_____________________________1. Rittel, H., and M. Webber; Dilemmas in a General Theory of Planning pp 155-169, Policy Sciences, Vol. 4, Elsevier Scientific Publishing Company, Inc., Amsterdam, 1973.

2. DeGrace, P., and L. H. Stahl. Wicked Problems, Righteous Solutions. Prentice Hall, Yourdon Press, 1990.

3. Yourdon, Edward; Death March Projects, the Complete Software Developer's Guide to Surviving Mission Impossible Projects. Prentice Hall, 1999, 1997

4. The Standish Group. Charting the Sea of Information Technology Chaos. The Standish Group International, 1994.

5. Herbold, Robert J.; Inside Microsoft: Balancing Creativity and Discipline, Harvard Business Review, January 2002.

6. Takeuchi, H., and I. Nonaka. The New New Product Development Game Harvard Business Review, January-February 1986

7. Schwaber, Ken, and Mike Beedle. Agile Software Development with SCRUM. Prentice Hall, 2001