Raford PhD defense final

Large-Scale Participatory Futures Systems

A Comparative Study of Online Scenario Planning Approaches

Michael Flaxman (Chair)Assistant Professor, Urban Information Systems Group, MIT

Joseph FerreiraProfessor of Urban Planning and Operations Research, Associate Department Head and Head of Urban Information Systems Group, MIT

Andres SevtsukLecturer, Department of Urban Studies and Planning, MIT

Noah RafordPhD Candidate, Urban Information Systems Group, City Design and Development Group, Department of Urban Studies and Planning, MIT

Candidate:

Committee:

Noah Raford, PhD Defence, MIT DUSP, Large Scale Participatory Futures Systems: A Comparative Study of Online Scenario Planning Approaches

Outline1. Introduction

2. Review & Synthesis of the Literature

3. Study Design & Methodology

4. Findings & Discussion

5. Conclusion

2


Introduction

3

Qualitative Scenario Planning

Humans have important shortcomings that limit our ability to make effective decisions under conditions of dynamic uncertainty (Dorner, 1997)

“A disciplined methodology for imaging possible futures in which organizational decisions may be played out” (Shoemaker, 1995)

“Tools for foresight discussions... whose purpose is not a prediction or a plan, but a change in the mindset of the people who use them” (de Gues, 1997)

ID Issues

Generate key themes

ID driving forces

Rank factors

Develop draft scenario logic

Create draft final scenarios

Finalise scenarios

Consider implications

Identify indicators

Meetings, conversations

Expert interviews, brainstorm with client, desktop research

Extract key themes, create trends and timelines, key events

Select key uncertainties and forces, list by uncertainty / impact, predetermined drivers

Create scenario snippets, draft systems diagrams, mix and match trends, 2x2 grids

Integrate themes from draft scenarios, create headlines and scenario narratives

Get client feedback, refine, detail, elaborate narrative to final form

Identify key strategic themes, reflect on strategic questions in the context of each scenario

ID key indicators in each scenario for strategic concerns

Client defines key questions through initial conversations & meetings

F2F & phone interviews

Group workshop

Consultant report

Group workshop

Consultant report


Introduction

4

Scenarios

Increased learning

More accuratemental models

Betterdecisions

Improved performance

Reduced individual and group decision bias (Tetlock, 2006)

Gain appreciation of different stakeholders’ positions and attitudes (Chermack, 2003)

Enhanced awareness of environmental change, future risks & opportunities (Weick, 1999)

Greater !exibility and better decision-making (Schwartz, 1997)

Purported Bene!ts

(Chermack, 2003)


Introduction

5

Challenges to Scenario Planning in the Public Realm

Labor intensive & expensive

Bene"ts poorly documented (no veri"cation or reputation systems)

Limited participation (time, space & numbers)

Predominance of senior decision-making elite (participant bias)

Highly dependent on facilitator skills & consultant synthesis (facilitator & author bias)


IntroductionResearch Questions

Do web-based participatory approaches add value to the traditional scenario planning process? If so, where and in what ways?

If not, where do they fall short, in what ways, and why?

6






5. Conclusion

7


Literature Review & Synthesis

8

Urban Planning& Policy Policy

Scenario Planning

ICT Platforms& Web 2.0

Role of the FuturePlanning SupportSystems (PSS)

?


Literature Review & SynthesisUrban Planning & Public Policy

“The future orientation of planning is unique to the field's identity... The very substance of urban planning is founded in time'' (Myers and Kitsuse, 2000)

Four planning traditions (Freidman, 1987):

• Social Reform• Policy Analysis (Simon, 1945; Forrester, 1968; Stokey and Zeckhauser, 1978)

• Social Learning (Majone, 1989; Scott, 1998; Schon, 1983)

• Social Mobilization (Davidoff, 1965; Forester, 1989; Castells, 1977; Healey, 1992; Innes, 1996)

Growing demand for public participation (Arnstein, 1969; Hulchanski, 1977; APA, 1990)

“Urban planning has retreated from strategic, future-oriented topics to become absorbed in operational and managerial activities characterized by short time horizons and value choices likely to be equally short-sighted and ad hoc” (Coucelis, 2005)

9


Literature Review & SynthesisICT Platforms

Planning Support Systems (PSS) “Loosely coupled assemblages of computer-based techniques”, forming a mixed toolbox of techniques to help decision-makers in their daily tasks (Britton Harris, 1989; Brail and Klosterman, 2001; Batty, 2003)

• PPGIS (Warnecke, Beatie, & Lyday, 1998; Craig & Elwood, 1998; Geertman & Stillwell, 2003)

• Alternative Futures Analysis (Steinitz, 2003; Lagigno & Reed, 2003; Hopkins & Zapata, 2007)

• Participatory Agent Based Modeling (Bousquet & Le Page, 2004; Barnaud et al., 2007; Castella et al, 2005)

“Modelling as negotiation” (Guhathakurta, 1993)

“Complicated, convoluted, time-consuming, and intimidating... that do not achieve genuine participation in planning or other decisions” (Innes & Booher, 2004; Cooke & Kotari, 2001)

10


Literature Review & SynthesisCrowdsourcing & Web 2.0

Web 2.0 (O’Rielly; 2005; Anderson; 2007)

• Crowdsourcing (Howe, 2006)

• Collective Intelligence (Levy, 1994; Por, 2008; Malone et al., 2010)

• Human Computation (Quinn and Bederson, 2010; Sakamoto et al., 2010)

“The creation, aggregation and interpretation of strategically relevant information for decision-making through distributed means” (Por, 2008)

Wikipedia, Innocentive, Threadless, CrowdFlower, IdeaScale, Reddit, etc.

Have been studied but rarely used as research instruments themselves (Malone, 2010)

11


Literature Review & SynthesisScenario Planning

Creative, narrative, group-based processes for engaging with uncertainty and change (Wack, 1985; Van der Heijden, 1997)

• Double loop organizational learning (Argys & Schon; 1974)

• Constructivist & social learning theory (Piaget, 1977)

• Sensemaking & organizational awareness (Weick, 1979; Kleine,1999)

• Activity- & practice-based strategizing (Jarzabkowski, 2005; Orlikowski, 1992)

• Competitive advantages of perception management (Boyd, 1976)

Labor intensive & expensive, bene"ts poorly documented (no veri"cation or reputation systems), limited participation (time, space & numbers), predominance of senior decision-making elite (participant bias), dependent on facilitator skills & consultant synthesis (facilitator & author bias)

12


Introduction

13

Contribution of This Study

1. Operational: Help to understand the role that online systems might play in enhancing multi-stakeholder policy creation, speci"cally in the context of the challenges of future-focused, public planning initiatives

2. Methodological: Help to generate new analytical frameworks that can improve our understanding of how such systems may be used for measurement instruments and data analysis platforms






5. Conclusion

14


Study Design & MethodologyResearch Questions• Do web-based participatory approaches add value to the traditional scenario

planning process? If so, where and in what ways?• If not, where do they fall short, in what ways, and why?

Participation• The number and type of

participants involved, and in what phases?

• The geographic scope of participation enabled?

• The range of expert professional disciplines consulted?

Interaction• The number of variables and

opinions incorporated?• The mechanism of analysis,

ranking and clustering?• The time spent on data

collection and analysis?• The amount of user debate

and re!ection?

15


An Exploratory Case Study Approach

A three-tiered, mixed method, case-study based approach, including:

Study Design & Methodology

• Informant interviews to identify key themes and constructs (n=46)

• Creation of two novel, prototypical data generation platforms and application on in-depth cases

• Pair-wise comparison of case studies to a base case

• Evaluation of three additional comparative examples from secondary sources

Interviews

In-depth cases

Base case

Comparativeexamples

16


Study Design & MethodologyCase 1: Futurescaper: The Impact of Climate Change Impacts on the UK

17

186 drivers, ranked, analyzed and visualized as system maps


Study Design & MethodologyCase 2: SenseMaker Scenarios: Future of Public Services Under Financial Uncertainty

18

• 265 participants, micro-scenarios

• Aggregated to three sketch scenarios based on pre-de"ned archetypes


Study Design & MethodologyBase Case: Future of a Northern Region in Spain

19

• Face-to-face scenario method

• Expert scenario consultancy

• 15 in-depth interviews

• Two day workshop, 20 participants

• Four regional scenarios


Study Design & MethodologyComparative Examples

Institute for the Future’s Foresight Engine

• 700 participants• 81 countries• 5,000 submissions in 24 hours

WikiStrat Collaborative Strategy Platform

• 30 teams• 13 countries• ~35,000 words of high-quality content created in 4

weeks

The Future of Facebook Project• 25 video interviews• 109 Quora interactions• ~50 Facebook participants

20


Study Design & MethodologyData Constructs Measured

21

Participant Characteristics• Degree of public openness (including promotion & recruitment efforts)

• Amount of preparation required• The number of participants involved• Reasons for participation• Degree of user anonymity• Type of participants involved

• Level of Education• Professional Experience• Professional Discipline• Age• Geographic Origin

Interaction Characteristics• Tasks performed• Amount and types of input considered

• Amount and types of visualization tools used

• Amount and types of analytical tools used

• Amount and kinds of socialization enabled

• Amount and kinds of feedback provided

• Supplementary interviews


Study Design & MethodologyChallenges

1) The relevant categories and variables for measurement were unknown in advance

2) There was little empirical evidence for, or agreement on, the key outcome variables for scenario planning

3) There were no standard measurement instruments or protocols available that could be readily applied

Both dependent and independent variables were unknown and no standard method of comparison could be established.

An exploratory, or “revelatory” case study design (Yin, 1994) was appropriate.

22






5. Conclusion

23


Findings & DiscussionFinding 1: Greater Number and Diversity of Participants

24

More participants were involved

0

175

350

525

700

125150

700

265

35

Number of Participants

Base CaseSenseMakerForesight EngineWikiStratFoFB

(166)



25


From more diverse locations

0

23

45

68

90

30

82

18

5

Number of Countries Represented

Base CaseSenseMakerForesight EngineWikiStrat



26

Wider range of experts & professional disciplines


From more diverse locations

Base Case: ~20 different disciplines

Case 1: 35 different disciplines

Case 2: Signi"cant experience

WikiStrat: Mixed teams of highly trained inter-disciplinary contributors


Findings & DiscussionFinding 2: Most Participation Was Light, Skewed Towards a Few Heavy Users

27

Base Case: ~4.5 contributions per user, more extensive involvement & conversation through-out workshop

Case 1: ~ 1 contribution per user

Case 2: ~ 1 contribution per user

IFTF: ~6 contributions per user (1.5 original contributions, 4.5 responses to others), 20% of users = 70% of content

WikiStrat: Intensive contribution through-out process, ~7,000 words per team

Name Location Occupation

# of Cards Played

Forecasting Points

# SI Awards

!"#$%&$"'(")*'+ ,-./-. 012.3*' 456 447857 8 9*2&*.:*'(';"/";& <"&12.($-.=+

>??-;#@$"$2-."A+B2-A-(2&$

C5 5DE6 7 F*G.;".

&*!'*$*.(2.**' ?-A-'"/- H.(2.**' ID4 47J7 7 9*2&*.:*'(0H>+0"A3

;"$1#@.3 K-'$A"./=+LM N"$1*;"$2!2".I46 E84 7 N"!O'$1@'9*2&*.:*'(

$1*#A-$P$12!3*.&B?=+?"."/" HA*!$'2!"A+H.(2.**'

II5 5II 7

http://game.smartgrid2025.org/users/365











Findings & DiscussionFinding 3: Rapid Driver Generation & Exploration

28

Driver Generation:

Base Case: 80 hours + 120 minutes in workshop (5 hours per driver)Case 1: ~5 minutes per driverCase 2: ~10 minutes per driverIFTF: ~90 seconds per driver

Clustering & Ranking:

Base Case: ~2 hours in workshop, “not enough time to discuss”Case 1: Instantly sortable along number of dimensionsCase 2: Instantlysortable along number of dimensionsIFTF: N/AWikiStrat: N/AFoFB: Unknown, but “signi"cant and more than we thought”


Finding 4: Most In"uential at Early Stages

Findings & Discussion

29

ID Issues

Generate key themes

ID driving forces

Rank factors

Develop draft scenario logic

Create draft final scenarios

Finalise scenarios

Consider implications

Identify indicators

Case 1:Futurescaper

Case 2:SenseMaker

ForesightEngine

WikiStrat OpenForesight

ScenarioPlanning

Steps

Detailed Case Studies

ComparativeExamples

Increases the likelihood that a wide variety of forces and factors will be included

Increases likelihood that a diversity of perspectives will be achieved

Implies that individual and group biases may be less dominant at the early drivers exploration stage

Strong scaling potential


Findings & DiscussionFinding 5: “The Hourglass Effect”

Tension between structured / unstructured interfaces and analysis approaches

30

Trade off between ease of use & level of participation

More data = greater analytical burden

Case 1: Highly structured interface, open-ended analysisCase 2: Open-ended interface, highly structured analysis

IFTF: Largest number of drivers and social interaction, but very dif"cult tomake sense ofFoFB: “None of us had any idea it would take this long to complete.”

IFTF: Simple, game-like engaging interface, very light analytic powerWikiStrat: High barrier of entry, rich analytic input and deep participation


Findings & DiscussionFinding 5: “The Hourglass Effect”

Tension between structured / unstructured interfaces and analysis approaches

30

Trade off between ease of use & level of participation

More data = greater analytical burden

Case 1: Highly structured interface, open-ended analysisCase 2: Open-ended interface, highly structured analysis

IFTF: Largest number of drivers and social interaction, but very dif"cult tomake sense ofFoFB: “None of us had any idea it would take this long to complete.”

IFTF: Simple, game-like engaging interface, very light analytic powerWikiStrat: High barrier of entry, rich analytic input and deep participation

“People enter these activities with little background experience. Part of your job is to help model the thinking process that they should undergo.”


Findings & DiscussionFinding 6: Role of Visuals & Multimedia

31

Decreasing water qualityDecreasing water availability

Increasing toxic algal blooms

Increasing improved water and sanitation

Increasing diarrhea

Decreasing agricultural productivity

Increasing pollution

Decreasing deaths from cold temperatures

Increasing migration

Increasing water shortages

Increasing flooding

Increasing contamination of water supplyIncreasing droughts

Increasing air pollution

Increasing market

Increasing demand

Decreasing sustainablilty of crop production

Decreasing crop yields

Increasing population displacement

Increasing food prices

Increasing hardships for women

Increasing malaria

Increasing hydrological imbalance



31

Decreasing water quality

Decreasing water availability

Increasing toxic algal blooms

Increasing improved water and sanitation

Increasing diarrhea

Decreasing agricultural productivity

Increasing pollution

Decreasing deaths from cold temperatures

Increasing migrationIncreasing water shortages

Increasing flooding

Increasing contamination of water supply

Increasing droughts

Increasing air pollution

Increasing market

Increasing demand

Decreasing sustainablilty of crop production

Decreasing crop yields

Increasing population displacement

Increasing food prices

Increasing potency of airborne diseases

Increasing malaria

Increasing hardships for women

Increasing hydrological imbalance

Increasing uncertainty in food production



31



31


Findings & DiscussionFinding 7: Social Experience of Online Scenario Building

32

Base Case was far more effective at producing active socialization and interaction between participants

“People need feedback in order to stay involved. You can provide automated feedback, but other people are the best kind of feedback you can possibly ask for.”


Findings & DiscussionFinding 7: Social Experience of Online Scenario Building

32

Base Case was far more effective at producing active socialization and interaction between participants

“People need feedback in order to stay involved. You can provide automated feedback, but other people are the best kind of feedback you can possibly ask for.”

Different kinds of experience were possible with IFTF and WikiStrat

• Ranks and Roles• “Coopetition”


Findings & DiscussionSpeculative Finding 1: Better Outcomes?

The evidence suggests that the use of such systems on their own will not produce the desired outcome of the scenario process

Augment early-stages

• Transparency• Speed• Ef"ciency• Larger scale engagement

Suggests may be effective analytically, but is it psychologically? A hybrid approach is worth exploring to get the full bene"ts

33


Findings & DiscussionSpeculative Finding 2: Impact on Professional Standards

Greater transparency could facilitate reputation systems (eBay, Amazon)

“The futures profession is decentralized, eclectic and intellectually varied: there are no schools that train its elite, few barriers to entry, no certi!cation or regulatory body.” (Pang, 2009)

Commoditize the scenarios market, split between “fast & cheap” or “slow & bespoke”

Trade-off between quality (qualitative) aspects & quantity / speed

34


Findings & DiscussionSpeculative Finding 3: Impact on Scholarly Method

Continuous, self-re!ective and emergent

Allow for user re!ection on, and modi"cation of, research constructs

“Moderators... sometimes have the feeling that they’re barely holding on for dear life, because sometimes the carriage tries to run away without them.”

Requires post-hoc and real-time evaluation, dif"cult to determine what to study in advance

Signi"cantly enhanced potential for creativity, but signi"cant challenges for research design and rigor

35






5. Conclusion

36


Conclusion

37

Contribution

1) Creating under-explored connections between urban planning, public participation, online tools and scenario planning

2) The creation and evaluation of two unique online platforms for participatory scenario planning in urban planning and public policy

2) The creation of an intellectual framework for measuring and evaluating their role in the qualitative scenario planning process.


Conclusion

38

Limitations

Lack of a more rigorous experimental design, more controlled cases or a peer-reviewed evaluation framework

Lack of a controlled, standardized recruitment process for participation

Differences in de"nitions, processes and goals between cases and comparative examples

Strongly dissenting views and participants self-selected out of being interviewed, thereby biasing the results and discussion towards those available and interested in the subject


Conclusion

39

Possible Evolution of These Approaches

Personal Futures Systems

Real-time Horizon Scanning & Scenario Generation Systems

Crowdsourced Think Tank Policy Review

Mass Media Speculation Engines


Conclusion

40

Areas for Future Research

Continue to develop more rigorous measures for evaluating the scenario process and its outcomes

Conduct more controlled research on the impacts of speci"c design and interaction features

Explore the impact of various forms of socialization systems (chat, commenting, voting, etc.) on the process and outcomes


Thank You

41

Noah RafordPhD Candidate, Urban Information Systems Group, City Design and Development Group, Department of Urban Studies and Planning, MIT

[email protected]

August 29, 2011

Questions?

mailto:[email protected]

mailto:[email protected]