Towards Contested Collective Intelligence

Simon Buckingham ShumConnected Intelligence Centre • University of Technology Sydney@sbuckshum • http://utscic.edu.au • http://Simon.BuckinghamShum.net

Towards Contested Collective Intelligence

or… A tour of the CI design space for Hypermedia Discourse

UniversityofMelbourne•SWARMProject,12th Sept.2017

Contested Collective Intelligence...

In wicked problems, there is no master worldview, ontology or logic

So disagreement is a necessary process and vital ingredient

We can disagree well or badly

CI tools should scaffold and improve this proess(e.g. amplify awareness of how stakeholders are framing the problem, reading

the signals, seeing connections, and judging success)

2De Liddo, A., Sándor, Á. and Buckingham Shum, S. (2012). Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work, 21, (4-5), pp. 417-448. http://doi.org/10.1007/s10606-011-9155-x

Dilemmasand

(partial)

Solutions

Dilemma

If everyone just talks with no structure, it’s hard to

synthesise CI

© Simon Buckingham Shum 5

Hypermedia Discourse

the modelling of discourse / the discourse of modelling

…reading and writing networks of documents, concepts, issues, ideas and arguments

Buckingham Shum, S. (2006). Sensemaking on the Pragmatic Web: A Hypermedia Discourse Perspective. In: 1st International Conference on the Pragmatic Web, 21-22 Sept 2006, Stuttgart, Germany. ePrint: http://oro.open.ac.uk/6442


Discourse§ Dialogue§ Deliberation§ Argumentation § Reflection(Online & F-F Meetings)


Hypermedia§ Modelling discourse relations§ Expressing different perspectives on a conceptual space§ Supporting the incremental formalization of ideas § Rendering structural visualizations§ Connecting heterogeneous content


DiscourseModel

Key ingredients of a Hypermedia Discourse approach


Notation /Visualisation

DiscourseModel




UserInterface

DiscourseModel




UserInterface

ComputationalServices

DiscourseModel




UserInterface


Literacy/Fluency

DiscourseModel


Dilemma

If users are required to structure their contributions to a CI repository, the effort must

provide tangible benefit (not just potential benefits to future stakeholders)

Solution(in small synchronous settings)

A skilled mapper resolves the cost-benefit tradeoff, adding

immediate value to the sensemaking

Issue Mapping (or in a meeting real-time: Dialogue Mapping) based on Horst Rittel’s IBIS scheme

Buckingham Shum, S. (2003). The roots of computer supported argument visualization. In P. Kirschner, S. Buckingham Shum, & C. Carr (Eds.), Visualizing Argumentation (pp. 3–24). London: Springer. ePrint: http://bit.ly/VizArgRoots

http://compendiuminstitute.net


https://www.youtube.com/watch?v=pxS5wUljfjE


this simple set of moves — combined with hypertext,

and mapping fluency —goes a long way…

UKResearchExcellenceFramework(REF)2014ImpactCase

Compendium software (open source)visual hypermedia for managing the connections between ideas flexibly

Deep acknowledgements:

Jeff Conklin CogNexus Institute

Al Selvin & Maarten Sierhuis NYNEX Science & Technology —> Bell Atlantic —> Verizon—> NASA

http://compendiuminstitute.net

20

Structure management in Compendium

§ Associative linkingnodes in a shared context connected by graphical Map links

§ Categorical membership nodes in different contexts connected by common attributes via metadata Tags

§ Hypertextual Transclusionreuse of the same node in different views

§ Templates reuse of the same structure in different views

§ HTML, XML and RDF data exports for interoperability

§ Java and SQL interfaces to add services

Compendium Institute: international communityhttp://CompendiumInstitute.net (now archived)

Global SensemakingNetwork

(2008~2012)http://GlobalSensemaking.net

CogNexus consulting: Issue/Dialogue Mapping http://cognexus.org • http://cognexusgroup.com

Groupaya+CogNexus consulting: Issue/Dialogue Mappinghttp://delta.groupaya.net

Seven Sigma consulting: Issue/Dialogue Mappinghttp://www.sevensigma.com.au/what-we-do/sensemaking.html

“Knowledge Artistry” (Al Selvin)

Selvin, S. & Buckingham Shum, S. (2015). Constructing Knowledge Art: An Experiential Perspective on Crafting Participatory Representations. Morgan Claypool. http://doi.org/10.2200/S00593ED1V01Y201408HCI023

HypermediaDiscoursefluencyatahighlevel

27

Mapping with IBIS Issue-templates to harvest the firm’s collective

intelligence on Y2K contingencies

Selvin, A.M. and Buckingham Shum, S.J. (2002). Rapid Knowledge Construction: A Case Study in Corporate Contingency Planning Using Collaborative Hypermedia. Knowledge and Process Management, 9, (2), pp.119-128.

28

Modelling organisational processes in Compendium using a Template

29

Completing a Compendium template

30

Generating Custom Documents and Diagrams from Compendium Templates

Selvin, A.M. and Buckingham Shum, S.J. (2002). Rapid Knowledge Construction: A Case Study in Corporate Contingency Planning Using Collaborative Hypermedia. Knowledge and Process Management, 9, (2), pp.119-128.

31

Using Compendium for personnel recovery operations planning

Conversational Modelling: real time dialogue mapping combined with model driven templates (AI+IA)

DARPA Co-OPR Project (PI: Austin Tate, AIAI, U. Edinburgh)http://www.aiai.ed.ac.uk/project/co-opr


Mission Briefing: Intent template

Answers to template issues provided in the JTFC Briefing. Answers may be constrained

by predefined options, as specified in the XML schema


Capturing political deliberation/rationale

Dialogue Map capturing the

planners’ discussion of this

option


Planning Engine input to Compendium

Issues on which the I-X planning engine provided candidate Options

35

Mapping with IBIS to build a NASA science team’s collective intelligence for planetary geological exploration

Clancey, William J.; Sierhuis, Maarten; Alena, Richard L.; Graham, Jeffrey S.; Tyree, Kim S.; Hirsh, Robert L.; Garry, W. Brent; Semple, Abigail; Buckingham Shum, Simon J.; Shadbolt, Nigel and Rupert, Shannon M. (2007). Automating CapCom Using Mobile Agents and Robotic Assistants. In: 1st Space Exploration Conference: Continuing the Voyage ofDiscovery, 30 Jan-1 Feb 2005 , Orlando, FL, US. http://eprints.aktors.org/375

NASA: Mars Habitat field trials in Utah desert

NASA remote science team tools

Scientist (Mars)

Scientist (Earth)

Scientist (Earth)

Scientist (Mars)

Scientist (Earth)

Software Agent Architecture

(Mars)

Compendium used as a collaboration medium at all intersections: humans+agents reading+writing IBIS maps

Geology dialogue map between Earth-based scientists and ‘Mars’

Copyright, 2004, RIACS/NASA Ames, Open University, Southampton UniversityNot to be used without permission

Compendium activity plans for surface exploration, constructed by scientists, interpreted by software agents

Compendium science data map, generated by software agents, for interpretation by Mars+Earth scientists

Meeting Replay tool: Earth scientists can browse a (simulated) Mars crew’splanning meeting using Compendium

this simple set of moves — combined with hypertext and mapping fluency —

goes a long way…

BUT…

Dilemma

While co-located mapping is fine for ‘micro-CI’, can we scale this to support asynch. ‘macro-CI’?

Solution

Web-based IBIS mapping

Numerous IBIS-based web apps

http://oystr.cohttp://debatemapper.net

http://evidence-hub.net

http://litemap.net

http://cci.mit.edu/klein/deliberatorium.html

Where our tools fit… Given a wealth of documents…

46

Where our tools fit… and tools to detect and render potentially significant patterns…

47

Where our tools fit… and tools to detect and render potentially significant patterns…

48

Where our tools fit: we need ways to express interpretations

49

50

interpretation

interpretation

interpretation

interpretation


51

interpretation

interpretationinterpretation

interpretation

interpretation

(a hunch – no grounding

evidence yet)

interpretation


…and optionally make meaningful connections

52

predictscauses

interpretation


interpretation

interpretation


evidence yet)

interpretation

Is pre-requisite for

53

prevents

predictscauses

interpretation


interpretation

interpretation


evidence yet)Is inconsistent with

interpretation

challenges

Is pre-requisite for

…and optionally make meaningful connections

Potentially moving towards stories that make sense of the evidence… i.e. plausible narratives / arguments

54

Question

Answer

Supporting Argument… Challenging

Argument…

challengessupports

responds to

Assumption

motivates


55

Question

Answer


Argument…

challengessupports

responds to

Hunch

motivates

56

Question

Answer


Argument…

challengessupports

responds to

Data

motivates


57

Convergence of…web annotationsocial bookmarkingconcept mappingstructured debate

a prototype platform for collective intelligence

Opening demo 2:30-10:30:https://www.youtube.com/watch?v=hxI5jPGScoU

Cohere demo (2011): web annotations with discourse connections

Structured deliberation and debate in which Questions, Evidence and Connections are first class entities (linkable, addressable, embeddable, contestable…)

59

60


— web annotation of document (Firefox extension)

User/community-defined visual language

62

63


Comparison of one’s own ideas to others

De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L.(2011). Discourse-Centric Learning Analytics. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff

Does the learner compare his/her own ideas to that of peers, and if so, in what ways?

De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. (2011). Discourse-centric learning analytics. 1st Int. Conf. Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr). ACM: New York. Eprint: http://oro.open.ac.uk/25829

What epistemic contributions are learners making in the community?

65

Rebecca is playing the role of broker,

connecting different peers’ contributions in

meaningful ways We now have the basis for recommending that

you engage with people NOT like you…

Evidence

Many users can make reasonable contributions to IBIS web apps,

without trainingBUT…

Dilemma

Asynchronous online mapping is tougher to curate: no on-the-

spot sensemaking from a mapper

Solution

Familiar looking web interfaces that guide users on how to

contribute good IBIS

Evidence Hub: structured storytelling for students, practitioners and researchers

Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net

A wizard guides the user through the submission of a structured story:• What’s the Issue?• What claim are you

making/addressing?• What kind of evidence

supports/challenges this?• Link it to papers/data• Index it against the core

themes

Evidence Hub: Argument Maps

Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net

The wizard then generates a structured IBIS tree showing evidence-based claims (and disagreements)

Evidence Hub: professional developmenthttp://learningemergence.net/2013/07/17/deed-elli-ai-ci-systemic-school-learning

Issue

PotentialSolution

SupportingEvidence

(practitionerstory)

Dilemma:

Unstructured deliberation platforms provide no scaleable assistance in making sense of

the collective’s progress

PainPointsinSocialInnovationPlatforms

Catalyst Project Deliverable:

http://catalyst-fp7.eu/wp-content/uploads/2014/02/CATALYST-Analysis-of-pain-points-and-user-feedback.pdf

PainPointsprioritised byorgs whorunsocialinnovationplatforms

Hardtovisualise thedebatePoorsummarisationPoorcommitmenttoactionSustainingparticipationShallowcontributionsandunsystematiccoveragePoorideaevaluation



Effectivevisualisation ofconcepts,newideasanddeliberationsisessentialforsharedunderstanding,butsuffersbothfromalackofefficienttoolstocreatethemandfromalackofwaystoreusethemacrossplatformsanddebates

“Asauser,visualisation ismybiggestproblem.Itisoftendifficulttogetintothediscussionatthebeginning.Asamanageroftheseplatforms,showingpeoplewhatisgoingonisthebiggestpainpoint.”



Participantsstruggletogetagoodoverviewofwhatisunfoldinginanonlinecommunitydebate.Onlythemostmotivatedparticipantswillcommitalotoftimetoreadingthedebateinordertoidentifythekeymembers,themostrelevantdiscussions,etc.

Themajorityofparticipantstendtorespondunsystematicallytostimulusmessages,anddonotdigestearliercontributionsbeforetheymaketheirowncontributiontothedebate,suchisthecognitiveoverheadandlimitedtime.



Bringingmotivatedaudiencestocommittoactionisdifficult.Enthusiasts,thosewhohaveaninterestinasubjectbuthaveyettocommittotakingaction,areleftbehind.

Needtopromptactionincommunitymembers

Reachingaconsensuswasconsideredlessimportantthanbeingenabledtoact.



Motivatingparticipantswithwidelydifferinglevelsofcommitment,expertiseandavailabilitytocontributetoanonlinedebateischallengingandoftenunproductive.

Sustainingparticipationismoreimportantthanenlargingparticipation.

“Itisbettertohavequalityinputfromasmallgroupthanalotofmembersbutverylittlecontent”.



Openinnovationsystemstendtogeneratealargenumberofrelativelyshallowideas.

Poorcollaborativerefinementofideasthatcouldallowthedevelopmentofmorerefined,deeplyconsideredcontributions.

Noeasywaytoseewhichproblemfacetsremainunder-covered.

Verypartialcoverageofthesolutionspace.



Patchyevaluationofideas

Poorqualityjustificationforideas.

Hardtoseewhyratingshavebeengiven.

Unclearwhichrationalesareevidencebased.

Solution

Activity analytics + IBIS semantics permit automated checking of the ‘health’ of a

conversation

CI in Organisations (CSCW journal special issue)

SeearticlebyMarkKleinonattentionmetrics

Crowd-scale deliberation quality metrics + alertsLead: Mark Klein (MIT/Zurich)

https://www.youtube.com/watch?v=UZMJ9mti8h0

Problem-Goal-Exception (PGE) analysis using IBIS syntax checking for potential weaknesses in reasoning

http://catalyst-fp7.eu/wp-content/uploads/2016/01/CATALYST_WP4_D4.2b.pdf

Integrating deliberation metrics in the CI-dashboard


Integrating deliberation metrics in DebateHub


87

“Semantic Google Scholar” — ClaimFinder

Victoria Uren, Simon Buckingham Shum, Michelle Bachler, Gary Li, (2006) Sensemaking Tools for Understanding Research Literatures: Design, Implementation and User Evaluation. International Journal of Human Computer Studies, Vol.64, 5, (420-445).

88

ClaiMaker returns a Lineage tree (the roots of a concept)

Dilemma:

Deliberation schemas focus attention on cold rationality, at the expense of social warmth

Solution

Addition of social channels in an IBIS mapping web app can

restore a sense of connectedness

L. Iandoli, I. Quinto, S. Buckingham Shum, A. De Liddo (2015), On Online Collaboration and Construction of Shared Knowledge: Assessing Mediation Capability in Computer Supported Argument Visualization Tools, Journal of the Association for Information Science and Technology, 75 (5), pp.1052-1067

Async online IBIS Mapping + Social Cues is better than IBIS alone in some respects



Solution

Addition of social channels in an IBIS mapping web app can restore a sense of connectedness

BUT…

But the group using a Ning discussion forum still outperforms Social-IBIS and Plain-IBIS

MutualUnderstanding PerceivedEffectivenessofCommunication


DebateDashboard

sociallyaugmentedCoheremapping

Ningdiscussionforum Cohere

But the group using a Ning discussion forum still outperforms Social-IBIS and Plain-IBIS

AccuracyofPrediction(commodityprices)PerceivedEaseofUse


Writing is endlessly expressive and hard to improve on as a

medium for collective reflection/argumentation

(also a social process)

Dilemma:

But we would still like the machine to do some work for us in making sense of the state of

the CI process or product

Solution

NLP could move us beyond simple forum metrics, and help make sense of

the quality of contribution

Academic Writing Analytics: feedback on analytical/argumentative or reflective writing

Infohttps://utscic.edu.au/tools/awa

101

Highlighted sentences are colour-coded according to their broad type

Sentences with Function Keys have more precise functions (e.g. Novelty)

CIC’s automated feedback tool: analytical writing

CIC’s automated feedback tool: reflective writing

Anearlyparagraphwhichissimplysettingthescene:

CIC’s automated feedback tool: reflective writing

Aconcludingparagraphmovingintoprofessionalreflection:

1

CIC’s Text Analytics Pipeline (TAP) A set of linguistic analysis modules + AWA UI —> OSS release

Preparation of texts:text cleaning –> de-identification –> indexing –> metadata management

Analysis of texts:• Metrics: lengths of words, sentences, paragraphs, and statistics of these• Syllables: metrics at the word level based on syllables• Named Entities: e.g. names of People, Places• Statistics: e.g. noun-verb ratio• Vocabulary: compound words, occurrences at sentence, paragraph and document level• Expressions: epistemic, self-critique and affective compound words• Spelling: feedback on spelling and basic grammar• Rhetorical moves: in analytical and reflective writing• Complexity: measures of the complexity of words, sentences and paragraphs

Disputational talkcharacterised bydisagreementandindividualised decisionmaking.Fewattemptstopoolresources,toofferconstructivecriticismormakesuggestions.Disputational talkalsohassomecharacteristicdiscoursefeatures- shortexchangesconsistingofassertionsandchallengesorcounterassertions('Yes,itis.''Noit'snot!').

Cumulativetalkinwhichspeakersbuildpositivelybutuncriticallyonwhattheothershavesaid.Partnersusetalktoconstructa'commonknowledge'byaccumulation.Cumulativediscourseischaracterised byrepetitions,confirmationsandelaborations.

Mercer,N.(2004).Socioculturaldiscourseanalysis:analysing classroomtalkasasocialmodeofthinking.JournalofAppliedLinguistics,1(2),137-168.

Disputational/Cumulative/Exploratorytalk

Exploratorytalk• Partnersengagecriticallybutconstructivelywitheachother'sideas.

• Statementsandsuggestionsareofferedforjointconsideration.

• Thesemaybechallengedandcounter-challenged,butchallengesarejustifiedandalternativehypothesesareoffered.

• Partnersallactivelyparticipateandopinionsaresoughtandconsideredbeforedecisionsarejointlymade.

• Comparedwiththeothertwotypes,inExploratorytalkknowledgeismademorepubliclyaccountableandreasoning ismorevisibleinthetalk.

Disputational/Cumulative/Exploratorytalk

Mercer,N.(2004).Socioculturaldiscourseanalysis:analysing classroomtalkasasocialmodeofthinking.JournalofAppliedLinguistics,1(2),137-168.

-60

-40

-20

0

20

40

60

80

9:28

9:32

9:36

9:

40

9:41

9:

46

9:50

9:

53

9:56

10

:00

10:0

5 10

:07

10:0

7 10

:09

10:1

310

:17

10:2

3 10

:27

10:3

1 10

:35

10:4

0 10

:45

10:5

2 10

:55

11:0

4 11

:08

11:1

1 11

:17

11:2

0 11

:24

11:2

6 11

:28

11:3

1 11

:32

11:3

5 11

:36

11:3

8 11

:39

11:4

1 11

:44

11:4

6 11

:48

11:5

2 11

:54

12:0

012

:03

12:0

412

:05

Average Exploratory …

Discourse analytics on webinar textchat

Sheffield, UK not as sunny as yesterday - still warm

Greetings from Hong Kong

Morning from Wiltshire, sunny here!

See you!

bye for now!

bye, and thank you

Bye all for now

Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations?

Not at the start and end of a webinar…

Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664

-60

-40

-20

0

20

40

60

80

9:28

9:32

9:36

9:

40

9:41

9:

46

9:50

9:

53

9:56

10

:00

10:0

5 10

:07

10:0

7 10

:09

10:1

310

:17

10:2

3 10

:27

10:3

1 10

:35

10:4

0 10

:45

10:5

2 10

:55

11:0

4 11

:08

11:1

1 11

:17

11:2

0 11

:24

11:2

6 11

:28

11:3

1 11

:32

11:3

5 11

:36

11:3

8 11

:39

11:4

1 11

:44

11:4

6 11

:48

11:5

2 11

:54

12:0

012

:03

12:0

412

:05

Average Exploratory …


…but if we zoom in on a peak…



-100-50

050

100

9:28

9:40

9:

50

10:0

0 10

:07

10:1

7 10

:31

10:4

5 11

:04

11:1

7 11

:26

11:3

2 11

:38

11:4

4 11

:52

12:0

3 Classified as “exploratory

talk”

(more substantive for learning)

“non-exploratory”

…language is used in a manner more akin to “Exploratory Talk” (Neil Mercer)




UserInterface


Literacy/Fluency

DiscourseModel

So, this is the Hypermedia Discourse design space…

Practitioner Fluency

ModellingFrameworks

ComputingPlatform

LearningCurve

Mastery

Domain

Services Interoperability

Discourse

Interaction Design

EffectivenessExperience

Helpful evaluation criteria for CI platforms?

Consolidation of the previous elements into 3 classes of evaluation criteria

How does the Hypermedia Discourse design space and its tradeoffs compare to the SWARM platform?

What can we learn from each other?