DanielWalsh,FounderandPrincipalofnuCognitive,LLC,Austin,TX daniel.walsh@nucognitive.comCopyright2017isheldbytheauthor www.nucognitive.com

SensemakingApplicationsforAgile:CombiningQualitativeandQuantitativeMetricsDANIELWALSH,nuCognitive,LLC.

Sensemaking is a form of distributed ethnography that helps organizations detect emergent patterns, trends, and weak signals. Thenarrative-basedapproach linksqualitativedatawithquantitativedata inorder tounderstand,manage,andmeasuresituations thatarecomplex,uncertain,andambiguous.Thisexperiencereport focusesonhowsensemakingwasusedtounderstand impediments toAgileadoptionandimproveemployeeengagement.

1. INTRODUCTION

Leadersfaceunprecedentedchallengesasbusinessenvironmentsandecosystemsbecomeincreasinglyvolatileand uncertain. How do we measure progress and make sense of complex adaptive systems like corporateculture and business ecosystems? How do we detect weak signals and unarticulated, emergent user needsquickly, before the competition? How do we know if our interventions have desirable effects or harmfulunintendedconsequences?Asorganizationsencountermore“unknown-unknowns,”differentapproachesareneededtomeasureprogressandsenseemergentopportunitiesunderconditionsofextremeuncertainty.

In prior roles as a change agent, engineer, and manager at Intel corporation, I worked on increasinglycomplexsituationsacrossavarietyofdomainsincludingculturechange,innovation,sales,strategicplanning,andproductdevelopment.Iamexcitedtoshareoneapproachformanagingcomplexadaptivesystemscalledsensemaking. Sensemaking helps decision makers at all levels make sense of situations in order to takeeffectiveaction.Ihopeyouwillfindtheapproachinsightful,useful,&intriguingenoughtowanttolearnmore.

2. BACKGROUND

While there is no commondefinition for complexity, some characteristics of complex systems are that theyhavemanyinterdependentpartsthatinteractinnonlinearandsurprisingways

• emergentbehaviorandhigher-orderpropertiesthataredifferentthantheresultantpropertiesoftheconstituentparts

• limitedpredictabilitywheresmallchangesinconditionscanleadtovarydifferentdynamicsovertime• theabilitytoself-organizeandoperatewithoutcentralcontrol

Manyaspectsofchangemanagement,conflictresolution,trustbuilding,andemployeemoraleandengagementare complex. Most socio-technical problems involving humans and technology are complex as well. Whenleaders attempt to take an ordered approach (e.g. roadmaps, Gantt charts, detailed project plans) to solveunordered,complexproblems,thingsnevergoasplanned.Organizationsstrugglewiththelackofinformation,ambiguity,highlevelsofuncertainty,andunforeseencircumstances.Increasedinvestmentsinamoredetailedanalysisormorerigorousplanningprocessrarelyhelp.Differentapproachesareneededwhenthesituationiscomplex.

Sensemaking isanarrative-basedmethodologythatcapturesandanalyzesa largenumberofexperiencesandobservationsinordertounderstand,measure,andmanagechangewithinacomplexadaptivesystem.Thesensemaking approach used in this report was developed by David Snowden (founder and chief scientificofficerofCognitiveEdge)andisbasedonconceptsfromanthropology,neuroscience,andcomplexitytheory.Themethodology bridges the gap between qualitative data (e.g. narratives, observations, experiences) andquantitative data (e.g. questionnaires, surveys) by linkingnarrativeswith quantitative data providedby theparticipants. The combination of narrative and metadata provides a nuanced and holistic perspective thatenables leadership teamsto identifyemergentpatternsand trends inbehaviorsandperceptions.Teamscanalsousethepatternsandunderlyingnarrativestoinformactionplansandaffectchange.

SensemakingApplicationsforAgile:CombiningQualitative&QuantitativeMetrics:Page-2

Like traditional surveys, sensemaking can be structured as either a one-time pulse or as a continuousprocess over time (weeks to months). The degree of anonymity afforded to participants is another designchoice. Anonymous processes promote transparency and perceptions of safety. Non-anonymous processesmakeiteasiertotracktheparticipationofindividualparticipants.

Whilebothsensemakingand traditional surveysoftenutilizequestions,demographicdata,and free formtext,thereareseveralsignificantdifferencesbetweensensemakingandtraditionalsurveys.Thesensemakingnarrative collection process often relies on multiple entries from the same individual participant. Thecollection process is like keeping a continuous journal or diary of small observations. It’s like a captain'slogbook, where participants capture their daily experiences, observations, events, and interactions.“Watercooler”conversations,informaldiscussionsinhallways,gossip,andrumorsareafewexamplesofwhatmayinspireanentryfromaparticipant.

Participantsalsotagtheirentriesbyansweringseveralquestionsabouttheexperienceorobservationtheyshare. What feelings are associated with their story? What is the main theme of the story or what is theexperiencemostlyabout?Whatistheeffectonbusinessormorale?Thesequestionsenabletheparticipanttoaddlayersofmeaningbeyondtheplaintextofthenarrative.Thisissignificantbecausenarrativesfrequentlycontain sarcasm, metaphor, and innuendo which is often misunderstood or misinterpreted by othersunfamiliarwiththesituation.Thequestionsfunctionasmetadatathatisusedforquantitativeanalysisandforidentifying emergent patterns across a large population. By answering questions about their experience,participantslinktheirqualitativenarrativewithquantitativedata.

Sensemaking works well for understanding andmanaging almost any complex adaptive human system.This experience report contains examples and lessons learned from several sensemaking initiatives thatweworkedonatInteltohelpunderstandsomecomplexchallengesincluding

• WhataretheimpedimentsandbarrierstotheadoptionofAgileandLeanmethodsandprinciplesacrossalargemulti-nationalenterprise?

• Howcanorganizationsimproveemployeeretentionratesandincreaseemployeeengagementlevels?• Howcanorganizationsimprovediversityandfosteramoreinclusiveculture?• Whataretheemergentneedsacrossalarge,distributedgroupoftechnologyusers?

3. SENSEMAKING APPLICATIONS TO UNDERSTAND AGILE ADOPTION IMPEDIMENTS AND IMPROVEEMPLOYEEENGAGEMENT

Partofourteam’scharterwastopromotetheadoptionofLeanandAgilemethods,cultivateacommunityofpractice,andimproveproductdevelopmentteams.Afterlearningaboutsensemaking,wefeltthatitcouldyieldinsights about Agile and development work that we had not received from prior approaches, anecdotalevidence,andtraditionalsurveys.

Oursensemakingworkbeganbyidentifyingtheprimarydomainsofinterestanddefiningafewhigh-levelresearchobjectives.Weneededtoalignonwhatwewantedtolearnordiscoverfromthissensemakingeffort.For example,wewere interested in understanding the factors related to intrinsicmotivation and employeeengagement.Thetablebelowcontainsafewofourresearchobjectives.Domainsofinterest Specifictopicsandresearchobjectives

Adoptionimpediments

WhatAgileadoptionimpedimentsarecommonacrossthecompany?Whatfactorsarerelatedtoentrenchedmiddlemanagementthatresistsadoption?Whatpolicies&culturalnormscauseproblemsforAgileandLeanpractitioners?

Effectivepracticesandprinciples

WhatAgile&Leanmethodsandprinciplescorrelatewithpositivebusinessoutcomes?Whatprinciplescorrelatewithpositiveandnegativeemotions?

Self-organization

Canwemeasureandmonitorthelevelsofself-organization?Whatfactorsaffectateam’sabilitytoself-organize?Whatarethecommonimpedimentsinhibitingself-organization?Arethereanybrightspotswhereself-organizationisworkingwell?

After we aligned on the high-level research objectives, our next step was to develop a set of questions orsensemaking framework forcapturing therelevantdata.Thethreeprimarycomponentsofoursensemakingframeworkwere

• anopeningquestiontopromptparticipantstoshareanexperience• asetofquestionsaskingparticipantstodescribeorevaluatewhattheyshared• asetofdemographicquestionsabouttheparticipantprovidingtheresponse

SensemakingApplicationsforAgile:CombiningQualitative&QuantitativeMetrics:Page-3

Weused an open-endedquestionprompt to narrowand constrain the range of possible responses fromparticipants.Wedesigned the prompts to triggermemories and solicited specific experiences.Wewere notinterestedinanyrandomstory,onlytheonesrelatedtoAgileandeventsaboutwork.Weaskedparticipantstosubmitexperiencesbyrespondingtooneofthefollowingthreepromptsoftheirownchoosing.

• ShareaspecificsituationormomentatworkthatgivesyouhopeorconcernforthefutureofAgile.• Describeaspecificeventoractivitythathappenedduringworkthatinspiredorbotheredyou.• Tellastoryaboutworkthatyouwouldsharewithaclosefriend.

Participantsthenrespondedtotheprompttheychosewithfree-form,narrativetext(notalistofbullets).

Participantresponsestothesepromptscouldbeeitherpositiveornegative.Therewereno“right”answerstotheambiguous,open-endedpromptquestionsbutparticipant responseswerehopefully related toAgileandeventsatwork.

Thenext sectionwas a set of questionsdesigned to answer our research objectives.We created a set ofsixteenquestions thatwere informedby the theory in the fieldofpsychologyandneuroscienceor from theknowledgeandexperienceofAgileandLeansubjectmatterexperts.Forexample,wewanted tounderstandthefactorsthat influencedintrinsicmotivationofemployees.OnesetofconceptsweusedwasadaptedfromDanielPink’s(2012)researchonautonomy,mastery,andpurpose.AccordingtoPinkandrelatedresearch,theintrinsicmotivationofknowledgeworkerscanbeimprovedwithincreasedlevelsofautonomy,helpingthemdevelop skills and mastery, and creating connections with purpose greater than themselves. This researchinformedthedesignofoneofourquestionsshownbelow.

Thistriadquestiontypewascomposedofabalancedsetofthreerelatedconcepts.Allthreeconceptswere

eitherpositiveornegative.Weaskedparticipantstopositionamarkerwithinthetrianglewheretheybestfeltitdescribedorreflectedtheirexperience,story,orobservation.Thecloser theypositionedthemakertoanyoneverticeordescription,thestrongerthatstatementwasinthecontextoftheirexperience.Forexample,ifaparticipantpositionedamarkerinthecenterofthetriangle,itmeantthatpeopleintheirstoryweremotivatedequallyby[autonomy]AND[masteringtheirskills]AND[agreaterpurpose].

We also offered anot applicable [N/A] toparticipants if thequestiondidnot apply to the situation theyshared. In this example, participants were asked if people in the story or observation they shared weremotivatedbysomecombinationof[autonomy],[masteringtheirskills],and[agreaterpurpose].Again,therewereno“right”answerstothisquestion.Participantsweresimplyaskedtodescribetheeventorexperiencetheyshared.Questionresponsesaddedlayersofmeaningbeyondthenarrativetextprovidedintheirresponse.Notethatthedescriptionandevaluationquestionswerecompletedbythepersonsubmittingtheresponse,notbyourteamoranyotherresearchers,algorithms,orthirdparties.

Anotherquestiontypeweusedweredyads,whichaskedparticipantstopositionamarkerwheretheyfeltitbestdescribedtheirexperience,story,orobservation.Thecloserthemakerwastoanyonesideordescription,the stronger that statement was in the context of their experience. In our dyad example below, we askedparticipants to evaluate the overall effect on business based on the events that transpired in their story orexperienceshared.Notapplicable[N/A]wasagainofferedasanoptionif thequestiondidnotapplytotheirsituationoriftheydidnotknowhowtoanswer.

Thelastsectionofoursensemakingframeworkcontainedasetofsixdemographicquestions.Wefeltitwas

importantforresponsestobecollectedanonymouslytopromotetransparency,sothedemographicquestion

SensemakingApplicationsforAgile:CombiningQualitative&QuantitativeMetrics:Page-4

provided us with a generic description of the person submitting the response. The demographic questionswere also important for qualitative analysis to understand the similarities and differences across differentpopulations. Demographic data also provided additional context and helped people who were reading theresponsestounderstandwherethepersonsubmittingtheresponsemightbecomingfrom(e.g.biases).Hereareafewexamplesofthedemographicdatawecollected:gender,role,businessunit,geography,gradelevel,andtenure.Theappendixofthisreportcontainsseveraladditionalexamplesofquestionsthatweusedinthesensemakingframeworks.

Withadraft frameworkdeveloped,wetesteditwithasmallpopulationofparticipantspriortocollectingresponsesatalargerscale.Thiswasdoneinitiallywithafewsubjectmatterexpertsusingpaperforms.Afteracouple of iterations, we then tested the framework on small groups of about five to ten people. Severaliterationswere needed to refine and tune the framework tomake it efficient and effective. Here are a fewcriteriaweevaluatedduringtheframeworktestingphase:

• Arethepromptquestionsproducingresponsesrelevanttoourresearchobjectives?Orarewegettingresponsesthatareunrelated?

• Arethequestionswordedasclearlyandsimplyaspossible?Areanyquestionsconfusingparticipantsorarethereanythattakeparticipantsawhilecomprehend?Arethequestionsself-explanatory?

• Arethereanyquestionswithalargenumberofpeopleselectingnotapplicable?Ifso,whateditswouldmakethequestionmoreapplicable?Orshouldthequestionbedroppedentirelyfromtheframework?

• Howlongdidittakeforaparticipanttocompletearesponse?Aretheretoomanyquestions?• Howcouldtheusabilityandoveralluserexperienceforparticipantsbeimproved?

Afterourframeworkwassufficientlytestedandvalidated,webegantheexecutionphasesofthesensemakingmethodology which included narrative collection, emergent pattern analysis, and intervention design anddeployment.Thediagrambelowdepictsourexecutionprocessatahigh-levelstartingwithstorycollection.

3.1 CollectionWecollectedexperiencesfrompeopleanonymouslyusingawebapplication(viaasmartphone,PC,orotherdevice).Printedformswerealsoconsidered,butwedecidedagainstthembecauseoftheextraeffortneededtotranscribe responses. Participants responded to the prompt question(s) by sharing a short story (i.e. afragmentedmicro-narrative).Mostresponseswerenotfullyconstructedstorieswithabeginning,middle,andend. They were mostly simple observations, rumors (e.g. café conversations or hallway chats), or feelingsaboutaspecificeventor incident.Amajorityof theresponsesrangedfromafewofsentencestoacoupleofparagraphs.

We then asked participants to answer several questions aboutwhat they shared. The questions enabledparticipants toaddadditionalmeaning to their storybeyondwhat theyshared inplain text.Thisprocessofself-significationbyparticipantseliminatedtheneedforaresearcheroralgorithmtotagtheresponse,whichoften introduces bias and errors. We found that it took about ten to twenty minutes for participants to

SensemakingApplicationsforAgile:CombiningQualitative&QuantitativeMetrics:Page-5

complete a response the first time. However, subsequent responses took less time as participants becamefamiliarwiththeframework.

3.2 EmergentPatternAnalysisOncewehadabouttwohundredresponsescollectedforoursensemakinginitiativeonAgileadoptionbarriers,themetadatafromtheparticipants’answerstoquestionswasanalyzedwithstatisticalsoftware.Thedatawasalso visualized graphically for coherent patterns and correlations between question parameters werecalculated. In the following pattern analysis example, we wanted to understand which Agile and Leanprinciples correlated most with stories that had positive effects on business. We asked participants whatprinciples were present or lacking in the story they shared. The chart ranked principles along the y-axisaccordingtohowwelltheycorrelatedwithstoriesthathadapositiveeffectonbusiness.Thecoloringreflectedwhetherthestoryhadeitherpositive(green)ornegative(red)effectsonbusiness.Thex-axisrepresentedthepercentageofstoriesineachofthesevenbusinesseffectcategories.

Accordingtothisdataset,75%ofthestoriesthatwereabouttheprincipleof[RespectforIndividuals]alsohadastronglypositive(darkgreen)effectonbusinessaccordingtothepersonthatsubmittedthestory.Storiesabouttheprinciplesof[trust]and[learning]alsohadstrongcorrelationswitheffectsonbusiness.Whentheseprincipleswere present in a story, the story’s effect on businesswas positive.When these principleswerelacking but needed in a story, the story’s effect on business was negative, according to the participantsubmittingthestory.

Wealsoanalyzedpatternsaroundattitudestowardchange.AftersubmittinganexperienceaboutworkorthefutureofAgile,participantswereaskediftheattitudestowardchangeintheirstorywassomecombinationof [thoughtful consideration], [enthusiastic], or [keep as is]. Each of the dots within the triad represents aspecific experience, observation, or story shared by a participant. The hashed lines represent the averageposition for each set of stories.Again, the coloring reflectswhether the storyhad eitherpositive (green) ornegative(red)effectsonbusiness.Thesizeofthedotreflectswhethertheeventwasrare(small)orcommon(large). Stories where the attitude towards change were [keep as is] correlated with negative effects onbusiness,while storieswitha [thoughtful]or [enthusiastic]attitudewerecorrelatedwithpositiveeffectsonbusiness.Forexample, thedotontheright triadclosest to the topwasastory fromaparticipant thathadapositiveeffectonbusiness(green),thatwassomewhatcommon(mediumsizeddot),andtheattitudetowardschangeinthestorywas[thoughtfulconsideration].

SensemakingApplicationsforAgile:CombiningQualitative&QuantitativeMetrics:Page-6

As part of this effort we also wanted to find a way to measure effects related to managers that wereresistanttochange.WhileworkingonAgileandLeantransformations,weencounteredmiddlemanagersthatresisted changeeven though frontline employees andexecutive sponsorswereadvocates for adoptingAgileandLean.Wemetaphoricallycalledthisthe“permafrost”layerwhichactedasifitwerefrozen,perhapsduetobureaucraticpolicies,statusquomindsets,orconflictingpriorities.

Thefollowinggraphwasoneattempttomeasureandmakesenseofpermafrostrelatedstories.They-axisrepresents the attitude towards change in the story. The x-axis measures the alignment between theparticipantandtheirmanagement.Forexample,ifaparticipantstronglysupportedtheeventsinthestorytheysharedandtheythoughtthattheirmanagementdidnot,thenitwouldresultinalowdegreeofalignmentalongthex-axis.

We founda strongcorrelationbetweenstorieswithanegativeeffectonbusiness (i.e. coloredred)anda

[keep as is] attitude towards change combinedwith a perceived lack of alignment between employees andtheirmanagers(lowerleftcorner).

3.3 InterventionDesignandDeploymentThenextphaseforourAgileadoptionsensemakingwastomakesenseoftheemergentpatternsandtrendsinorder to take action. As part of a facilitated action planning workshop, several patterns, trends, and weaksignals were sharedwith participants. The narratives associatedwith patterns were purposefully withheldwhileparticipantswereaskedtocomeupwithseveralguessesforwhatwasbehindthesepatternsandtrends.Forexample,participantsspeculatedthatthestorieswitha[keepasis]attitudetowardschangeandanegativeeffectofbusinesswerefromaparticularpartoforganization.Aftertheirguesseswerecaptured,participantswere allowed to read the individual stories that generated the patterns and trends. We challenged theparticipants to findevidencethatvalidatedor invalidatedtheirguesses.Thisprocessgenerated insightsandlearningwhichcouldthenbeturnedintoideasforinterventionsandactionplans.

Here isasimpleexample forhowthedataandstorieswereused todevelop interventions.Oneaspectofself-organization thatwewanted tounderstandwas the focuson teamversus individuals.Usingadyad,weaskedparticipantstodescribewhetherrecognitionorconsequencesinthestorywerefocusedon[teamsoverindividuals]or[individualsoverteams].Severalofthestoriestoward[individualsoverteams]werenegative.One of the negative stories heavily skewed toward [individuals over teams] was titled “How IndividualContributor Culture can be blocker for Success.” Insights from this pattern and specific story inspired aninterventiontoreducethepredominanceofthephrase“individualcontributor”fromthecompany’slanguage.Instead,ourteamofAgilecoacheswouldtrytousethephrase“teamcontributor”over“individualcontributor”tohelpshiftculturetowardteamsuccessvsindividualsuccess.Whileindividualcontributorwasstillusedatthecompany,wefeltthatdiscussionsabouttheindividualcontributorculturehelpedimproveAgileadoptionsanddevelopmentworkingeneral.

SensemakingApplicationsforAgile:CombiningQualitative&QuantitativeMetrics:Page-7

Aspartofanothersensemakingworkshoptounderstandemployeeengagement,aportfolioofsafe-to-failinterventionswas developed by participants to amplify beneficial patterns and dampen negative ones. Theinterventionsweredesignedas safe-to-fail (vs. fail-safe)because changeswithina complexadaptive systemare unpredictable and unintended consequences were expected for every intervention. One of the keyquestions we asked participants during ideation and brainstormingwas “How dowe getmore stories liketheseandfewerlikethose?”Whatmighttheorganizationdotogetmorestories,ordots,about[enthusiastic]attitudes towards change that have positive effects on business?What interventions might reduce or helpprevent some of the red negative stories about a [keep as is] attitude toward change? These safe-to-failinterventions alsowent through a process of constructive, critical feedback tomake themmore robust andimprovethechancesofsuccess.Severalinterventionswerethendeployedinparallel.Asnewexperiencesandobservationswereshared,thenarrativeschangedandnewpatternsemerged.

4. WHATWELEARNED

HerearesomeofourinsightsfromthesensemakinginitiativeonAgileandLeanadoptionimpediments:• Stories that were about the principle of Respect for Individuals had the strongest correlation with

businessresults.LearningandTrustwerestronglycorrelatedaswell.• Stories that were about Lean and/or Agile had disproportionately higher rates of Respect for

Individuals,LearningandTrustpresent.• The sensemakingmethodwas able tomeasure some aspects of thepermafrost problemwhichwas

correlatedwithnegativeeffectsonbusiness.• Thesensemakingmethodwasable tomeasuresomeaspectsofself-organizationand teamcohesion

whichwerecorrelatedwithpositiveeffectsonbusiness.• Weneedtoimprovethedegreeofalignmentbetweenpeopleandtheirmanagement.• Weneedtodampena[keepasis]attitudetowardschange.• Weshouldfocusonlong-termbenefitsdespiteshort-termpain.• Weshouldpushformoreteam-levelcommitmentsandteamrecognition.• Weneedtohelpteamsunderstandandconnectwiththeirhigherpurpose.

4.1 ReflectingonthebenefitsofthesensemakingmethodologyTherewere several unique benefits we experienced from the sensemakingmethodology thatwe didn’t getfrom traditional surveys and focus groups. The combination of quantitative metrics connected to richnarratives helped leaders understand the challenges in context and created a sense of urgency. A series ofpersonal, compelling, and emotional stories fromparticipantswas difficult for leaders to ignore or dismiss.Afterreadingonestory,avice-presidentcommentedthat itdidn’taccuratelyreflectthewholesituation.Thevice-president asserted that the person who submitted the story only understood part of the problem.However,theleaderquicklyrealizedthattheemployee’sperceptionofthetroublingsituationwasgenuineandthat the leadershipteamhadaroletoplayto improvetransparencyandmoreeffectivelycommunicatewithemployees.

Leadersatall levelsoftheorganizationusedthedatafromthesamesensemakinginitiativetotakeactionandaffectchange.Thesensemakingdatasetwasfractal,orself-similar,atdifferentlevelsoftheorganization.Thismeant thatsomeof the team-levelpatternsweresimilar to thepatternsof the largergroup.Thewholeorganization was engaged to develop and deploy interventions to try and get “more stories like these andfewerlikethose.”

The ability to quantify andmeasure progress within a complex adaptive systemwithout knowing whatsuccess looks like in advancewas another powerful advantage of sensemaking. For example, in a previousgraphicaboutattitudestowardschange,abouthalfofthestorieswereontheleft-handsideofthetriadtowardthe [keepas is] cornerandwere signifiedbyparticipantsashavinganegativeeffectonbusiness (red).Theorganizationcouldsetagoal toreducethe fractionof thestoriesonthe leftside from50%to40%overthenextquarter. Several safe-to-fail interventions could thenbedeployed to shift thepattern toward the right-handsideandprogresscouldbemeasured.“Morestoriesliketheseandfewerlikethose”couldbeconvertedintoasetofquantitativegoalsandmetricsfortheorganization.

4.2 RecommendationsforfuturesensemakinginitiativesAfter leading many sensemaking initiatives we learned a few things about the methodology that otherpractitioners may find useful in future work. For instance, we found it particularly difficult to keep the

SensemakingApplicationsforAgile:CombiningQualitative&QuantitativeMetrics:Page-8

frameworksshort.Ifpossible,limitthenumberofquestionstoaboutthreeorfourtriads,fourtosixdyads,andthreetofourmulti-choicequestionsatmost.Wehadanecdotalevidencethatpeopledidnotsubmitresponsesfrequentlyiftheframeworkwastoolong.Wealsofoundthatpresentingallofthequestionsatonceimprovedthe completion rate and user experience for participants versus splitting the questions up across separatepages.

Oneofthemoreusefulquestionswastoaskparticipantstogivetheirentryanameorheadline.“Ifthiswereanewsstory,whatwouldtheheadlineread?”Thenameprovidedinterestinginsightsandenabledustoscanentries quickly. We didn't ask participants to name their stories in one framework in order to reduce thenumber of questions and we greatly regretted it later during analysis. Names are a powerful way to addmeaning.

One challenge we encountered was getting people to submit stories. Many organizations suffered fromsurvey fatigue, and it was difficult to get people's attention throughmass emails and electronic marketingcampaigns. Word of mouth and leveraging personal relationships worked better than mass mediacommunication.Anothertechniquethatworkedwellwastohavepeoplevolunteertobestorychampionsforashortperiodof time, typicallyoneor twoweeks.During this time, itwas their job to shareexperiencesandobservationsonadailybasis.Whentheyweredone,theypassedtheresponsibilitytoanothervolunteer.Wealsobelievedthat integratingthecollectionprocesswithsomepartofaworkprocesshelped(e.g.responsesarecollectedasparteveryretrospectiveanddemo).

Alessonwelearnedfromstatisticalanalysisworkwasthatwehadtobecarefulwithstatisticalclaimsduetothefactthatstorycollectionwasbasedonconveniencesampling.Thesampleswerenottrulyrandom,wemostly received responses fromparticipants that felt like submitting a story.Rather than framepatterns asstatisticallysignificant,weinsteadaskediftherewasenoughevidenceinordertotakeaction.

Another technique that worked well was to develop dashboards and automated reports to enableparticipantstodotheirownquantitativeanalysis.Thisdistributedapproachtoanalyzethedataimprovedouroverallabilitytoscanthedata forweaksignalsandpatterns. Italsohelpedtoengagea largegrouptoaffectchange.

5. ACKNOWLEDGEMENTS

I’dliketothankandacknowledgemymentors,coachesandco-travelersonthissensemakingjourney.IoweadebtofgratitudetoDavidSnowdenforhisthoughtleadershiponsensemakingandcomplexitymethods,andhismentoringovertheyears.I’dalsoliketothankMichaelCheveldaveforhiscoachingaswecameuptospeedon the design and execution of sensemaking initiatives. A big thank you is due to Rhea Staddick asmy co-travelerandconfidanteformuchofthesensemakingworkatIntel.ThanksalsotoNanetteBrownforbeingmyshepherdonthisexperiencereportfortheAgile2017conference-youradvicewashighlyvaluedandgreatlyappreciated!

6. APPENDIX

REFERENCES:Pink,D.H.(2012).Drive:thesurprisingtruthaboutwhatmotivatesus.NewYork:RiverheadBooks.Snowden,David.SenseMaker™website,http://cognitive-edge.com/sensemaker/