Action Design Research and Visualization Design

Nina McCurdy

University of Utah

nina@cs.utah.edu

Jason Dykes

City University London

j.dykes@city.ac.uk

Miriah Meyer

University of Utah

miriah@cs.utah.edu

ABSTRACTIn applied visualization research, artifacts are shaped by aseries of small design decisions, many of which are evaluatedquickly and informally via methods that often go unreportedand unverified. Such design decisions are influenced not onlyby visualization theory, but also by the people and contextof the research. While existing applied visualization modelssupport a level of reliability throughout the design process,they fail to explicitly account for the influence of the researchcontext in shaping the resulting design artifacts. In thiswork, we look to action design research (ADR) for insightinto addressing this issue. In particular, ADR o↵ers a frame-work along with a set of guiding principles for navigating andcapitalizing on the disruptive, subjective, human-centerednature of applied design work, while aiming to ensure relia-bility of the process and design, and emphasizing opportuni-ties for conducting research. We explore the utility of ADRin increasing the reliability of applied visualization design re-search by: describing ADR in the language and constructsdeveloped within the visualization community; comparingADR to existing visualization methodologies; and analyz-ing a recent design study retrospectively through the lens ofADR’s framework and principles.

CCS Concepts•Human-centered computing ! Visualization the-ory, concepts and paradigms;

KeywordsDesign study; applied visualization design research; actiondesign research; concurrent evaluation

1. INTRODUCTIONThroughout the visualization design process, artifacts are

shaped by many design decisions, only some of which are for-mally validated. Many of these decisions are instead evalu-ated through quick, informal, and light-weight mechanisms,

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full cita-tion on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-publish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from permissions@acm.org.

BELIV ’16, October 24 2016, Baltimore, MD, USAc� 2016 ACM. ISBN 978-1-4503-4818-8/16/10. . . $15.00

DOI: http://dx.doi.org/10.1145/2993901.2993916

most of which are not reported or verified. Furthermore, incollaborative settings these design decisions are influencednot just by visualization theory and guidelines, but also bythe people and context in which artifacts are designed.

Documented adherence to applied visualization process [15,23] and decision [18, 20] models a↵ords a level of reliabilityfor the resulting artifacts. These models stress reliabilitythrough the grounding of design decisions in established vi-sualization principles and the validation of artifacts withinthe application domain. They do not, however, explicitlyaddress the more subjective shaping of artifacts by the peo-ple and context involved in the project, a shaping that canat times ignore or even go against established visualizationconventions. Nor do they explicate the role that deliberatedisruption on the part of the visualization designer plays inthe shaping of artifacts. Taken together, these gaps revealimportant influencing factors within the visualization designprocess for which there are not yet established guidelines forensuring reliability of the resulting artifacts.

Within information systems research, a recently proposedmethodology called action design research (ADR) o↵ers aframework that aims to ensure the reliability of designed ar-tifacts through adherence to a set of principles [24]. Likevisualization design research, ADR seeks to contribute de-sign knowledge by solving real-world problems, while sup-porting the messy, iterative, human-centered nature of thedesign process. ADR, however, explicitly incorporates ap-proaches from social science that acknowledge and facilitatethe e↵ects of people and context on the shaping of artifacts,and specifically those that occur when actions taken by thedesign researcher result in a disruption of the target users’processes or understanding, and vice versa. This explicitincorporation of established social science methods, namelythose from action research [5, 16], provides guidance for re-liably designing artifacts in a real-world context — preciselywhere gaps exist in current visualization models.

In this paper we explore the use of ADR for visualiza-tion design research. More specifically, we are interested inhow ADR can systematically account for the disruption andsubjectivity involved in the visualization design process inorder to address reliability threats. We do this by: describ-ing ADR in the language and constructs of visualizationresearch; comparing ADR to existing visualization designmodels; and analyzing a recent design study retrospectivelythrough the lens of ADR’s framework and principles. Thisexploration has led us to conclude that ADR is helpful inunderstanding the role of people and context in the shap-ing of visualization artifacts, and in providing pointers to

places in the visualization design process where this shap-ing can, and should, be captured and reported. Moreover,ADR provides sca↵olding for better understanding the re-lationship of controlled, experimental, in vitro research anddesign-orientated in vivo research within visualization.

2. ACTION DESIGN RESEARCHThe ADR methodology consists of a set of guiding prin-

ciples and a high-level process model. While ADR has beenreported as a development methodology in visual analyticssystems research [27], it has yet to be explored in light ofapplied visualization design methods. In the following sec-tion we discuss the development of the methodology withinthe information systems community, followed by an expli-cation of each of the principles and process stages. Eachof these discussions includes an analysis of how ADR con-cepts reflect, apply to, or inform current visualization designresearch theory and practice.

2.1 OverviewInformation systems research is driven by a dual mis-

sion: to generate valuable information systems knowledgeand to create e↵ective solutions to real-world problems thatinform an application’s domain [24]. Early process modelsto achieve this mission focus on incorporating both exper-imental and design methods, while emphasizing relevancethrough a grounding in real-world problems [12, 26]. Seinet al. [24] critique this approach for not recognizing or cap-turing the influence of people, organizations, and contexton shaping technology throughout the design process. Theyargue further that appropriate forms of evaluation that con-sider such elements must be interwoven throughout the de-sign process.

S2: Building, Intervention,and Evaluation

S3: Reflectionand Learning

S4: Formalization of Learning

P1: Practice-Inspired ResearchP2: Theory-Ingrained Artifact

P3: Reciprocal ShapingP4: Mutually Influential RolesP5: Authentic and Concurrent Evaluation

P6: Guided Emergence

P7: Generalized Outcomes

S1: Problem Formulation

Figure 1: An overview of the ADR methodology(adapted from [24]) indicating the relationship of thestages to each other, as well as the guiding princi-ples . The deliberate omission of an arrow pointingfrom stage 1 to stage 2 may seem counterintuitive,but emphasizes the key role of reflection and learn-ing: movement from stage 1 to stage 2 must occurvia stage 3. Whereas stages 1 - 3 are iterative andcyclic, stage 4 is isolated and visited only after thepreceding stages are completed.

The action design research (ADR) methodology [24], anadapted version of which is shown in Figure 1, is an attemptto address these shortcomings. Specifically, ADR explicatesthe influence of the research environment as well as the rolesand influences of members of the research team in and on thedesign process and the resulting technology artifact. ADRstructures the design process around cycles of building, in-tervention, and evaluation that mirror the cycles of plannedintervention and reflection used in action research. Actionresearch embraces disruption and action on the part of thedesigner as a means to learn about a problem [5, 16]. As aresult, within the ADR methodology, the resulting artifactis considered an instantiation of the space, time, community,and process in which it is developed [21, 22], and termed theensemble artifact to reflect this quality.

Our own experience suggests that, whether consciouslyor not, many visualization designers learn through actions,using explicit methods like technology probes [6] and datasketches [11], or more implicit approaches such as abstrac-tion and visualization suggestions during interviews. Through-out the design process, visualization designers disrupt andinfluence both the target users and the problem context,while simultaneously being disrupted themselves. We sus-pect that adherence to the principles and process of ADRcould increase the reliability of visualization design researchby applying qualitative methods, such as those described ingrounded evaluation [7], and providing guidance for captur-ing important details around insightful disruptions.

2.2 ADR PrinciplesADR’s seven guiding principles express the core values of

the methodology and serve as a system of reminders to helpensure that research conducted by the ADR team — com-prising visualization researchers and domain practitioners— is reliable throughout. Principles associated with earlystage research (P1, P2) stress the importance of ground-ing design in theory and real-world problems, whereas thoseassociated with design development (P3, P4, P5) focus onthe influences of members of the collaboration on each otherand on the design, and on the need for continuous evaluationthat takes such influences into account. Additional princi-ples (P6, P7) emphasize the importance of acknowledgingand responding to such influences throughout the design pro-cess, as well as the importance of generating usable designknowledge from specific research outcomes. While some ofthese principles (P1, P2, P5, P7) reflect those explicitlyarticulated in the visualization literature [1, 8, 15, 18, 20,23, 25], several of these principles (P3, P4, P6) provide newand potentially valuable guidance around the validation ofvisualization design research.

P1: Practice-inspired Research.The first principle emphasizes that applied research should

be motivated and inspired by real-world problems. This no-tion is analogous to what the visualization community hastermed problem-driven research. This approach helps to en-sure domain relevance and paves the way for in vivo evalu-ation — two core tenets of design studies [23].

P2: Theory-ingrained Artifact.The second principle stresses the importance of design the-

ory and domain theory in informing a design researcher’sunderstanding of the problem and solution space, and in

helping guide the design process. This principle serves thesame purpose as the learn stage in the design study method-ology [23], which emphasizes that researchers must learn thespace of visualization possibilities in order to design e↵ec-tively. Additionally, it relates to the nested model’s [20]encoding and interaction threats, which stress that theoryshould inform decisions at all levels of the design, therebyensuring that resulting artifacts are theory-ingrained. Visu-alization design studies frequently focus on the applicationof theory to inform and justify design.

P3: Reciprocal Shaping.Principle 3 emphasizes the constant shifting and shaping

of both the artifact and the design process by the di↵er-ent perspectives within the team. While this element ofdesign research may feel familiar and perhaps obvious tomembers of the visualization community, few attempts havebeen made to account for this element in existing visual-ization methodology [11]. Acknowledging the occurrence ofreciprocal shaping can increase reliability by: 1) providingan explicit opportunity to document impactful activities andinsights throughout the design process; and 2) revealing op-portunities for structured approaches to ensure and supportthe e↵ects of all people involved on the design process andresulting artifacts. We suspect that reciprocal shaping ismore prevalent in applied visualization research than in in-formation systems research due to the deeply collaborativeand highly iterative nature of visualization design and theinfluential nature of data-led discovery. Methods for en-suring and capturing reciprocal shaping within visualizationremain underdeveloped, and are potentially of high-interestfor the BELIV community.

P4: Mutually Influential Roles.Principle 4 emphasizes the learning and cross-fertilization

that occurs among ADR team members. Each member ofthe team brings a unique suite of knowledge, theory, andexpertise. Through close collaboration the team memberslearn about each other’s expertise, sometimes o↵ering valu-able insight into another member’s primary research domain.These insights can create substantial shifts in how anotherteam member thinks about, or approaches, his or her re-search or domain, which within the visualization communityis informally considered a sign of success for design studies.Beyond the citation of publications in an application do-main, or anecdotal stories [4, 13, 17], however, methods andmechanisms for reliably assessing and reflecting on mutualinfluence are underdeveloped.

P5: Authentic and Concurrent Evaluation.Principle 5 stresses that evaluation should happen through-

out the design process to both influence the process itself andinform design decisions. Importantly, this principle encour-ages researchers to prioritize authenticity when evaluatingartifacts and their e↵ects through methods that are ecolog-ically valid and conducted in-the-wild; this value is echoedin visualization design methodologies [15, 23, 25]. Authen-tic and reliable evaluation in the context of P3 and P4 —which value and encourage the influence of all members ofthe team on the shaping of the artifact and even in devel-oping the insights achieved across domains — however, isat odds with controlled studies that aim to establish predic-tive models through in vitro experiments that seek to remove

subjectivity. Instead, ADR provides a means of achieving re-liability within a subjective context by articulating the rolethat people and context have on the design process itself.Furthermore, ADR, and visualization design research moregenerally, provide an environment for: 1) the evaluation ofthe results of controlled studies; and 2) a real-world scenariothat may draw attention to the need for additional visual-ization research that may itself require controlled studies.

P6: Guided Emergence.Principle 6 encourages researchers to be aware of and sen-

sitive to the reciprocal shaping of theory-ingrained artifactsthat happens throughout the design process: to nurture andincorporate the shaping into the design process, and to cap-ture and apply it toward generating new design principles.While design research should be guided in part by theory, re-searchers should also be open to incorporating insights thatemerge from the research context, interactions, and evalua-tion. This principle can help ensure reliability of resultingartifacts by encouraging the explicit awareness, and docu-mentation, of the emergence itself, such as the evaluation ofdesign decisions and the evidence used to develop them.

P7: Generalized Outcomes.Principle 7, which recognizes the unique and highly spe-

cialized outcomes of a design process, also emphasizes theimportance of generalizing and abstracting research find-ings. This principle specifically encourages researchers togeneralize the problem and the solution, as well as to de-rive design principles. The visualization community sub-scribes to a similar process of generalizing research findingsfor the purpose of broader application. Abstractions, prob-lem characterizations, and guidelines are examples of suchgeneralizations [18, 20]. This principle, however, suggeststhat other kinds of learning, particularly surrounding thereciprocal shaping and mutually influential nature of the de-sign process, could also be formalized, benefitting the greaterresearch community.

2.3 ADR StagesADR is a high-level framework that encompasses many

of the details found in existing visualization process modelsand practices. Research begins with preliminary investiga-tion and articulation of the problem, continues with a periodof iterative and cyclic human-centered design and develop-ment, and ends with critical reflection and synthesis of re-search. ADR di↵ers from visualization design methodologiesin its focus on intervention as a critical element of the designprocess and its objective of learning through design. Equallyimportant, however, is the actionable framework that ADR’sstages provide for adhering to and reflecting on the princi-ples discussed in Section 2.2 to underpin the design processin ways that aim to achieve reliability. Tight cycles of ac-tion and evaluation are core to this and, unlike the emphasisin models of visualization design, reflection is required andongoing.

S1: Problem Formulation.The ADR process is triggered by a real-world domain

problem, either expressed by domain experts or discoveredby design researchers. The problem formulation stage in-volves the preliminary research and investigation of the prob-lem, including narrowing in on the research opportunity.

This stage also involves what ADR terms “casting the prob-lem as an instance of a class of problems”— similar to theinitial problem characterization and abstraction of visualiza-tion design research. This stage emphasizes the principle ofpractice-inspired research (P1), stressing the importance ofa real-world context for developing appropriate tasks as wellas for establishing an ecologically valid context for valida-tion of artifacts. The principle of theory-ingrained artifact(P2) is also stressed in this stage, indicating the importanceof a prepared mind for developing e↵ective solutions.

S2: Building, Intervention, Evaluation.The second stage of ADR is grounded in the core tenet of

action research that an e↵ective way to learn about some-thing is to try to change it [5]. In this stage, design re-searchers collaborate closely with domain practitioners, bothto continue to develop and refine the problem space, as wellas to design, develop, and refine the artifact. This is accom-plished via cycles of building, intervention, and evaluation(BIE). As these cycles progress, new interventions are de-signed based on the results from previous cycles, are evalu-ated in real time, and are used to inform subsequent cycles.Technology probes [6] are a common intervention instrumentused within design study research. Our own experiences ofconducting visualization design research suggest that BIEcycles occur frequently and at multiple scales, with overar-ching cycles exploring high-level questions, mid-level cyclesexploring core concepts surrounding the data abstractionand design of a visualization artifact, and low-level rapid,iterative feedback and informal evaluation cycles through-out. In Section 4.2, we illustrate the multi-scale nature ofBIE cycles and adapt the model proposed in Sein et al. [24].

The principles of reciprocal shaping (P3) and mutuallyinfluential roles (P4) emphasize the highly collaborative,messy, human-centered nature of BIE cycles, as well as theshifting nature of the problem being studied. These prin-ciples provide structure to incorporate these dynamic andunpredictable elements of applied research into the designprocess. This stage also emphasizes authentic and concur-rent evaluation (P5) as designers probe with technology tofind out what works, and what their design ideas reveal.Evaluation needs to be quick, as well as concurrent with thebuild and intervene activities.

S3: Reflection & Learning.The reflection and learning stage happens continuously

and in parallel with S1 and S2. In this stage, researchersare encouraged to reflect on: ongoing evaluation in orderto guide the design process; how well the research processadheres to guiding principles and how to encourage deeperadherence; and potential, broader implications of the re-search. This stage may occur either momentarily or in longerstretches, and is often triggered by an insight — a revelation,a moment of validation, or a design challenge — developedduring S1 or S2. While this stage has similar objectives tothe reflect stage in the nine-stage framework for design stud-ies [23], ADR is explicit about the repeated and central roleof reflection throughout the design process. Reflection andlearning is guided by one principle, guided emergence (P6),encouraging researchers to adhere to P2-5 throughout thedesign process and to reflect critically on the impact of suchprinciples on the design and on the greater contribution oftheir research.

S4: Formalization of Learning.The final stage of ADR is the formalization of learning.

This stage occurs once the BIE cycles are completed andbuilds on the reflection and learning conducted throughoutthe design process — casting the insights and artifacts to abroader class of problems and solutions. Stage 4 embracesthe generalization of outcomes (P7), pushing visualizationresearchers to think more broadly about the scope of theircontributions to provide guidance around generalizing andabstracting elements of the design process.

3. COMPARISON TO VIS MODELSAs described in Section 2, ADR marks an evolution of

thinking within the information systems community aboutthe role of design in research, and specifically about how tomake design research reliable and generalizable. The visual-ization community is engaged in a similar conversation, andhas put forth a number of models for structuring the designprocess [23, 15] and validating design decisions [7, 18, 20].In this section, we briefly discuss these models as they relateto ADR.

The nine-stage framework represents the first formalizedprocess model and methodology for conducting design stud-ies [23]. The framework is organized into three phases: thefirst describing a set of activities that should occur beforetriggering a design study project, the second describing thecore design activities in the production of visualization ar-tifacts, and the third describing analysis and reflection tomove design insights toward generalizable knowledge. Eval-uation is stated as a concurrent step across the entire nine-stage framework, but its specific role or guidance on whattypes of evaluation are appropriate is not discussed in detail.

The design activity framework (DAF) [15] was a responseto some of the shortcomings of the nine-stage framework.Specifically, the DAF emphasizes evaluation as a primarycomponent of each design activity within the design processwhile also o↵ering guidance for appropriate evaluation meth-ods. The DAF also attempts to give a more flexible structureto the design process by supporting iterative, nested, andparallel design activities. In an e↵ort to boost the action-ability of the framework, the DAF bridges between the stepsdesigners take and the decisions they make by explicatingthe levels of the nested model [18, 20] that are considered ineach design activity. The nested model describes four levelsof decisions — problem characterization, data and task ab-straction, visual encodings and interactions, and algorithms— and the interconnected relationship of decisions with re-spect to validation. The artifacts produced as a result ofthese decisions are often considered to be the contributionsmade by a design study [23].

ADR encompasses both the latter two stages of the nine-stage framework as well as the entire DAF by describing thedesign process from a problem trigger through formalizationof the knowledge acquired. Similar to the DAF, ADR hasan explicit treatment of evaluation as an essential step thatis repeatedly visited throughout all design activities. Unlikethe DAF, however, ADR makes reflection a primary activitythroughout the design process, extending the role of reflec-tion from that detailed in the nine-stage framework in a waythat mirrors action research cycles.

The biggest shift that ADR presents over existing visual-ization models, however, is the adoption of action research [10,16], impacting the design process in a number of ways: first,

the emphasis of the role of learning through planned actionsas a primary driver of the design process; second, the viewthat the development of an artifact is both a contributor toand consequence of the research process; and third, the fram-ing of the design process in a manner that achieves reliabilityby incorporating established values from social science.

4. APPLYING ADR TO POEMAGEWe now consider ADR in a visualization context by ap-

plying it retrospectively to a recent design study [13]. Thisdesign study involved a two-year collaboration between vi-sualization researchers (two of the co-authors of this work)and poetry scholars, resulting in various artifacts, including:new views on visualization design in poetry scholarship, in-sights into the role of computers in this discipline, a seriesof guidelines that can be applied to visualization in human-istic domains, and a new visualization tool called Poemage.In what follows, we present and critique the design study byreframing it through the lens of ADR. We argue this refram-ing sheds new light on contributions of the design study andillustrates the applicability of ADR to visualization designresearch.

4.1 Problem FormulationThe Poemage design study was triggered by the poetry

scholars’ interest in exploring the potential role of visualiza-tion in poetry scholarship, and in particular in the experi-ence of a close reading of a poem. Close reading involvesthe in-depth analysis of a poem and all its literary devices,and is central to the poetry scholars’ research. During theinitial portion of the problem formulation stage, the visual-ization researchers conducted informal and semistructuredinterviews with the poetry scholars to learn about the closereading process [7]. From these interviews, the visualizationresearchers discovered that influencing close reading couldhappen in many di↵erent ways, and that there was no ex-plicit notion of data in this context.

From there the visualization researchers dug through textanalysis literature to determine what types of literary de-vices — metaphor, imagery, a↵ect, and sound to name afew — can be extracted from a poem. The goal was to finda device that was both robustly computable and interestingto the poetry scholars. Eventually, the team narrowed in onsonic devices — a class of poetic device that utilizes soundand the relationships between sounds in words to e↵ect theinterpretation of a poem. Next, the team developed an ini-tial data abstraction; the data under consideration would besets of sonically similar words within a poem. Additionally,the visualization researchers reviewed literature around textvisualization, close reading, and digital humanities.

Adherence to P1 and P2.The investigation in this stage was grounded in the poetry

scholars’ interest in exploring the role of visualization in po-etry scholarship, which was continually revisited through thenumerous discussions among the team members. The theoryacquired within this stage came from various approaches totext analysis and visualization, and the values and method-ologies of digital humanities. In addition, learning aroundapproaches to digital humanities scholarship inspired the vi-sualization researchers to pursue a highly experimental andexploratory approach to the design process, which was main-tained throughout the design study. As the problem formu-

lation stage was revisited later in the design study, the visu-alization researchers turned once again to visualization the-ory, digging into specific visualization techniques that bestsupported the e↵ective encoding of the evolving data ab-straction for the tasks at hand.

4.2 Building, Intervention, EvaluationDuring the BIE stage, the team developed a broad ar-

ray of technology probes to understand three core questions:What sonic devices are interesting to the poetry scholars?What are the scholars interested in doing with the sonic de-vices? How can visualization support their exploration ofthe sonic devices? Overarching to these questions was thelarger investigation into the role visualization could play inpoetry scholarship. Sein et al. [24] describe small numbersof BIE cycles relating to beta and alpha prototypes of soft-ware systems in their examples of ADR. On reflection, ourdesign study consisted of a series of BIE cycles that oc-curred at multiple scales: one high-level, overarching BIEcycle examining the role and impact of technology on po-etry scholarship; three mid-level cycles focusing on sound,visualization design, and the development of the Poemagetool; and many rapid, low-level cycles of iteration, expan-sion, and refinement, each involving a planned and activeintervention, evaluation of e↵ect on the poetry scholars, andsubsequent reflection to establish knowledge gained and todrive design decisions. Each scale warranted di↵erent typesof evaluation, with the higher level scales incorporating moreformal evaluation, and the lower level scales using quicker,lighter weight methods. These low-level BIE cycles are ofparticular interest for future investigations into reliabilityvia documentation and recording of evaluation e↵orts. Fur-thermore, the di↵erent scales of BIE cycles may be partic-ularly important in visualization as it is so often driven bydiscovery.

Figure 2: Multi-scale BIE cycles: (a) high-level BIEcycle focusing on the role of technology in poetryscholarship, (b) mid-level BIE cycles focusing onsound, vis design, and the development of Poemage,(c) low-level cycles involving fast, informal feedback.

The multi-scale BIE cycles for the Poemage design studyare roughly depicted in Figure 2. The three horizontal linesreflect contributions from di↵erent members of the Poemageteam — the visualization researchers; the poetry scholars,or practitioners in ADR parlance; and end-users beyond theteam. The top line relates to development on the part ofthe visualization researchers as they produced functionalityfor intervening in the practices of the poetry scholars. Themiddle line indicates an intervention as the developed arti-fact was deployed to the scholars. We should emphasize that

this is the crucible of action — where the ADR team, de-sign, and data interact in an authentic setting, and where aplausible, theory-ingrained artifact is used by a practitionerto establish knowledge in both the application and visual-ization domains. Further development, and reflection andlearning, result from evaluation of these planned actions.The bottom line corresponds to the deployment of the Po-emage tool to users beyond the team. Lam et al. [8] describethis as “deployment ... in the field,” which o↵ers opportu-nities for summative evaluation, as is described in multi-dimensional in-depth long-term case studies (MILCs) [25].In what follows, we outline the three mid-level BIE cyclesthat were core to the Poemage design study.

The first mid-level BIE cycle focused on sound and sonicdevices. Via an informal survey followed by semistructuredinterviews, the visualization researchers worked with the po-etry scholars to determine which sonic devices would be mostinteresting to explore in the close reading of a poem. Theidentified sonic devices were translated to code within aninteractive system that extracted sets of words in a poemthat were related via the various devices. The visualiza-tion researchers used this software as a technology probe totest the selected devices, and to understand how the po-etry scholars might explore such devices within a poem.Evaluation of the technology probe ranged from casual feed-back to highly structured interviews. Insights from the ini-tial technology probe motivated the visualization expertsto develop a language along with a formalism for specify-ing and analyzing a broad range of sonic devices — all ofwhich the poetry scholars blanketed under an extended def-inition of rhyme — within a poem. This language and for-malism was subsequently implemented in a system calledRhymeDesign [14]. Evaluation for RhymeDesign was for-mal, including both case studies and a survey testing theexpressivity of the RhymeDesign language against examplesof interesting sonic devices collected from an extended net-work of poetry scholars.

The second mid-level BIE cycle focused on the design ofvisual representations. This cycle included explorations ofdi↵erent visual representations of the data abstraction aswell as experimentation with di↵erent visualization and in-teraction techniques to support the exploratory tasks ob-served and identified in the previous BIE cycles. As thesecond BIE cycle progressed through a series of rapid, high-frequency interventions, the poetry scholars’ interest evolvedfrom browsing through sets of words detected by the systemto instead exploring the interaction between these sets acrossthe space of the poem. This new focus inspired the team torevisit a metaphor relating sound in poetry to flow devel-oped previously by one of the poetry scholars, which in turninformed the visual notion of sonic topology. This cycle wasguided by regular, rapid, and informal feedback from thepoetry scholars on ideas and prototypes — sketches, screencaptures, live demos, etc. — shared in person or remotely.

The third and final mid-level BIE cycle focused on thedevelopment of the Poemage visualization tool. During thiscycle, valuable features, interactions, capabilities, and de-sign elements were extracted from previous BIE cycles andcompiled into a multi-linked view system. Following an ini-tial beta-testing deployment period in which poetry scholarsfrom an extended network were given several weeks to exper-iment with incorporating Poemage into their practices, thevisualization researchers conducted contextual interviews and

case studies. In preparation for these focused evaluation ses-sions, the poetry scholars wrote experiential, qualitative nar-ratives about their experiences using Poemage, which theydiscussed during the interviews. Although this preparatorywriting was not asked of the poetry scholars, they expressedthat this was a natural and productive method of reflec-tion within their field, and an exercise they were inclinedto complete regardless. In reflection, we note that insightslike these could point to new forms of evaluation for othersworking in the digital humanities.

Adherence to P3, P4, and P5.In reflecting on the BIE cycles, we found that reciprocal

shaping (P3) often occurred during close collaboration be-tween the visualization researchers and poetry scholars, thatevaluation (P5) occurred rapidly and informally during pe-riods of intervention, and that the cycles supported mutualinfluence (P4) by creating a gradual decrease in separationbetween the knowledge states and the roles of the researchersand scholars.

One specific example of reciprocal shaping occurred aroundthe development of a particular feature in Poemage, whichcame to be known as the beautiful mess. The beautifulmess, shown in Figure 3, displays all the detected sets for agiven poem, resulting in visual clutter and significant over-plotting. Although this feature was explicitly requested bythe poetry scholars, it was met with a degree of resistance bythe visualization researchers as it contravened visualizationconventions that value clarity and readability [2]. The po-etry scholars argued, however, that the messiness resonateddeeply with them as it captured the energy and excitementthey felt during a close reading, as well as serving as a visualrepresentation of the untangling task they confront with anew poem. Ultimately, the inclusion of the beautiful messnot only led to one of the more important insights of thework, but also helped to engage and gain the trust of thepoetry scholars.

The reciprocal shaping of the beautiful mess contradictsthe visualization theory brought to bear on the design pro-cess (P2). The design, however, worked for the poetry schol-ars, emphasizing the importance of reciprocal shaping andmutual influence. The beautiful mess was considered to be astrange anomaly of the design study, and precisely what tomake of it remained unclear to the visualization researchers.An understanding of the notion of reciprocal shaping alongwith structured guidance to embrace and nurture this ele-ment of the design process and its contribution to designand knowledge acquisition, such as that provided by ADR,may have resulted in more features like the beautiful mess,and in more directed learning and evaluation around suchfeatures.

Mutual influence also played a significant role in this re-search. Each team member contributed a di↵erent level ofexpertise in her own field, a di↵erent level of expertise in theother domain, and a di↵erent level of openness to deviatingfrom theory and convention. Throughout the design pro-cess, the poetry scholars developed a computational way ofthinking about their scholarship, which they discussed andreflected on in multiple articles and talks to the humani-ties and DH communities [3, 9]. This, along with insightsgained throughout the collaboration, led one collaboratorto develop new theoretical thinking about the relationshipbetween human and machine in the context of the digital

Figure 3: The beautiful mess feature of Poemageapplied to Clark Coolidge’s “Machinations Calcite.”

Development of this feature exhibited elements ofreciprocal shaping (P3) and guided emergence (P6).

humanities. On the other end of the collaboration, the vi-sualization researchers learned to embrace the poets’ broadand imprecise definition of rhyme and developed an open-ness to deviating from conventional visualization methodsand principles. Additionally, the visualization researcherslearned to incorporate a more extemporaneous element intotheir research — one that reflected the nature of their collab-orators’ poetry scholarship. Furthermore, revisiting a closereading of a particular poem, and the particular analysisthat led to a new interpretation or insight, was a regular tac-tic used by the poetry scholars to illustrate a point. Thusthe visualization researchers had to develop enough of anunderstanding of poetry, poetry analysis, and close readingin order to interpret the point being made, and translate itto the space of visualization research.

Lastly, authentic evaluation played an integral role in shap-ing the research and design process. Fast and informal feed-back guided the research team toward pursuing sonic de-vices and facilitated the design process. At various pointsthroughout the first and second BIE cycles, the visualiza-tion researchers sat with the poetry scholars and iterativelytested and evaluated new features, interactions, and visualencodings. In addition, consistent feedback helped the vi-sualization researchers identify and build on elements of theresearch process that engaged the poetry scholars, increasedtheir trust in the technology, and were disruptive in someinteresting sort of way. In retrospect, recording and report-ing these kinds of findings in a more structured and perhapscomprehensive fashion, as ADR begins to facilitate, wouldhave increased the reliability of the design process.

The evaluation strategies, particularly as they applied tointerviewing techniques, evolved and shifted throughout theBIE cycles based on feedback and reflection. For exam-ple, the first round of interviews was highly structured, butit became clear that semistructured interviews were muchmore appropriate since the poetry scholars needed very lit-tle prompting and came to the interviews with valuable in-sights that would have been hard to elicit via preconceivedquestions. As another example, elements of poetry scholar-ship found their way into evaluation tactics. The primaryexample of this was the experiential, qualitative narrativeswritten by the poetry scholars that were incorporated intothe evaluation of Poemage. Thus, mutual influence and re-ciprocal shaping had an e↵ect on evaluation as well as ondesign. This is not something that would be welcome inthe kinds of isolated objective evaluation that lab studies

permit, but a reflective methodology such as ADR providesa means for such flexibility while providing reassurance re-garding reliability in applied work. Had this type of mutualinfluence and reciprocal shaping occurred earlier in the de-sign study, or had the team been following a methodologythat explicitly encouraged this awareness and flexibility, theteam may have sought and benefitted from more opportu-nities of this kind.

4.3 Reflection & LearningThroughout the design process, the visualization researchers

reflected in order to shift and shape the direction of theproject, operationalize poorly defined tasks, and extract in-sights. For example, during the first BIE cycle it becameclear that the poetry scholars embraced a broad and impre-cise definition of sonic similarity, motivating the visualiza-tion researchers to move beyond straightforward rhyme de-tection. The result was the development of a formalism fordescribing sonic similarity computationally, and the imple-mentation of the RhymeDesign tool. Another reflective mo-ment occurred when the visualization researchers observeda spectrum of ways in which the poetry scholars were usingthe Poemage tool, leading to a realization that one role oftechnology in poetry scholarship is for creativity support, asopposed to data analysis.

Adherence to P6.Moments of guided emergence occurred throughout the

design process. An illustration of this is the beautiful messexample described in section 4.2. The visualization researcherswere guided by conventions surrounding clarity and read-ability, and initially resisted even experimenting with thefeature. As the poetry scholars continued to push for thefeature, however, one of the visualization researchers be-came more receptive. As mutual influence was establishedthrough validation of the technique, the other visualiza-tion researcher was eventually persuaded. In hindsight, thisexperience taught the visualization researchers to be moreopen to precisely the notion of guided emergence. At thetime, the precise impact and takeaway of this anecdote re-mained unclear to the visualization researchers; however, thelesson was presented in the visualization publication aboutthe Poemage design study as a kind of guideline that encour-aged others to adopt the same openness in their research.P6 directly confirms and articulates the importance of suchexperiences to the design process, and gives weight to anyassociated lessons and formulated guidelines.

4.4 Formalization of LearningFormalization of learning in this project occurred, by and

large, during the writing phase of the research. During thisperiod, the visualization researchers looked back through theentire project, gathering and formalizing the elements of theproject that had potential for benefiting the visualizationcommunity as a whole. Some formalization came out of theproblem characterization and data abstraction, as is typicalin the reporting of design work in visualization research [19].Other formalization came out of reflecting on the project asa whole, including insights surrounding creativity supporttools and conducting design research in the digital human-ities. Additionally, the visualization researchers revisitedthe most interesting challenges encountered throughout theresearch — especially those surrounding evaluation and ap-

propriate measures of success — and formalized them intoopen research questions for future work. While there was adesire to formalize learning surrounding the reciprocal shap-ing and the disruption that occurred throughout the designprocess, the lack of guidance and language for doing this inexisting methods for visualization design left the visualiza-tion researchers with little confidence in such an endeavor.

Adherence to P7.While the results of this research were highly specific and

designed to meet the interests and needs of a very smallgroup of poetry scholars, the visualization researchers gen-eralized elements of the process and design to various levelsof abstraction. For example, the poetry scholars’ interestin exploring the role of technology in their scholarship prac-tices motivated some speculation about possible implicationsin the arts and in other fields that value novel interpreta-tions and creative thinking. At a much lower level, whilePoemage was designed to support a very specific researchactivity — the close reading of American English free versepoetry — formalizing the data abstraction allowed the vi-sualization researchers to speculate about possible applica-tions to other set visualization problems. In retrospect, wewonder whether taking an ADR approach might have facili-tated framing these outcomes more e↵ectively, consistently,and ultimately more reliably.

5. DISCUSSIONIn applied visualization research, visualization systems are

often shaped by a series of small decisions made by design-ers and researchers who have invested heavily throughoutthe design process. These decisions are typically made withestablished visualization principles in mind, and rely on fast,informal, and light-weight evaluation strategies. How to con-duct such evaluation in a manner that is reliable and thatcontributes to our knowledge of visualization as a methodol-ogy and to the domains in which we work is an open questionwithin the visualization community.

In this work we look to information systems research forinsight into conducting reliable, informed evaluation in set-tings that are applied and dynamic. In particular, we turn toaction design research, which through adherence to a set ofguiding principles o↵ers a framework for reliably structuringand reporting on the design process in ways that can con-tribute to the acquisition of knowledge. ADR shares manycommonalities with existing visualization design methodolo-gies, but deepens the theoretical underpinnings through itsuse of action research as a basis for design research. Thisfoundation — adapted from an established method of en-quiry in social science, in which researchers directly influencethe context they study through planned intervention — af-fords a new perspective on the forces that shape the natureof visualization design, and on the way we define reliabilityof research.

We describe the ADR principles and stages using visual-ization parlance, highlighting the similarities and overlapswith existing visualization theory and practice. However,we are also able to point out a number of places where ADRprinciples and stages are not reflected in visualization mod-els, indicating gaps that present reliability threats for visu-alization design research. Furthermore, applying ADR ret-rospectively to the Poemage design study revealed a numberof highly significant moments and insights that we struggled

to articulate using existing models for conducting and re-porting visualization research. ADR provided structure andorganization for analyzing the impact of the human-centeredand disruptive elements on the process and design, as well asthe impact of the collaboration on the learning that occurredin both domains — visualization and poetry. This learninghas extended beyond the scope of the project and continuesto influence research in both domains. While we are un-able to draw conclusions around ADR’s utility as a guidingmethodology, our retrospective application of the frameworkleads us to hypothesize that incorporating elements of ADRinto future design studies will enable better navigation andevaluation of the design process, as well as the facilitationof new kinds of learning. Furthermore, while we acknowl-edge the plausible risk of confirmation bias associated with aretrospective application of ADR, our collective experiencewith Poemage and other past design studies is precisely whatmotivated us to seek the added guidance o↵ered by ADR.

ADR emphasizes the di↵erences between the methods usedto frame controlled in vitro research and those applicable todesign-oriented research where reciprocal shaping and mu-tual influence occurs. This contrast, however, also points tothe synergy between subjective and objective research en-deavors: applied visualization contexts o↵er an environmentto evaluate the results of controlled studies while simultane-ously providing inspiration for new research questions thatcould benefit from empirical experiments.

6. FUTURE WORKThe emphasis on the use of design as a research tool is

largely implicit in visualization research. Furthermore, theuse of design as a deliberate means of disruption to observee↵ect is an exciting approach through which visualizationresearchers may beneficially engage with a broad range ofapplication areas. In addition to these potential benefits, wesee major implications as well as several open questions sur-rounding the application of ADR to visualization research.

First, ADR explicates the role of reciprocal shaping (P3),mutual influence (P4), and guided emergence (P6), none ofwhich are captured explicitly in existing visualization mod-els. These principles stem from the application of actionresearch, which could provide guidance for how to applythese principles in a structured, reliable way. Furthermore,visualization design research is in need of mechanisms forcapturing and reporting on moments guided by these prin-ciples, and well as for reflecting on them to produce newvisualization knowledge.

Second, the subjective nature of ADR questions the ap-plicability of P7 itself — what does it mean to general-ize results in the context of such a subjective, uncontrolled,and specific scenario? Others have suggested that for designstudies, the goal is “transferability not reproducibility” [23],and we believe that the notion of transferability is also im-portant for ADR. How do we define reliable, transferable re-sults for visualization design research, and what mechanismscan we develop to support transferability? These questionsare core components of what we believe to be interestingand exciting future work.

AcknowledgementsThis work was funded in part by NSF grant IIS-1350896.

7. REFERENCES[1] K. Andrews. Evaluation comes in many guises. In AVI

Workshop on BEyond time and errors (BELIV)Position Paper, pages 7–8, 2008.

[2] W. S. Cleveland. A model for studying displaymethods of statistical graphics. Journal ofComputational and Graphical Statistics, 2(4):323–343,1993.

[3] K. Coles. Slippage, spillage, pillage, bliss: Closereading, uncertainty, and machines. WesterHumanities Review, pages 39–65, Fall 2014.

[4] S. Goodwin, J. Dykes, S. Jones, I. Dillingham,G. Dove, A. Du↵y, A. Kachkaev, A. Slingsby, andJ. Wood. Creative user-centered visualization designfor energy analysts and modelers. Visualization andComputer Graphics, IEEE Transactions on,19(12):2516–2525, 2013.

[5] G. R. Hayes. The relationship of action research tohuman-computer interaction. ACM Transactions onComputer-Human Interaction (TOCHI), 18(3):15,2011.

[6] H. Hutchinson, W. Mackay, B. Westerlund, B. B.Bederson, A. Druin, C. Plaisant, M. Beaudouin-Lafon,S. Conversy, H. Evans, H. Hansen, et al. Technologyprobes: inspiring design for and with families. InProceedings of the SIGCHI conference on Humanfactors in computing systems, pages 17–24. ACM,2003.

[7] P. Isenberg, T. Zuk, C. Collins, and S. Carpendale.Grounded evaluation of information visualizations. InProceedings of the 2008 Workshop on BEyond timeand errors: novel evaLuation methods for InformationVisualization, page 6. ACM, 2008.

[8] H. Lam, E. Bertini, P. Isenberg, C. Plaisant, andS. Carpendale. Seven guiding scenarios for informationvisualization evaluation. 2011.

[9] J. Lein. Sounding the surfaces: Computers, context,and poetic consequence. Wester Humanities Review,pages 84–109, Fall 2014.

[10] K. Lewin. Action research and minority problems.Journal of social issues, 2(4):34–46, 1946.

[11] D. Lloyd and J. Dykes. Human-centered approaches ingeovisualization design: Investigating multiplemethods through a long-term case study. IEEE Trans.Vis. Comp. Graphics, 17(12):2498–2507, 2011.

[12] S. T. March and G. F. Smith. Design and naturalscience research on information technology. Decisionsupport systems, 15(4):251–266, 1995.

[13] N. McCurdy, J. Lein, K. Coles, and M. Meyer.Poemage: Visualizing the sonic topology of a poem.Visualization and Computer Graphics, IEEETransactions on, 22(1):439–448, 2016.

[14] N. McCurdy, V. Srikumar, and M. Meyer.Rhymedesign: A tool for analyzing sonic devices inpoetry. on Computational Linguistics for Literature,page 12, 2015.

[15] S. McKenna, D. C. Mazur, J. Agutter, and M. Meyer.Design activity framework for visualization design.Visualization and Computer Graphics, IEEETransactions on, 20(12):2191–2200, 2014.

[16] J. McNi↵. Action research: Principles and practice.Routledge, 2013.

[17] M. Meyer, T. Munzner, and H. Pfister. Mizbee: amultiscale synteny browser. Visualization andComputer Graphics, IEEE Transactions on,15(6):897–904, 2009.

[18] M. Meyer, M. Sedlmair, P. S. Quinan, andT. Munzner. The nested blocks and guidelines model.Information Visualization, 2013.

[19] T. Munzner. Process and pitfalls in writinginformation visualization research papers. InInformation visualization, pages 134–153. Springer,2008.

[20] T. Munzner. A nested model for visualization designand validation. Visualization and Computer Graphics,IEEE Transactions on, 15(6):921–928, 2009.

[21] W. J. Orlikowski and C. S. Iacono. Researchcommentary: Desperately seeking the “it” in itresearch—a call to theorizing the it artifact.Information systems research, 12(2):121–134, 2001.

[22] S. Purao, O. Henfridsson, M. Rossi, and M. Sein.Ensemble artifacts: From viewing to designing inaction design research. Systems, Signs & Actions,7(1):73–81, 2013.

[23] M. Sedlmair, M. Meyer, and T. Munzner. Designstudy methodology: Reflections from the trenches andthe stacks. Visualization and Computer Graphics,IEEE Transactions on, 18(12):2431–2440, 2012.

[24] M. Sein, O. Henfridsson, S. Purao, M. Rossi, andR. Lindgren. Action design research. MIS Qarterly,35(1):37–56, 2011.

[25] B. Shneiderman and C. Plaisant. Strategies forevaluating information visualization tools:multi-dimensional in-depth long-term case studies. InProceedings of the 2006 AVI workshop on BEyondtime and errors: novel evaluation methods forinformation visualization, pages 1–7. ACM, 2006.

[26] R. H. von Alan, S. T. March, J. Park, and S. Ram.Design science in information systems research. MISquarterly, 28(1):75–105, 2004.

[27] C. J. Zimmerman, H. T. Wessels, and R. Vatrapu.Building a social newsroom: Visual analytics for socialbusiness intelligence. In Enterprise Distributed ObjectComputing Workshop (EDOCW), 2015 IEEE 19thInternational, pages 160–163. IEEE, 2015.