News Production Workflows in Data- driven, Algorithmic Journalism: A Systematic Literature Review

MEDIA & DESIGN

News Production Workflows in Data-driven, Algorithmic Journalism

A Systematic Literature Review

Julian Ausserhofer1,2 ,@julauss

Robert Gutounig1 &

@sextus_empiricoMichael Oppermann2

@oppermann_m

1: FH Joanneum University of Applied Sciences Graz

2: University of Vienna

Dubrovnik Media Days, 31.10.15

#DuMD2015 & #DUMediaDays

MEDIA & DESIGN

*Tool evaluation partners

**

*

Supported by:

MEDIA & DESIGN

Research Interest

MEDIA & DESIGN

#DuMD2015 & #DUMediaDays @julauss, @sextus_empirico & @oppermann_m 4

"experimental use of algorithms, data and social science methods" in journalism (Gynnild, 2014, p. 715)

MEDIA & DESIGN

@julauss, @sextus_empirico & @oppermann_m 5

(Cox, 2000)

http

://ww

w.co

mp

uterh

istory.o

rg/revolu

tion

/early-com

pu

ter-com

pan

ies/5/102/445

(Meyer, 1973)

1952


Computational methods in journalism

http://www.computerhistory.org/revolution/early-computer-companies/5/102/445

http://www.computerhistory.org/revolution/early-computer-companies/5/102/445

MEDIA & DESIGN


(Garrison, 1996)


MEDIA & DESIGN

The datafication of journalism


"datafication" of society (Mayer-Schönberger & Cukier, 2013)

"increased focus on measurement, outcomes assessment" (Anderson, 2015, p. 363)

"journalism's quantitative turn" (Coddington, 2015)

… has also affected the "infrastructures of journalism" (Russ-Mohl, 2006, p.199)


MEDIA & DESIGN

Research literature on data journalism


"internalist tendencies at [... the] early stage of academic research" (Anderson, 2013, p. 1007)

↓

"an explosion in data journalism-oriented scholarship" (Fink & Anderson, 2015, p. 476)*

"rapidly growing body" of scientific studies (Lewis, 2015, p. 322)*

*cited via (Loosen, Reimer & Schmidt, 2015, p. 2)


MEDIA & DESIGN


In what way have journalistic workflows and routines changed due to the broad introduction of data in the newsroom?

How has this impacted journalistic norms and ethics, and the skill requirements for journalists?


Research questions

MEDIA & DESIGN

Method

MEDIA & DESIGN

#DuMD2015 & #DUMediaDays @julauss, @sextus_empirico & @oppermann_m 11

Systematic literature review

MEDIA & DESIGN


"to develop insights, critical reflections, future

research paths and research questions" (Massaro, Dumay & Guthrie, forthcoming)

It adopts "a replicable, scientific and transparent

process [...] that aims to minimize bias [...]"(Tranfield, Denyer & Smart, 2003)


Why a systematic literature review?

MEDIA & DESIGN

Undertaking a systematic literature review

13

Adapted from Massaro et al. (forthcoming)Own visualization

@julauss, @sextus_empirico & @oppermann_m#DuMD2015 & #DUMediaDays

Writing a literature review protocol

Developing insights and critique through analyzing the dataset

Developing future research paths and questions

Determining the type of studies and carrying out a comprehensive literature search

Coding dataDefining the questions that the literature review should answer

MEDIA & DESIGN

14



Coding data



Defining the questions that the literature review should answer

In what way have journalistic workflows and routines changed due to the broad introduction of data in the newsroom?

How has this impacted journalistic norms and ethics, and the skill requirements for journalists?

MEDIA & DESIGN

15



Coding data




● Empirical research on DDJ● Social science focus, but open to other disciplines● Published after 1995

Included

Journal articles

Book sections

Conference papers

Reports (from industry and research projects)

PhD theses

Not included

Bachelor's and Master's theses

Press reports

Blog posts

MEDIA & DESIGN

16



Coding data




● Preliminary search with “data-driven journalism”

● Extracting related terms from the keyword section of research papers

MEDIA & DESIGN

17



Coding data




Search terms

algorithmic journalism

computational journalism

computer-assisted reporting

data journalism

data-driven journalism

data-driven reporting

database journalism

datajournalism

datenjournalismus

quantitative journalism

No search terms

accountability journalism

crowdsourced journalism

dataviz

datavis

ddj

drone journalism

investigative journalism

online journalism

open journalism

MEDIA & DESIGN

18



Coding data




Scientific Databases

ACM Digital Sowiport

EBSCO Springer

IEEE SpringerLink

JSTOR Taylor & Francis Online

ProQuest Web of Science

Science Direct Wiley

Scopus Google Scholar

Sociological Abstracts

MEDIA & DESIGN

19



Coding data




772 search results↓Assessment of title, abstract & keywords

- by two independently working researchers(Thomas et al., 2004)

↓33 research publications

MEDIA & DESIGN

20





Coding data


qualitative coding

computational analysis

MEDIA & DESIGN

Results

MEDIA & DESIGN

Literature Corpus

MEDIA & DESIGN

23

Research literature on workflows in DDJDevelopment of literature over time

n=33


MEDIA & DESIGN

24@julauss, @sextus_empirico & @oppermann_m#DuMD2015 & #DUMediaDays

Research literature on workflows in DDJPublication type

Parasie & Dagiral, 2013

Lewis &Usher, 2013

Diakopolus, 2015

Karlsen &Stavelin, 2014

n=33node size = number of citations in Google Scholar

MEDIA & DESIGN


Research literature on workflows in DDJPublication year of referenced papers1787-2015

n=1151

Constitution of the United States

MEDIA & DESIGN


Research literature on workflows in DDJPublication year of referenced papers1999-2015

n=1004

2013: 176 cited papers

MEDIA & DESIGN


Research literature on workflows in DDJMost-cited references

MEDIA & DESIGN


edges = 165corpus publications, n=33

references, n = 70 size = nr. of citations from the corpus

Research literature on workflows in DDJCorpus and most-cited references

The Journalist as Programmer

Precision JournalismAccountability through Algorithm

MEDIA & DESIGN

Nr. of studies

Survey 3

In-depth interviews 22

(Particpant) observation 4

Method(s) of data collection


MEDIA & DESIGN

Geographical scope


Nr. of studies Nr. of studies

United States 9 Multinational (>4) 2

United Kingdom 9 Norway 1

Germany 4 Netherlands 1

Sweden 2 Argentina 1

Switzerland 2 Austria 1

Finland 2 Belgium 1

MEDIA & DESIGN

Qualitative analysis -News Production Workflow

MEDIA & DESIGN

News production workflowCollecting

Collecting digitalized information from

● publicly available data sets ● requested data● own research or data collected through crowdsourcing ● automated research (scraping)● leaked data

(17; Loosen, 2015)


MEDIA & DESIGN

Type of data source

● government or public bureau● private corporation ● other organizations like NGOs, research institutes

(28, Loosen, 2015)



MEDIA & DESIGN

Access to informationin Norway and USA generally considered good (due to freedom of information legislation)

Barriers to easy data access● fees ● lacking competencies for exporting data at public offices● the form data is often delivered (e.g. as PDF)● lack of legal legal resources to get access to data that is denied

to journalists in the first place(6; 14)



MEDIA & DESIGN

Selecting dataBecause data journalists don’t have much time in the newsroom, they prefer easy-accessible data, e.g. datasets from public bodies.

They are very cautious when it comes to work with data from companies or private sources.

(14)



MEDIA & DESIGN

Methods● Journalists use statistical programs (e.g. MySQL, Excel, Access,

SPSS, Google Docs, Maps, Google Refine, Google Fusion Tables etc.)

● Statistical methods and packages including cluster, network & regression analysis

--> Journalists are quite cautious with it, since they consider this outside of their field of expertise.

(6, 14)


News production workflowAnalyzing

MEDIA & DESIGN

Results of DDJ research depend in increasing manner on (graphical) display of (automated) data analysis software

Tools and programming languages● Developers use Python, Ruby on Rails, JavaScript, HTML● Used third-party software includes MySQL, Access, Excel,

Caspio, Tableau, Arcview, TimelineSetter etc.(14, 17, 28)


News production workflowBuilding and visualizing

MEDIA & DESIGN

DDJ is connected to a special way of representation which consists of ● displaying key messages as graphics and/or ● building interactive web applications

Storytelling is done through(1) interactive web applications with continuous update of data (2) interactive application based on a stable data set (3) continuously updated info graphic or (4) static infographics

(17)


News production workflowBuilding and visualizing

MEDIA & DESIGN

● Working alone at small and medium-sized newspapers, and

part of teams at larger news organizations

● New connections between journalists and programmers

● Members of teams usually have complementary backgrounds(6, 9, 14, 18)


News production workflowCollaboration

MEDIA & DESIGN

● Rise of "annotative journalism"

● Regarding the collaboration between newsroom and ICT-

department in media corporations data journalists perceive

the relation as troublesome.

Rather data journalists try to bypass ICT department when

installing extra software or setting up databases or servers(6, 28)


News production workflowCollaboration

MEDIA & DESIGN

● The goal of DDJ activities is by definition the [very accurate]

journalistic publication

● Publication of data sources and raw data following open data

principles is mentioned too as an integral part of DDJ. (6, 17, Coddington, 2014)


News production workflowPublishing the product

MEDIA & DESIGN

Backgrounds of data journalists

MEDIA & DESIGN

Backgrounds of data journalists● Data journalists are appx. 5 years younger than the average

age of German journalists

● All of them have started a university program, 80% have also finished it

● There is no generalizable “data journalism” career path. Background is mostly either social science or informatics.

● Backgrounds include programming, design, typography, info graphics, usability, databases, Web and journalism

● USA: dominance of National Institute for Computer-Assisted Reporting (NICAR)

(14, 17, 18; 6)


MEDIA & DESIGN

Enabling and Hindering Factors

MEDIA & DESIGN

Enabling and indering factors

Critical factors● Skills of the involved journalists● Support networks (internal and external) important for the

development of skillsavailability of data ● Tools● Legal resources● Available resources

(6, 14, 17)


MEDIA & DESIGN

Conclusion

MEDIA & DESIGN

@julauss, @sextus_empirico & @oppermann_m 47#DuMD2015 & #DUMediaDays

● New kind of people arriving in the newsroom● New skills needed in the newsroom● Increase of collaborative newswork● Shift to data-driven approaches (and

epistemologies debates connected to that)

MEDIA & DESIGN


Scope:

● Practices outside of Europe and North America● The construction of "facts"● Gender aspects

Methods of data collection:

● Survey ● (Participant) observation● Content analysis● Digital methods


Research gaps

MEDIA & DESIGN

49

Explore the (preliminary) corpus online – it's a protoype: http://tinyurl.com/valid-survey

http://validproject.at

Julian Ausserhofer1,2 ,@[email protected]

Robert Gutounig1 &

@[email protected]

Michael Oppermann2

@[email protected]





http://tinyurl.com/valid-survey




http://twitter.com/julauss


http://twitter.com/sextus_empirico


http://twitter.com/oppermann_m


MEDIA & DESIGN

Tools & Literature

50

MEDIA & DESIGN

51

NVIVO [Computer Software]. (2015). Retrieved from http://www.qsrinternational.com/

Bostock, M., Ogievetsky, V., & Heer, J. (2011). D³ data-driven documents.Visualization and Computer Graphics, IEEE Transactions on, 17(12), 2301-2309.

Lopez, P. (2009). GROBID: Combining automatic bibliographic data recognition and term extraction for scholarship publications. In Research and Advanced Technology for Digital Libraries (pp. 473-474). Berlin: Springer.

Zotero [Computer Software]. (2015). Retrieved from http://zotero.org/

Tools & software libraries


MEDIA & DESIGN

52

Anderson, C. W. (2013). Towards a Sociology of Computational and Algorithmic Journalism. New Media & Society, 15(7), 1005–1021. http://doi.org/10.1177/1461444812465137

Anderson, C. W. (2015). Between the Unique and the Pattern: Historical Tensions in Our Understanding of Quantitative Journalism. Digital Journalism, 3(3), 349–363. http://doi.org/10.1080/21670811.2014.976407

Cox, M. (2000). The Development of Computer-assisted Reporting. Presented at the Newspaper Division, Association for Education in Journalism and Mass Communication, Southeast Colloquium, Chapel Hill, North Carolina.

Fink, K., & Anderson, C. W. (2015). Data Journalism in the United States: Beyond the "Usual Suspects". Journalism Studies, 16(4), 467–481. http://doi.org/10.1080/1461670X.2014.939852

Garrison, B. (1996, September). Computer-Assisted Reporting: A National Use Update. Presented at the National Society of Professional Journalists Meeting, Washington, D.C.

Lewis, S. C. (2015). Journalism In An Era Of Big Data. Digital Journalism, 3(3), 321–330. http://doi.org/10.1080/21670811.2014.976399

Loosen, W., Reimer, J., & Schmidt, F. (2015). When Data Become News: A Content Analysis of Data Journalism Pieces. Presented at the The Future of Journalism 2015 Conference, Cardiff.

Mayer-Schönberger, V., & Cukier, K. (2013). Big Data: A Revolution That Will Transform How We Live, Work, and Think. Boston: Houghton Mifflin Harcourt.

Primary Sources


MEDIA & DESIGN

53

Massaro, M., Dumay, J. C., & Guthrie, J. (forthcoming). On the Shoulders of Giants: Undertaking a Structured Literature Review in Accounting. Accounting, Auditing & Accountability Journal.

Meyer, P. (1973). Precision Journalism: A Reporter’s Introduction to Social Science Methods (2nd ed.). Oxford: Rowman & Littlefield.

Russ-Mohl, S. (2006). The Economics of Journalism and the Challenge to Improve Journalism Quality: A Research Manifesto. Studies in Communication Sciences, 6(2), 189–208.

Thomas, J., Harden, A., Oakley, A., Oliver, S., Sutcliffe, K., Rees, R., … Kavanagh, J. (2004). Integrating Qualitative Research with Trials in Systematic Reviews. BMJ : British Medical Journal, 328(7446), 1010–1012. http://doi.org/10.1136/bmj.328.7446.1010

Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a Methodology for Developing Evidence-Informed Management Knowledge by Means of Systematic Review. British Journal of Management, 14(3), 207–222. http://doi.org/10.1111/1467-8551.00375

Primary Sources


MEDIA & DESIGN

54

(34) Aitamurto, T., Sirkkunen, E., & Lehtonen, P. (2011). Trends in Data Journalism. Retrieved from http://virtual.vtt.fi/virtual/nextmedia/Deliverables-2011/D3.2.1.2.B_Hyperlocal_Trends_In%20Data_Journalism.pdf

(2) Ananny, M., & Crawford, K. (2015). A Liminal Press: Situating news app designers within a field of networked news production. Digital Journalism, 3(2), 192–208. http://doi.org/10.1080/21670811.2014.922322

(12) Appelgren, E., & Nygren, G. (2012). Data Journalism in Sweden – Opportunities and Challenges: A Case Study of Brottspejl at Sveriges Television (SVT). In C. Mills, M. Pidd, & E. Ward (Eds.), Proceedings of the Digital Humanities Congress 2012. Sheffield: HRI Online Publications. Retrieved from http://www.hrionline.ac.uk/openbook/chapter/dhc2012-appelgren

(11) Appelgren, E., & Nygren, G. (2014). Data Journalism in Sweden: Introducing New Methods and Genres of Journalism into “Old” Organizations. Digital Journalism, 2(3), 394–405. http://doi.org/10.1080/21670811.2014.884344

(18) Ausserhofer, J. (2015). „Die Methode liegt im Code”: Routinen und digitale Methoden im Datenjournalismus. In A. Maireder, J. Ausserhofer, C. Schumann, & M. Taddicken (Eds.), Digitale Methoden in der Kommunikationswissenschaft (Vol. 2, pp. 87–111). Berlin: digitalcommunicationresearch.de. Retrieved from http://nbn-resolving.de/http://dx.doi.org/10.17174/dcr.v2.5

(3) Baack, S. (2013). A New Style of News Reporting: Wikileaks and Data-driven Journalism. In B. Rambatan & J. Johanssen (Eds.), Cyborg Subjects: Discourses on Digital Culture (pp. 113–122). CreateSpace Independent Publishing. Retrieved from http://www.ssoar.info/ssoar/handle/document/40025

(26) Bakker, P. (2014). Mr. Gates Returns: Curation, Community Management and Other New Roles for Journalists. Journalism Studies, 15(5), 596–606. http://doi.org/10.1080/1461670X.2014.901783

(31) Daniel, A., & Flew, T. (2010). The Guardian Reportage of the UK MP Expenses Scandal: A Case Study of Computational Journalism. In Record of the Communications Policy and Research Forum 2010 (pp. 186–194). Sydney: Network Insight. Retrieved from http://eprints.qut.edu.au/39358/

(35) De Maeyer, J., Libert, M., Domingo, D., Heinderyckx, F., & Le Cam, F. (2015). Waiting for Data Journalism: A Qualitative Assessment of the Anecdotal Take-up of Data Journalism in French-Speaking Belgium. Digital Journalism, 3(3), 432–446. http://doi.org/10.1080/21670811.2014.976415

(1) Diakopoulos, N. (2015). Algorithmic Accountability: Journalistic Investigation of Computational Power Structures. Digital Journalism, 3(3), 398–415. http://doi.org/10.1080/21670811.2014.976411

(21) Dick, M. (2014). Interactive Infographics and News Values. Digital Journalism, 2(4), 490–506. http://doi.org/10.1080/21670811.2013.841368

(19) Felle, T. (2015). Digital Watchdogs? Data Reporting and the News Media’s Traditional “Fourth Estate” Function. Journalism, 1464884915593246. http://doi.org/10.1177/1464884915593246

Corpus 1/3


MEDIA & DESIGN

55

(14) Fink, K., & Anderson, C. W. (2015). Data Journalism in the United States: Beyond the “Usual Suspects.” Journalism Studies, 16(4), 467–481. http://doi.org/10.1080/1461670X.2014.939852

(27) Garrison, B. (1999). Newspaper Size as a Factor in Use of Computer-Assisted Reporting. Newspaper Research Journal, 20(3). Retrieved from http://com.miami.edu/car/baltimore1.htm

(22)Gynnild, A. (2014). Journalism Innovation Leads to Innovation Journalism: The Impact of Computational Exploration on Changing Mindsets. Journalism, 15(6), 713–730. http://doi.org/10.1177/1464884913486393

(7) Hannaford, L. (2015). Computational Journalism in the UK Newsroom: Hybrids or Specialists? Journalism Education, 4(1), 6–21. http://doi.org/http://hdl.handle.net/2173/575953

(30) Howard, A. B. (2014). The Art and Science of Data-Driven Journalism (pp. 1–144). New York: Tow Center for Digital Journalism. Retrieved from http://towcenter.org/wp-content/uploads/2014/05/Tow-Center-Data-Driven-Journalism.pdf

(8) Hullman, J., Diakopoulos, N., Momeni, E., & Adar, E. (2015). Content, Context, and Critique: Commenting on a Data Visualization Blog. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 1170–1175). New York: ACM. http://doi.org/10.1145/2675133.2675207

(6) Karlsen, J., & Stavelin, E. (2014). Computational Journalism in Norwegian Newsrooms. Journalism Practice, 8(1), 34–48. http://doi.org/10.1080/17512786.2013.813190

(13) Knight, M. (2015). Data Journalism in the UK: A Preliminary Analysis of Form and Content. Journal of Media Practice, 16(1), 55–72. http://doi.org/10.1080/14682753.2015.1015801

(33) Koch, C., & Stollorz, V. (2014). Transparenz oder Konfusion? Methodik und empirische Basis des Datenjournalismus in der Medizin. In V. Lilienthal, D. Reineck, & T. Schnedler (Eds.), Qualität im Gesundheitsjournalismus (pp. 367–387). Wiesbaden: Springer Fachmedien. Retrieved from http://link.springer.com/chapter/10.1007/978-3-658-02427-7_21

(28) Lewis, S. C., & Usher, N. (2013). Open Source and Journalism: Toward New Frameworks for Imagining News Innovation. Media, Culture and Society, 35(5), 602–619. http://doi.org/10.1177/0163443713485494

(5) Lewis, S. C., & Usher, N. (2014). Code, Collaboration, and the Future of Journalism: A Case Study of the Hacks/Hackers Global Network. Digital Journalism, 2(3), 383–393. http://doi.org/10.1080/21670811.2014.895504

(29) Lugo-Ocando, J., & Brandão, R. F. (2015). Stabbing News: Articulating Crime Statistics in the Newsroom. Journalism Practice, 9, 1–15. http://doi.org/10.1080/17512786.2015.1058179

Corpus 2/3


MEDIA & DESIGN

56

(10) Parasie, S. (2015). Data-driven Revelation? Epistemological Tensions in Investigative Journalism in the Age of “Big Data.” Digital Journalism, 3(3), 364–380. http://doi.org/10.1080/21670811.2014.976408

(9) Parasie, S., & Dagiral, E. (2013). Data-driven Journalism and the Public Good: “Computer-assisted-reporters” and “Programmer-journalists” in Chicago. New Media & Society, 15(6), 853–871. http://doi.org/http://dx.doi.org/10.1177/1461444812463345

(32) Radchenko, I., & Sakoyan, A. (2014). The View on Open Data and Data Journalism: Cases, Educational Resources and Current Trends. In D. I. Ignatov, M. Y. Khachay, A. Panchenko, N. Konstantinova, & R. E. Yavorsky (Eds.), Analysis of Images, Social Networks and Texts (pp. 47–54). Cham: Springer International Publishing. Retrieved from http://link.springer.com/chapter/10.1007/978-3-319-12580-0_4

(25) Uskali, T., & Kuutti, H. (2015). Models and Streams of Data Journalism. The Journal of Media Innovations, 2(1), 77. http://doi.org/10.5617/jmi.v2i1.882

(24) Veglis, A. (2005). Modeling Computer Supported Collaborative Work in a Newspaper Editorial Process. In Proceedings of the 9th WSEAS International Conference on Computers (pp. 1–6). Stevens Point: World Scientific and Engineering Academy and Society (WSEAS). Retrieved from http://dl.acm.org/citation.cfm?id=1369599.1369726

(16) Weber, W., & Rall, H. (2012). Data Visualization in Online Journalism and Its Implications for the Production Process. In 2012 16th International Conference on Information Visualisation (IV) (pp. 349–356). Montpellier: IEEE. http://doi.org/10.1109/IV.2012.65

(36) Weber, W., & Rall, H. (2013). “We Are Journalists”: Production Practices, Attitudes and a Case Study of the New York Times Newsroom. In W. Weber, M. Burmester, & R. Tille (Eds.), Interaktive Infografiken (pp. 161–172). Berlin: Springer. Retrieved from http://link.springer.com/chapter/10.1007/978-3-642-15453-9_9

(17) Weinacht, S., & Spiller, R. (2014). Datenjournalismus in Deutschland: Eine explorative Untersuchung zu Rollenbildern von Datenjournalisten. Publizistik, 59(4), 411–433. http://doi.org/10.1007/s11616-014-0213-5

(20) Young, M. L., & Hermida, A. (2015). From Mr. and Mrs. Outlier to Central Tendencies: Computational Journalism and Crime Reporting at the Los Angeles Times. Digital Journalism, 3(3), 381–397. http://doi.org/10.1080/21670811.2014.976409

Corpus 3/3


MEDIA & DESIGN

57

Explore the (preliminary) corpus online – it's a protoype: http://tinyurl.com/valid-survey


Julian Ausserhofer1,2 ,@[email protected]

Robert Gutounig1 &

@[email protected]

Michael Oppermann2

@[email protected]















News & Politics

News Production Workflows in Data- driven, Algorithmic Journalism: A Systematic Literature Review