Learning modules on media interpretation and disaster response data generation

FP7-317705 D4.4 GeotagX

CitizenCyberlab_D4.4_UNITAR_GeotagX_PU_Final Page 1 of 57

D4.4 Learning modules on media interpretation

and disaster response data generation



Deliverable Identification Sheet Project ref. no. 317705

Project acronym Citizen Cyberlab

Project full title Technology Enhanced Creative Learning in the field of Citizen Cyberscience

Document name CitizenCyberlab_D4.4_UNITAR_GeotagX_PU_Final

Security (distribution level) PU

Contractual date of delivery Month 24, 30/09/2014

Actual date of delivery Month 24, 30/09/2014

Deliverable number D4.4

Deliverable name Learning modules on media interpretation and disaster response data generation

Type Report

Status & version Reviewed, v21

Number of pages 57

Author(s) Eleanor Cervigni and Cobi Smith

Other contributors Sylvia Nagl, Rosita Haddad, Lars Bromley, Daniel Lombraña González

Project Officer Juan PELEGRIN, [email protected]

Abstract Geotag-X is an open-source crowdsourcing platform designed to engage volunteers in analysing media about disasters. It is a pilot project within Citizen Cyberlab to develop and test tools to engage and educate volunteers in supporting relief and recovery efforts of disaster response agencies. Participation has been targeted at people with expertise related to the project, who can get involved beyond completing crowdsourcing tasks by giving advice, making connections or codeveloping modules. Focusing on developing a community of practice for GeoTag-X is important given the range of actors already active in humanitarian disaster response and open technologies. GeoTag-X modules have been through user testing and codesigning modules involved usability analysis and iterative development. Concepts of collaborative learning and legitimate peripheral participation are relevant. Early data about website users and knowledge of participants from hackathons and other events reflect research about engagement and learning from digital communities.

Keywords crowdsourcing, humanitarian, disaster, media

Sent to peer reviewer 03.09.2014

Peer review completed 10.09.2014



Executive Summary Geotag-X is an open-source crowdsourcing platform designed to engage volunteers in analysing media about disasters. It is a pilot project within Citizen Cyberlab to develop and test tools to engage and educate volunteers in supporting relief and recovery efforts of disaster response agencies. Beta testing for GeoTag-X has primarily been focused on engaging participants at hackathons and other face-to-face events, as well as online participation supported by GeoTag-X staff by email or Skype. At this early pilot stage with limited documentation, having someone with knowledge of the project on-hand to support user testing is valuable. All engagement so far has involved connecting with existing communities and networks, for example the AAAS On-call Scientists program, CERN Summer Student Webfest and Humanitarian Data Exchange. Participation has been targeted at people with expertise related to the project, who can get involved beyond completing crowdsourcing tasks by giving advice, making connections with other networks and people, or potentially codeveloping future modules on the platform. Focusing on developing a community of practice for GeoTag-X is important given the range of actors already active in humanitarian disaster response and open technologies. Promoting collaboration and building on existing knowledge and practice is key to the project’s success. GeoTag-X modules have been through user testing, which was well documented in the related report, D6.1 Evaluating the design of Citizen Cyberlab pilot projects and platforms. Codesigning modules has also involved usability analysis and iterative development. Feedback from user testing informed iterations of changes in module development over the past nine months since modules were first made live online. Reviewing research about related platforms has informed development priorities, while the needs of specific communities has shaped priorities, such as for example integrating Hindi translation into introductory text of the Yamuna - Women for Sustainable Cities project. Concepts of collaborative learning and legitimate peripheral participation are of relevance in understanding learning from the development of modules on media interpretation and disaster response data generation. Early data about website users and knowledge of participants from hackathons and other events reflect research about engagement and learning from digital communities. Alongside module development has been discussion and documentation about learning objectives, which will be tested in the next phase of the project and reported in the future Citizen Cyberlab deliverable D6.3: Learning and Knowledge Acquisition Evaluation Report.



Table of contents Executive Summary ..................................................................................................................... 3 Table of contents .......................................................................................................................... 4 1 Introduction ............................................................................................................................. 6

1.1 Aims ................................................................................................................................. 7 2 GeoTag-X as a research experiment ..................................................................................... 7

2.1 Related research .............................................................................................................. 7 2.2 Why GeoTag-X exists ...................................................................................................... 9 2.3 Challenges and opportunities ........................................................................................ 12

3 Designing GeoTagX ............................................................................................................. 13 4 System development ............................................................................................................ 13

4.1 Finding and storing relevant media ................................................................................ 15 4.1.1 How data is stored in the system ............................................................................ 17 4.1.2 Experimental integrations ........................................................................................ 17

4.2 Types of analyses .......................................................................................................... 18 4.2.1 Binary or polar analyses .......................................................................................... 18 4.2.2 Geotagging analyses ............................................................................................... 18 4.2.3 Multiple-choice analyses ......................................................................................... 19

4.3 Tutorials ......................................................................................................................... 22 5 Developing modules on the platform .................................................................................... 23

5.1 Face-to-face engagement .............................................................................................. 26 6 Case Study 1: Identifying Drought ........................................................................................ 27 7 Case Study 2: Yamuna - Women for Sustainable Cities ...................................................... 28 8 Data design .......................................................................................................................... 30 9 Learning from GeoTagX ....................................................................................................... 31

9.1 Participant data .............................................................................................................. 32 9.2 Understanding who participates ..................................................................................... 34 9.3 Project knowledge .......................................................................................................... 35

9.3.1 Data ......................................................................................................................... 35 9.3.2 Information .............................................................................................................. 35 9.3.3 Knowledge ............................................................................................................... 35

9.3.3.1 Tacit Knowledge ............................................................................................... 35



9.3.3.2 Explicit Knowledge ............................................................................................ 36 9.4 What participants learn .................................................................................................. 36 9.5 Monitoring and evaluation .............................................................................................. 39

9.5.1 Quantitative data and analytics ............................................................................... 39 9.5.2 Communications with volunteers ............................................................................. 39 9.5.3 Usability testing ....................................................................................................... 40 9.5.4 Surveys ................................................................................................................... 40 9.5.5 Outcome Mapping ................................................................................................... 40

10 Future development ............................................................................................................ 41 10.1 Free tagging media versus structured analyses .......................................................... 41 10.2 Analyses to develop humanitarian thinking .................................................................. 42 10.3 Engaging with Migrant Communities ........................................................................... 42 10.4 Verifying volunteer contributions .................................................................................. 42 10.5 Analyses to develop humanitarian thinking ................................................................. 43 10.6 Integration with the growing PyBossa developer community ....................................... 43 10.7 Documentation of tacit knowledge ............................................................................... 43

11 Conclusion .......................................................................................................................... 44 12 List of acronyms ................................................................................................................. 45 13 References ......................................................................................................................... 46 14 Appendix A: collaboration log of the Yamuna - Women for Sustainable Cities Project ...... 50 15 Appendix B: participant surveys developed in partnership with the University of Geneva . 55



1 Introduction This report serves as Deliverable 4.2.2 for the Citizen Cyberlab project and is intended to document the creation of learning modules on media interpretation and disaster response data generation. GeoTag-X is developed by the United Nations Institute for Training and Research (UNITAR) Operational Satellite Applications Programme (UNOSAT). GeoTag-X has evolved in the last year from an idea emerging from prototypes into a platform on which several pilot modules are being tested. New methods for ingesting media are being tested to complement the initial Firefox plugin, for example experimental compatibility with Flickr and Epicollect. The pilot project is now live online and open for user testing on the domain: geotagx.org and this site also serves as Deliverable 3.3.2 as the platform for learning through collaborative problem solving. The name GeoTag-X was chosen by the initial development team to take forward the project, as it expresses key components of the system: Geo: all media should be georeferenced as accurately as possible; Tag: all media should have data tags relevant to the humanitarian and disaster response community, compatible with existing disaster response methodologies; X: the system should be adaptable for diverse disaster situations anywhere in the world. For example a flood in Myanmar can occur at the same time as a flood in Sudan and we want GeoTag-X modules to support humanitarian responses for each event. Further background on the project is in the previous Citizen Cyberlab report D4.2.1: System design for collaborative disaster mapping. We now have five distinct categories of modules being piloted on the website, each of which can be described as a project. Within each project or category the number of modules vary, from 1 to 4 so far, although more are possible in future. For example the Yemeni Agricultural Water Assessment project has a single module, which involves volunteers geotagging images. In contrast, the Yamuna - Women for Sustainable Cities project category, representing a robust collaboration with a developing organization in India, contains four distinct modules focused on different types of learning and analysis. One, like the Yemeni module, is about geotagging. Another engages volunteers in analyses about water movement, flood protection measures and indicators of pollution. Another is focused on shelter and the other on animal health. Each of the modules presents different learning opportunities and challenges, as well as similarities that will be discussed in this report. GeoTag-X is designed so that volunteers can both contribute media and analyses. Volunteers can download the Firefox plugin and share media URLs relevant to a particular project. Volunteers can also analyse these media through structured modules. Both types of engagement are possible through volunteers interacting independently with the GeoTag-X site. Another significant area of learning is in module development, which happens through collaborative interactions within the GeoTag-X community of practice. At this stage of the project in which a community of practice is developing, much collaborative learning is happening during module development. In this sense, learning outcomes are the modules themselves, from which other volunteers can then learn.



As described in Citizen Cyberlab report D2.1, initial motivations and intentions transform during participation in a project. GeoTag-X volunteers may gain skills or knowledge beyond their area of expertise and change their understanding and confidence in their skills and knowledge develop. The concept of legitimate peripheral participation (Lave and Wegner 1991), which underpins the participation model from report D2.1, is an applicable concept for the GeoTag-X community. Participants are developing a community of practice (Wegner 1998) about open-source crowdsourcing and geotagging for humanitarian aims. This practice happens first-hand through collaborative processes of the core community in coding the website, scripting modules and testing usability. Reaching out to related organizations and disaster-affected communities brings valuable knowledge and expertise into our community of practice.

1.1 Aims GeoTag-X aims align with best practices in humanitarian knowledge management, adapted from research earlier this century (King 2005): ! identify media relevant to a disaster or emergency that are not already being categorized

and geotagged; ! analyse content to generate associated meta-data for sharing, pooling, comparison,

verification and mapping; ! establish a community of practice involving individuals in multiple organizations to develop

tacit knowledge associated with explicit knowledge generated through the project and with the knowledge of other organizations;

! focus on geotagging, to facilitate visualization and accessible representations of complex data and information;

! prototype humanitarian application of an open-source crowdsourcing platform and use prototype data and information to answer questions and respond to identified information needs;

! recognize the value of tacit knowledge gained from field experience, collaboration and learned expertise

! research if and how such knowledge can be passed on to new digital volunteers; ! promote the use of GIS technologies and internet technologies, including PyBossa for open-

source crowdsourcing and GitHub for virtual collaboration and version control. These aims are drawn from the Information Systems for Crisis Response and Management (ISCRAM) community, given that GeoTag-X has humanitarian goals above and beyond goals of citizen science that are reflected across the Citizen Cyberlab project.

2 GeoTag-X as a research experiment

2.1 Related research GeoTag-X is unusual within the Citizen Cyberlab project as its focus is humanitarian disaster responses more than science learning. However exploring the capacity of everyday people to contribute to knowledge through crowdsourced geotagging reflects research about citizen science, risk communication and the bounds of expertise. Late last century Brian Wynne



(1992,1996) wrote about differing understandings of Cumbrian sheep farmers, civil servants and scientists about the impact of radioactive fallout from the Chernobyl disaster. This work, influential in research on public understanding of science and risk communication, contrasted the unique, localised expertise of farmers with the powerful expertise of civil servants and scientists. GeoTag-X could have served for the case study Wynne described. Using GeoTag-X, farmers and other local residents could have shared photographs of sheep in Cumbria and geotagged them, tracking the location of diseased sheep in datatsets that civil servants and scientists, as well as farmers themselves, could have used to inform responses to the disaster. Indeed, several pilot modules for GeoTag-X allow people to analyse photographs of livestock, assess their health or geotag them. Research indicates that public attitudes to science, as well as trust in science, are impacted not only by peoples’ scientific literacy and ability to use scientific methods. Public attitudes and trust are influenced by how those in power in science and governance use science to respond in times of crises and disaster (Haerlin & Parr 1999; Longstaff and Yang 2008). So having an open-source, crowdsourced humanitarian disaster response project within a broader exploration of citizenship and science is relevant and valuable. Experiments related to GeoTag-X have been documented by the community of researchers in ISCRAM. Existing research allows us to anticipate needs as the system grows, for example challenges in data storage and integrating new social media sources (Schram & Anderson 2012). Complementary systems exist to support disaster response, for example an Australian system developed in collaboration with official crisis coordination teams. This system was motivated by a 2009 Australian Royal Commission on bushfires, which heard evidence that official services lacked information reported in near-real-time on social media (Yin et al. 2012, p54). Another system uses mobile phone calling data to inform emergency responses (Madey et al. 2006). How GeoTag-X differs from existing systems will be described in the next section of this report. Crisis informatics has been defined as study of the social, technical and informational concerns of emergency response; including interactions and concerns of formal responders as well as affected citizens (Palen et al. 2010; Starbird et al. 2012). While the Citizen Cyberlab refers to participants as citizen scientists, within crisis informatics research the same participants may be described as everyday analysts (Palen, Vieweg & Anderson 2010) or first responders (Palen et al. 2010). All terminologies reflect the importance of meaningful participation and valuing local knowledge, alongside that of experts.

By viewing the citizenry as a powerful, self-organizing, and collectively intelligent force, ICT has the potential to play a remarkable and transformational role in the way society responds to mass emergencies and disaster. Furthermore, this view of a civil society that can be augmented by ICT is based on social and behavioral knowledge about how people truly respond in disaster, rather than on simplified and mythical portrayals of people unable to help themselves. Research has shown that disaster victims themselves are the true first responders, frequently acting on the basis of knowledge not available to officials. (Palen et al. 2010, p1-2)



A strength of the GeoTag-X project is collaboration led by UNOSAT, an office of the United Nations Institute for Training and Research routinely involved in disaster response, with practitioners and researchers from a range of institutions. The project emerged from disaster response experts recognising the value in crowdsourcing knowledge during crises, indicating that researchers’ vision of a participatory future for humanitarian disaster response aligns with the needs of the expert disaster response community.

This view challenges fundamental assumptions about how information should be controlled and disseminated to ensure public safety. That officials can and should provide the best information during emergencies, and that the public can and should primarily rely on official information appears to be a difficult-to-resist, near-universal human hope. However, it is critical that we adopt a view of broad public participation in emergency response as soon as possible. (Palen et al. 2010, p2)

The speed and willingness that digital volunteers have shown in collecting and compiling information in disaster responses has already influenced the United Nations Office for the Coordination of Humanitarian Affairs (UNOCHA). Digital volunteers have helped to collect information for relevant datasets much more rapidly than officials could alone, with huge potential impacts on officials’ responsibilities in information management (UNOCHA 2011).

2.2 Why GeoTag-X exists GeoTag-X has evolved from UNOSAT’s experimentation with crowdsourced geotagging in collaboration with humanitarian and technology organizations. In 2011, a collaborative project with the Citizen Cyberscience Centre and the University of Geneva developed a system, then called CyberMappr, for engaging volunteers to find and georeference online photographs depicting damage resulting from conflict. In 2012 the system was piloted under the name Geotag Libya when Lars Bromley, Daniel Lombraña González and digital volunteers collaborated to experiment with crowdsourced geotagging of civilian infrastructure damaged in explosions in the Libyan crisis. Volunteers helped to source relevant photos online or from their own media, filter them into categories and attempted to geotag photos to their location on satellite imagery (UNITAR-UNOSAT 2012). Lars Bromley was employed by UNOSAT at the time and Daniel began a contract emerging from his work with Francois Grey at the Citizen Cyberscience Centre. Lars remained at UNOSAT while Daniel became a Shuttleworth Foundation Fellow, allowing him to independently focus on PyBossa development. The concepts developed in the CyberMappr pilot evolved into several current and distinct projects including Forest Watchers, CrowdCrafting and GeoTag-X, all based on the evolving PyBossa open-source crowdsourcing code. GeoTag-X was a project of UNOSAT and has been predominantly led by Eleanor Rusack, who began maternity leave in July 2014 and handed project leadership to Cobi Smith. GeoTag-X differs from existing systems and may be complementary given its focus on image analysis rather than text or audio, as well as human rather than machine analysis. Figure 1 (Yin



et al. 2012, p53) shows relevant data shared on the social media site Twitter during the 2011 Christchurch earthquakes. Figure 1. Data from Twitter during the 2011 Christchurch earthquakes (Yin et al. 2012, p53)

The system associated with Figure 1 is designed for geotagging and text analysis, using algorithms to monitor streams of text from Twitter and cluster them based on relevance to events. There are two significant differences in the GeoTag-X system. Firstly, these systems rely on algorithms and machine interpretation. Secondly, these systems to do not analyse images. Human interaction with images is the focus of GeoTag-X, which presents unique opportunities and challenges within the realm of crisis informatics and disaster response data. For example, Figure 2 shows an uncredited photo shared in eyewitness accounts of the same disaster.



Figure 2. Uncredited eyewitness photo from the 2011 Christchurch earthquakes (Stuff.co.nz 2011)

This photo lacks the specificity of origin and textual content seen in Figure 1, yet a wealth of information is present. This information ranges from evidence of damage and first responders in action, as well as street and shop signs in the background that someone with local knowledge could potentially use to geotag the location. Images are more universal and potentially data-rich than language, reflected in the expression “a picture paints a thousand words”. Research indicates that disaster response officials create pictures and maps in their minds to help with situational awareness when given only audio and text inputs (Blandford & Wong 2004). While text can be understood only by those people or machines literate in the given language, images can be interpreted by people from diverse cultural and linguistic backgrounds. Other researchers involved with Citizen Cyberlab have used pictures for illiterate people to use participatory mapping with a new software called Sapelli (Vitos et al. 2013) which we hope to test with GeoTag-X in future, as we are testing with EpiCollect now. While images do not need to be translated for different societies, different people may consider different parts of an image significant. This diversity, while more challenging to manage within an information technology system, can also be a strength. The value of allowing communities to decide for themselves what information they engage with is explored by other researchers in Citizen Cyberlab (Ellul et al. 2013), working with communities to develop their own community maps and like GeoTag-X, supporting creation of bespoke modules or apps tailored to community needs.



The algorithms and machine interpretations built into any system design reflect certain values. Also, machine learning classifiers still need training for different events as well as validation (Palen et al. 2012; Cobb et al. 2013), so expectations of greater responsiveness or accuracy from machines alone may be unrealistic. GeoTag-X is designed so that new modules with different questions can be flexibly developed depending on human input in response to environmental conditions, typically a new disaster event. This means the system has greater potential to reflect a diverse range of values and cultural perspectives. Rather than claiming to be a neutral, anonymised data analysis platform, GeoTag-X is transparent in allowing participants to consider for themselves the questions used in the system and potentially to create new modules asking different questions. This is why GeoTag-X is framed as a citizen science project with learning objectives as part of Citizen Cyberlab, as well as a technology platform for disaster response.

2.3 Challenges and opportunities GeoTag-X is designed to support volunteer learning about participating in humanitarian disaster response efforts. The need to support volunteer training has been articulated by UNOCHA (2011) as well as in a review of the use of OpenStreetMap for humanitarian aims (Westrope et al. 2014). The geotagging focus of GeoTag-X and future use of maps emerging from analyses may help address demand for training for digital volunteers. Another humanitarian learning opportunity was identified at a GeoTag-X workshop at a social inclusion in science communication conference, where future uses in education were explored. Discussion with a humanitarian fieldworker with a psychology background led to the idea that GeoTag-X modules could help train people to respond to images as humanitarians. The workshop explored the ethical issue of showing people potentially distressing media from disasters. Codesigning modules with humanitarian workers who frame analysis questions allows volunteers to approach such media with humanitarian questions in mind. This can be contrasted with feelings of overwhelm or fatigue that may result from the same exposure to media as news. Giving everyday people training opportunities to engage with disasters as active humanitarians rather than passive media consumers may have psychological and social benefits. A challenge in humanitarian work is finding sufficient funding for training and learning in disaster prevention and management, as most funding comes in response to disasters not in anticipation of them. This problem has been well articulated in past research (Tatham and Spens 2011). A strength of the GeoTag-X project is its home within UNOSAT, focused on humanitarian disaster response, with funding through FP7 as part of Citizen Cyberlab, a project framed around learning. This focus on training and research with GeoTag-X as a collaborative learning platform links it with development goals beyond situations of acute crisis, supporting plans for greater collaboration and knowledge generation in future. Designing a system for sharing information between organizations during complex and time-dependent disaster situations is challenging. Much of this difficulty is beyond the scope of the GeoTag-X project that has been documented in other research (Comfort & Zagorecki 2004; Kapucu 2006; Ren et al. 2008). Poor information sharing and coordination during disaster responses negatively impacts collective decision-making and resulting outcomes, including



resource allocation, delayed evacuations and casualties (Bharosa et al. 2009). The concept of boundary partners discussed later in this report is related to research on this challenge (Kapucu 2006). Thus a principle for GeoTag-X is to support information sharing among disaster responders. Given that humanitarian disaster responders are typically overworked and deal with high stress situations, we are drawing heavily on research and documentation from ISCRAM and disaster response fieldwork rather than seeking heavy time investments from such experts in GeoTag-X. Integrating with organizations supporting information sharing, such as the Humanitarian Data Exchange, is essential to addressing this challenge. The Humanitarian Data Exchange is the new collaborative organization leading development of Humanitarian eXchange Language (HXL), outlined in the earlier Citizen Cyberlab report D4.2.1: System design for collaborative disaster mapping.

3 Designing GeoTagX As of September 2014 there are five categories or projects hosted on the geotagx.org platform, with others in earlier stages of development. The first phase was focused on development of the underlying open-source system, which is now well-established as components of PyBossa. The second phase has involved codesigning the first modules with volunteers. Documentation and development of the underlying open-source system PyBossa has been led by Daniel Lombraña González and is now supported by the Shuttleworth Foundation. Development of the Firefox plugin happened during the first phase. Several iterations of GeoTag-X user testing at UCLIC are reported in D6.1 Evaluating the design of Citizen Cyberlab pilot projects and platforms (Jennet and Cox 2014). Modifications of the code in response to user testing at UCLIC were developed by Fausto Cipolli in consultation with Eleanor Cervigni, Cobi Smith and Lars Bromley. This underlying system development provided the infrastructure for module development. Modules have been developed in collaborative codesign processes described later in this report.

4 System development In GeoTag-X system design, data is collected in association with a particular project and analysed through modules associated with a project. Typically a project is an event, for example the 2013 Yamuna monsoonal floods, while a module is a type of analysis, for example geotagging or recording whether photos feature shelter. The GeoTag-X front page displays projects as categories. Figure 3 shows a screenshot from the homepage. When a volunteer clicks on a project category on the homepage they are taken to that project’s page with a brief description of the project, the option to help find photos for the project, and a list of the project modules.



Figure 3: Screenshot from the homepage

Research about citizen science and crowdsourcing indicates that volunteers should be able to usefully contribute limited time to a project (Eveleigh et al. 2014). This is supported by results of usability studies (Jennet and Cox 2014) which found that the GeoTag-X website, modules and tutorials need to be as simple as possible to avoid people becoming frustrated and discontinuing analysis. Initial engagement the platform has been designed to be quick and easy to understand for new volunteers, with minimal navigation before contributing. The landing page of the Geotag-X platform was customised from the standard PyBossa interface to address this need for simplicity and immediacy. Notable changes to the design of the homepage so far have included:

• Highlighting if a user is contributing anonymously to encourage them to either create an account or sign in.

• Changing how projects and modules were organised. On the original PyBossa landing page all active modules were displayed directly on the homepage. This was changed for



GeoTag-X so that project categories are displayed instead. Volunteers click on a project to view the modules within and choose to which they will contribute.

Simplifying the layout and colour scheme. We removed all sections of the original homepage that were not necessary for the volunteers. This simplification left three possible actions for the volunteer when they land on the homepage:

• Sign in or create an account; • Choose a project and begin contributing to modules, or; • Get more information about Geotag-X and how they can contribute via ‘about’, ‘find

photos’ or ‘analyse photos’ buttons. These changes were designed to make the homepage as simple and easy to understand as possible in order to retain volunteers on the site and keep them coming back. From a technical perspective, projects act like folders for saved links and are defined in the same terms in the GeoTag-X user interface as in the PyBossa server database. Modules are built using HTML and Javascript as a series of analysis steps; the same code for modules can be applied in different projects. For example, the same code for geotagging photos in a project about a flood in India can be used for a project about landslides in Peru.

4.1 Finding and storing relevant media Volunteers may codesign a module with a particular dataset in mind; media with which they frame their understanding of potential questions for analysis and develop shared understanding. One a module is published within a project category, volunteers can add more media to this initial dataset. A plugin for the most popular open-source browser Mozilla Firefox was developed in the first phase of the project, while experimental methods are now being integrated for ingesting photos from Flickr and preliminary integration with EpiCollect, another Citizen Cyberlab project that ingests raw media direct from mobile devices. The Firefox plugin allows GeoTag-X to store URLs for media from sources ranging from blogs and news articles to social media including Twitter and Facebook. Volunteers who visit geotagx.org wishing to help find photos are invited to download and install the plugin, then they are asked to choose the project within GeoTag-X to which they want to contribute. The plugin activates in response to detecting keywords defined as relevant to a project as volunteers browse the internet. The plugin triggers a popup if the defined keywords are present on a page, asking the volunteer if they want to add images on the page to the project. Figure 4 shows a screenshot of the plugin in action on a news page, associated with the Yamuna Women for Sustainable Cities project category.



Figure 4. Screenshot of the Firefox plugin in action on a news page

If the volunteer chooses to send a picture that the plugin finds, the plugin will save the URL of the photo, the URL of the webpage on which it was found, and the project for which is was collected into a database. This database is part of a specific PyBossa service module that converts the links into PyBossa tasks that are presented to volunteers for analysis in the different modules.



4.1.1 How data is stored in the system The program that handles the images for analysis is called Mnemosyne and was developed by Daniel Lombraña González alongside PyBossa in the first phase of the project. Figure 5 shows how the Mnemosyne system relates to the GeoTag-X PyBossa system, as well as how media URLs are associated with data resulting from analyses. Figure 5. How the Mnemosyne system relates to the GeoTag-X PyBossa system

4.1.2 Experimental integrations Further technical development to maximise benefits of integrating with photo sharing services such as Flickr will be useful. One of the reasons Flickr and Panoramio were identified as useful sites to develop compatibility with is that they have a geotagging option people can use when uploading photos from their devices. At this stage, our technology pulls across this existing metadata with the image URL, however it is not used within our system to inform analyses. Given that GeoTag-X aims to analyse media relevant to a disaster that is not already being categorized and geotagged, valuing existing metadata and avoiding duplication is important. Integration with another Citizen Cyberlab project EpiCollect is experimentally functional. We hope to test this integration further at another hackathon at CERN in November, as part of a simulation refugee camp, related to another UNOSAT project about elevation mapping. EpiCollect (Aanensen et al. 2009) is a web and mobile app for data collection including GPS and media, which is novel for GeoTag-X as it would allow data to be collected firsthand, rather than relying on already published online sources. However, ethical and legal issues raised by collecting media firsthand are being explored before plans to release this feature on the public platform.



4.2 Types of analyses Steps within prototype learning modules developed so far can be categorised as one of three types: polar analyses, geotagging analyses or multiple choice analyses. However in all cases, even polar analyses, participants are given the option of choosing a “don’t know” option. The importance of actively labelling uncertainty was noted in a review of the accuracy of OpenStreetMap digital volunteers assessments of damage (Westrope et al. 2014). Types of analysis questions are explained from a technical perspective, followed by explanation of how each of the modules were developed from a participatory perspective.

4.2.1 Binary or polar analyses Binary or polar analyses are focused on yes or no, absent or present analyses. Volunteers are asked to consider a pair of alternatives and select which one to associate with the media being analysed. Figure 6a is a screenshot of a polar analysis from the one of the Yamuna monsoon flooding modules, regarding the presence or absence of water. Figure 6a: screenshot of a polar analysis

Every module begins with the polar analysis step of determining whether media is spam or not, so polar analyses can be considered the base step for all learning modules.

4.2.2 Geotagging analyses Another standard type of analysis is geotagging media on a map, as seen in Figure 6b.



Figure 6b: screenshot of a geotagging analysis

Given the name and focus of GeoTag-X, geotagging analyses are core to the project. Current system design is based on the assumption that each project category will include one geotagging module, which is reflected in the centrality of this module shown in Figure X earlier. Past UNOSAT experience on the GeoTag Libya project, as well as GeoTag-X, indicates that precise georeferencing is a difficult challenge. UNOSAT analysts spend considerable time georeferencing photos and videos based on visual cues in the imagery, such as a tall building or visible landmark. This is challenging even for experts. Engaging digital volunteers in doing this is problematic and highly challenging. System development in this regard needs improvement and is still in an embryonic state.

4.2.3 Multiple-choice analyses

Thirdly modules may contain multiple-choice steps asking volunteers to identify for example specific crops, animals, water or landscape features. These analyses are the most challenging and the most recent addition to GeoTag-X development. There are several reasons multiple-choice analyses are more challenging. One reason is that there are two types of multiple-choice analyses: single-answer and multiple-answer. These two sub-categories of multiple-choice are different both from technical and theoretical perspectives. Researchers have studied these two types and found they lead to different answers in surveys and so cannot be assumed to be interchangeable in interpreting results (Smyth et al. 2006). Technically, single-answer analyses result in a single value associated with a media URL, while multiple-answer results in several



new associated values. Single-answer values are studied in behavioural psychology and decision science because it means volunteers must make a forced choice, even if that choice is ‘don’t know’. Research suggests forced-choice analyses encourages deeper processing of response options (Smyth et al. 2006). This is not practical however in situations where multiple choices are relevant data. Furthermore, displaying several valid values together as options may support learning in the form of pattern recognition. For example, Figure 7 shows a multiple-answer multiple-choice analysis. Volunteers are asked to do this analysis following a polar analysis that established if they see water in the photo. Figure 7. Screenshot of a multiple-answer multiple-choice analysis.

In this multiple-answer multiple-choice analysis, they can indicate if there are white peaks in the water; if the surface of the water is rough, and/or if water is swirling around objects. They can also choose ‘none’ or ‘don’t know’ options. While it could be argued it would be better to structure the module so that each one of these characteristics is displayed as a separate step, grouping them in this way implies a pattern that volunteers may learn to associate with fast-moving water. The hypothesis that grouping such characteristics together as a single step supports pattern recognition and learning will be tested in future stages of the project. Another reason multiple choice questions are challenging is that they are the most language-intensive, limiting participation to people with strong understanding of the English language given our current project capacity. One way to address this is using pictures rather than words in module design, as trialed for example as seen in Figure 8 below.



Figure 8: screenshot of a multiple choice question containing pictures

However this style of question design is challenging for several reasons, some of which are reflected in user testing (Jennet and Cox 2014). One reason is that it takes up much more space, forcing volunteers to scroll down to see options. Another is that the same crops can appear very different depending on the season or stage of plant development; so representing them with a single image may be misleading. This is why the question is designed with a ‘get more examples’ option under each image. Another is that the time and effort spent in selecting and storing image URLs for each option adds to the workload of module design. Another is that the options presented can be perceived as arbitrary. Why for example would we display corn but not wheat, or potato but not taro? This is why the ‘get some help’ button and associated explanatory test was developed. There may indeed be value in stimulating this type of questioning, for volunteers’ learning about underlying assumptions inherent in modules and critical thinking skills. This will be explored later in the project.



4.3 Tutorials Prototype module tutorials have been developed and are accessed via module information pages. Usability studies (Jennet and Cox 2014) indicated that volunteers did not want to spend a lot of time reading text. Rather they preferred to be given the basics necessary for completing the application, with links to external sources of information for learning more that they could access if interested. This allows participants to pursue their own on-topic learning and scientific literacy, once they are familiar with project mechanics and want to be more confident in their pattern recognition. To support these learning goals of volunteers, tutorials give a condensed explanation of why we are asking each question, along with examples of what to look for in the photos. Volunteers can click on links in the help text to get more information, or can search the internet to find out more about a particular topic. Figure 9 shows a screenshot from a tutorial associated with a module about crop identification. Figure 9: Tutorial screenshot

This part of a tutorial is notable because it teaches volunteers that selecting “I don’t know” can be the most accurate and thus highest quality response. The importance of supporting volunteers to document uncertainty is supported in a recent evaluation of digital volunteer mapping for disaster response (Westrope et al. 2014). Initially we had planned on to develop online courses using P2PU or Coursera, however participants with the skills to lead such courses so far do not have the time to develop and run online courses. We have also identified some relevant MOOCS and training resources in existence, so decided to focus on the tutorials for teaching volunteers basic analyses, and to investigate possible collaborations for training in later stages of the project.



5 Developing modules on the platform Development of GeoTag-X modules in collaboration with volunteers has been a four-step process. Step one has been finding and engaging with volunteers with relevant expertise. Step two has been working with sharing volunteers to develop the questions and steps that form modules. Step three has been coding and implementing the questions and steps of analyses as a module on GeotagX. Step four has been developing tutorials and releasing documentation about the modules. The first module development began in November 2013, initiated with a callout the American Association for the Advancement of Science (AAAS) On-call Scientists Program, driven by UNOSAT staff Lars Bromley and Eleanor Cervigni. This is described in Case Study 1: Identifying Drought. The second module development was a codesign process with the Yamuna - Women for Sustainable Cities organization in India. This organization, led by Dr. Sylvia Nagl, founder of Women for Sustainable Cities and the Yamuna's Daughters volunteer project, has emerged as one of the strongest collaborations for GeoTag-X as it provides a local cadre of ready volunteers with focused interests. It is therefore described in more detail below as Case Study 2. In early 2014 the third module called Yemeni Agricultural Water Assessment was developed by UNOSAT staff as part of a project in collaboration with the non-governmental organization ACTED. Figure 10 shows a screenshot from the Yemeni Agricultural Water Assessment module. ACTED was working with local partners in the Raymah governorate of Yemen to support evidence-based decision making about agricultural water use in the region.



Figure 10: Screenshot from the Yemeni Agricultural Water Assessment module

While this pilot module is functional and demonstrates capabilities of the GeoTag-X system, the project does not have a critical mass of volunteers to make it an effective crowdsourcing project. Strategies for engaging migrant communities in such projects in future is discussed later in this report. In July 2014 the modules about rights to shelter and for practicing geotagging were created during UNOSAT staff handover between Eleanor Cervigni and Cobi Smith. The project on rights to shelter also emerged from the callout to the AAAS On-call Scientists Program. As well as volunteers working on the drought project, Professor Brian Gran volunteered through the callout given his interest in human rights. By July, following the same expertise sharing strategy as described in the case study with other AAAS On-call Scientists below, Brian had documented his thoughts on what questions could produce relevant analyses. In July Cobi joined the GeoTag-X project team, when the code was developed enough that, with the support of Eleanor and Fausto, Cobi could look at the existing modules coded as described in the two case studies and adapt the code for new modules based on Brian’s analyses. Figure 11 shows a screenshot from one of the two rights to shelter modules currently being piloted on the platform. To support documentation of learning and development from this project, a dedicated repository has been set up on GitHub which is being tested by Brian and his collaborators in the United States.



Figure 11: Screenshot from a rights to shelter module

The rights to shelter project was also an opportunity to test newly coded integration of Flickr photo sets, where humanitarian organizations such as UNHCR and ECHO were sharing photos relevant to GeoTag-X aims. This project is still under development as the code is adapted and participants begin testing the modules. Informal feedback from people in Europe and Australia testing the website was that it was difficult to engage in the geotagging aspect of the project given that as of July, existing geotagging modules were focused on Yemen and New Delhi and required local knowledge of these places. So the practice geotagging project was developed to address the need for a place for new volunteers to learn about geotagging, regardless of local knowledge. Figure 12 shows a screenshot of the “practice geotagging” project. This project focused on regional satellite imagery and maps, which allow anyone with knowledge of world geography to test out geotagging and develop understanding of the GeoTag-X project.



Figure 12: Screenshot of the “practice geotagging” project

A second module was developed within this project to see whether volunteers can identify whether images are recent or historical. This module was developed for volunteer learning about timeliness in humanitarian disaster response. It was also in response to research about the importance of volunteers critically appraising content associated with disasters, since media from previous disasters can interfere with responses to a current disaster (Gupta et al. 2013).

5.1 Face-to-face engagement During development of the “practice geotagging” project, tracking and documentation of content licenses began, to support attribution, participation and building on GeoTag-X. This is evidenced in the descriptive text in the screenshot of Figure 11. This emerged partly from Cobi’s engagement with volunteers of the Humanitarian OpenStreetMap Team during Wikimania 2014 in London, the conference of the Wikimedia Foundation. Cobi opted to attend this conference independently on her free time, motivated by prior participation in Open Knowledge projects and Wikipedia. This overlapped with GeoTag-X work and led to, for example, reestablishing links with GeoTag-X volunteer Keren Limor-Waisberg, who invited Cobi to present about GeoTag-X at an Open Research Cambridge meetup following the Mozilla Festival 2014, a proposal for which was accepted linked to participation in the CERN Summer Students Webfest in August. This series of connections through face-to-face engagement and participation of the GeoTag-X project team in related projects demonstrates the value of collaboration and supporting community development. As demonstrated by limited engagement with the Yemeni Agricultural Water Assessment project described earlier, active community support needs to accompany technological development for project success. While outreach is also documented in the



Citizen Cyberlab deliverable D7.6 on dissemination activities, activities in which Eleanor Cervigni or Cobi Smith have participated in person and presented about GeoTag-X in 2014 as of September included:

• Citizen Cyberscience Summit • Science Museum Lates • CERN Summer Student Webfest • Visionary or Fantasy? Creating open spaces for science communication and social

inclusion Further to this, other members of the GeoTag-X community have been sharing the project themselves and influencing the project’s development. The depth of this influence is evidenced in the case studies reported below, as well as in Appendix A: collaboration log of the Yamuna - Women for Sustainable Cities Project. At the time of writing this report, the two first projects developed were established to the extent they had code for modules and tutorials and volunteer analyses resulting in data. So these two projects are detailed below as case studies about creating learning modules on media interpretation and disaster response data generation. As well as subsequent projects now being piloted, new projects are in development. The most significant in development is a project about health analyses, with participation from nurses and paramedics. Given the potentially gory and traumatic nature of these analyses, public release of this module may be dependent on future development of levels of participation that allow people to opt in to certain modules. This is discussed later in the report section titled ‘future development’.

6 Case Study 1: Identifying Drought The first module development began in November 2013, initiated with a callout the AAAS On-call Scientists Program, driven by UNOSAT staff Lars Bromley and Eleanor Cervigni. This program connects scientists with human rights organizations seeking their expertise. An announcement through the AAAS program called for people with a background in agriculture, social sciences or related areas. Applications from interested candidates were reviewed and filtered before being provided to GeoTag-X staff. Resulting participants in the project identifying drought included two professors in ecology and agriculture, an agricultural scientist and an expert in conservation science and policy with a background in animal science. These participants were Ilya Fischhoff, a senior scientist working on the National Climate Assessment at the United States Global Change Research Program; Lauren Young, an agricultural scientist with the Arcadia Center for Sustainable Food and Agriculture; Sarah Green, a chemistry professor at Michigan Tech; and Tom Sappington, a research entomologist with the U.S. Department of Agriculture's Agricultural Research Service. A collection of photographs



for crops in drought affected areas in Somalia were shared with UNOSAT partners from the UN Food and Agricultural Organization working in Somalia. This group of experts working on drought was based across the United States and did not know each other. Initial engagement was via email and a Skype call with each volunteer to discuss the project. These volunteers were then sent photos and asked to think about what they could see in the photo that would be relevant to drought and its effect on agriculture and livestock. This was followed by an online brainstorming session with all experts in which each photo was discussed and people identified what they noted in each photo. Eleanor Cervigni facilitated the session, guiding the group through the photos and translating what the experts were seeing into questions that could be asked within the parameters of the PyBossa system. The output of the online brainstorming session was an excel sheet with a set of questions and possible answers, ready to be translated into a module on the Geotag-X platform. These questions were then considered for relevance to disaster management, matching what the experts could see in the photos with what data can be collected in the field during disaster response work. Relevant questions were then coded into modules for the platform by Eleanor. The code developed for the modules from this first project has been adapted for newer projects, and is published on the PyBossa and GeoTag-X open-source repositories. Modifications to this code followed user testing (Jennet and Cox 2014) and the first tutorials were developed as part of this project.

7 Case Study 2: Yamuna - Women for Sustainable Cities This project was a codesign process founded by Eleanor Cervigni, Dr. Rosita Haddad from the University of Geneva and Dr. Sylvia Nagl from the Yamuna’s Daughters organization. Codesigners documented a collaboration log that forms Appendix A of this report. The concept of collaborative learning is a frame for understanding this case study, described later in this report under the title ‘learning from GeoTagX’. The idea for a codesign project emerged from a Citizen Cyberlab Meeting in Paris in January 2014. In February, Sylvia Nagl established contact about the Yamuna’s Daughters project ahead of the Citizen Cyberscience Summit. This led to a conceptual redesign from a top-down approach to a partnership of design for new modules. Researchers and practitioners have been calling for more participatory approaches to informatics development (Gurstein 2003; Jasanoff 2004). Coproducer Sylvia Nagl shared



her motivations for participation that reflect this research: The GeoTag-X coproduction model (crowd-sourced photographic image analysis) and open technology design (Pybossa and mobile apps) are extraordinarily versatile for integrating awareness raising, education and citizen participation for environmental disaster prevention and response. Coproduction using innovative mobile technology may lead to the emergence of new models of: a) open knowledge sharing and democratic knowledge networks across social divides b) active cooperation between citizens, government, public organisations and the private sector for environmental protection, resilience building and disaster response. (Sylvia Nagl, personal email, 18/8/2014)

The Yamuna’s Daughters project was used as a workshop example led by the new collaboration team at the Citizen Cyberscience Summit. Following the summit, partners agreed to continued collaboration for the purpose of co-creating GeoTag-X. In February Sylvia chose a set of images from the Yamuna’s Daughters project and annotated them, which underpinned the creation of two draft modules by Eleanor. Over the following month these were reviewed and feedback given, including a usability analysis by Sylvia. Modules were modified in an iterative development process in response to feedback. Sylvia used her local knowledge of New Delhi to add media to the project while testing the Firefox plugin. The partners collaboratively planned for and tested changes to the module and new modules, which included designing the questions for analysis. In March, plans began for a proposal dedicated to developing the project, including supporting Sylvia’s work on it in India. These plans have continued to develop, in consultation with others including Francois Grey who is working on the sustainability of Citizen Cyberlab projects, UNOSAT management, as well as several partners in India. Careful codesign and considered planning of the Yamuna - Women for Sustainable Cities Project has resulted in the best model for GeoTag-X development so far. Featuring this codesign project on the front page of the GeoTag-X site indicates the platforms potential to prospective collaborators and is a proof of concept for the codesign process in module development. It was testing modules in this project that influenced the decision of new staff to join the GeoTag-X team, following work on participatory media projects in Asia. The Yamuna - Women for Sustainable Cities project is now shared in workshops with prospective collaborators to demonstrate the capabilities of GeoTag-X as a platform. Active codesign and iterative development in this project was fertile ground for learning about creating modules on media interpretation and disaster response data generation. Areas of consideration included the impact of cultural differences on people’s analyses and the potential for using audio or pictures instead of script to support accessibility. Limitations of a web platform were discussed along with potentials of a better mobile



platform for use in developing countries like India. Mentioned also in the ‘future development’ section of this report, this project about monsoonal flooding of the Yamuna River in New Delhi involves discussion about translation into Hindi and Urdu as well as developing a better mobile version of GeoTag-X for use in India. It is possible the Yamuna - Women for Sustainable Cities project may spin off into its own versions GeoTag-X as ideas and practice matures – supporting this potential for innovation and community ownership was discussed in face-to-face meetings in September. This is a natural feature of open-source technologies and reflects the relationship between GeoTag-X and its underlying platform PyBossa. Partners have committed to sharing learning outcomes as the project progresses as experiences from this project can inform and enhance development of other projects. Also mentioned in the ‘future development’ section of this report is the need for greater documentation of tacit knowledge so it can become explicit knowledge to support learning among new volunteers.

8 Data design GeoTag-X data tables are designed to be adaptable for the needs of different organizations. At this stage results can be exported as raw, unsummarised data in both JSON and CSV formats. For data verification, the PyBossa platform on which GeoTag-X is built has a default value of 30 task runs, or individual user analyses, per task. While able to be modified, this value has been maintained for GeoTag-X as it is commonly used for statistical analysis. It is this repeat analysis that makes PyBossa a crowdsourcing platform. Users (anonymous and authenticated) can only participate once per analysis. Once a user has submitted an answer for a given analysis, the Pybossa system sends the user a new analysis to complete. A sample output of data from a single task in a module displays in the format below, with identifying numbers replaced with x. Note that this example comes from an anonymous user who was not logged in to the site, so the user ID is null. Data Output 1 is from a module about shelter for displaced people, specifically a task focused on giving yes/no answers to questions. Data Output 1: results from a yes/no analysis {"info": {"son_app_id": 32, "img": "https://farm3.staticflickr.com/2933/14388396346_de506bba9a_b.jpg", "task_id": 1650, "sheltervillage": "Yes", "shelter": "Yes", "shelterweather": "3", "children": "Yes"}, "user_id": null, "task_id": 1650, "created": "2014-08-02T14:07:22.022201", "finish_time": "2014-08-02T14:07:22.022231", "calibration": null, "app_id": 36, "user_ip": "xx.xx.xxx.xxx", "timeout": null, "id": 1219}



Data can be outputted in JSON format, compatible with many open data and humanitarian data projects. Data Output 2 in JSON format is from a task focused on geotagging:

Data Output 2: results from a geotagging analysis { "info": { "isRelevant": "Yes", "latlancoords": "2.626953125, 41.7138671875", "img": "http://www.railnews.co.in/wp-content/uploads/2013/06/6171_3530_yamu.jpg", "task_id": 582 }, "user_id": 2, "task_id": 582, "created": "2014-05-02T14:01:03.821552", "finish_time": "2014-05-02T14:01:03.821572", "calibration": null, "app_id": 27, "user_ip": null, "timeout": null, "id": 441 },

All analysis types can be exported as either CSV or JSON format, the two analysis and output types are used as examples for contrast. Sharing data in such formats is more transparent than sharing data in more processed formats defined by specific agencies, given that each organization has its own technologies and processes to which they are accountable. Different organizations can use this data to generate their own visualisations or analyses using this data in combination with other sources. The GeoTag-X system development is focusing on compatibility with the data standards of Open Knowledge, Ushahidi’s crisis.net API and the UN-associated Humanitarian Data Exchange now being developed, known as HXL. There is currently an experimental function for exporting the results to a CKAN server, part of the Open Knowledge system, the success of which indicates data is similarly exportable to crisis.net and for the Humanitarian Data Exchange. We are collaborating with people in organizations working on humanitarian data standards, including the three mentioned above, to decide priorities for data sharing and structure. We also also discussing with PyBossa developers potential data visualisation tools that would serve not only for GeoTag-X but also CrowdCrafting and other PyBossa projects.

9 Learning from GeoTagX The concept of collaborative learning applies to the GeoTag-X project and community, evidenced in the impact of Case Study 2: Yamuna - Women for Sustainable Cities. Collaborative learning can be simply defined as learning together, though it is a topic of research in itself (Dillenbourg 1999; Fransen et al. 2011). So far the project has primarily



focused on developing modules and analyses, so most outcomes have come from collaborative learning of the core community developing the modules. Writing this report has involved making tacit knowledge into explicit knowledge in a documented form, so is a learning outcome in itself. Documenting further tacit knowledge is reported as a priority for the next phase of the project in the section of this report titled ‘future development’. The concept of legitimate peripheral participation (Lave and Wegner 1991) underpins the participation model from Citizen Cyberlab report D2.1: Motivation factors, learning behaviour, creativity and social dynamics in citizen cyber science projects. It has been applied to larger and more established open technology communities, such as Wikipedia. (Halfaker, Keyes and Taraborelli 2013), with findings relevant to this project. For example, research indicates that participation in such online communities typically has long-tail distribution (Wilkinson 2008), meaning a small minority of the community produces most of the content. The majority are known as “lurkers”, people who view the community content without contributing. For example in the case of English Wikipedia, people only reading outnumber contributors 10,000 to 1 (Halfaker, Keyes and Taraborelli 2013, p849). So there are many people who are engaged as readers through viewing the content of these communities, compared to few who are more deeply engaged through creating code or modules or selecting media or questions for inclusion. This long-tail distribution is reflected in GeoTag-X participation so far, as described below.

9.1 Participant data The GeoTag-X website uses Google Analytics to track quantitative user data, displayed in Figure 13. This shows that from the website’s launch on February 24 2014 until September 22 2014 when this report section was written, there were 669 users on the website. These user visits were divided over 1,445 sessions, during which users spent an average of 8 minutes engaging with the website. They visited an average of nine pages during a session. This led to a total of 13,123 pageviews. However 38% of users simply visited the homepage and left, which is known as the bounce rate. Figure 13: Google Analytics data between February 24 and September 22, 2014.



Figure 14: Percentage of returning versus new users

Google Analytics also shows that after six months of operation more than half of website users are return visitors. Figure 14 is a pie graph showing that 53.7% of users are return visitors. This data is confounded by unauthenticated use of the GeoTag-X platform in workshops and presentations. When presenting the project at a workshop or hackathon, the GeoTag-X team do not typically login to the website, as doing so takes time and introduces security risks given uncertainty about shared computers in unfamiliar locations. Establishing a dedicated login for each future workshop has been planned as an outcome of writing this report. This will remove this confounding factor for future data and also allow tracking of the number of analyses completed in workshops. As well as Google Analytics, some statistics on user data within GeoTag-X modules are available. Each completion of a task within a module is saved as a single line of results in CSV or JSON format, seen in Data Output 1 and 2 on page 30 and 31. If the user is logged in, their results are associated with their user ID. Data is collected on the creation and finish time of a task, which means in future we can use this data to explore engagement and learning. For example, we can see how long participants take to complete an individual task. In association with their user ID, we can explore in what order volunteers choose to participate in tasks and modules in the system, as well as which ones are more time-consuming. If there are longer breaks in certain instances of task creation and completion, we could infer that participants may be engaging in on-topic extra learning, for example by searching the internet for more detail about the task topic. Experimenting with this type of analysis is planned for the next phase of the project.

Figure 15 shows the distribution of answers between users who were authenticated (logged in) versus anonymous, from the module ‘condition of animals and livestock’, which was created as part of the first project development reported in Case Study 1: Identifying Drought Project earlier.



Figure 15: Authenticated versus anonymous users for the module on ‘condition of animals and livestock’

This shows that two-thirds of users are contributing anonymously while one-third are authenticated users. This suggests that only about one-third of users may be able to participate in a deeper analysis of their learning and engagement planned with the University of Geneva. From the system analytics we also have data about which users are contributing most to each module and how many tasks within the modules they have completed. Reporting on analyses of this data for public reports will require ethical clearance and consent of authenticated users.

Analysis of data from authenticated users shows that GeoTag-X reflects the norm of online communities, in which a minority of participants engage heavily while the majority engage little (Wilkinson 2008). This is evidenced in the bounce rate showing that more than a third of users of the website have simply visited the front page without contributing any analyses.

9.2 Understanding who participates

The quantitative data about who is using GeoTag-X shared above is complemented by qualitative data and narrative stories of participation. Other researchers have described a ‘volunteering predisposition’ that may reflect the profile and motivations of our participants (Elshaug & Metzer 2001). One of the volunteers working on the health project still in development shared her motivations for participation:

For my part, I review the photos from the perspective of a healthcare worker - in that I am looking for and extracting information that would lend to the health status of the people in disaster situations, by which providing information the GeotagX team can use to better present media for analysis by the crowd; and to that end, provide better analysis and datasets to support response efforts. In all of this I find my motivation. From an early age it has been a part of my fabric to reach out, help, and otherwise make a positive difference in the world. A life long dream of mine has been to travel to underserved areas of the world and to be a part of a team in the field. My life path has not yet afforded me such an opportunity. Working with the GeotagX group, however, allows me to reach out and connect in a very small way. (Kimberly Pederson, personal email, 06/09/2014)



This articulation of motivations and attitude reflects many of our volunteers for whom volunteering and sharing knowledge are a normal part of life. Participants in the CERN webfest were motivated by humanitarian aims, for example saying “I believe that the kind of work you do can really make a difference” (Pedro Ferreira, personal email, 7/8/2014 2014). Research on motivations of digital volunteers interviewed people who began participating in response to a specific emergency.

“they first responded as a digital volunteer for a specific event that they felt impacted them, and the continued on to volunteer with other events. For these participants, their volunteerism was catalyzed by a disaster situation that hit especially close to home - e.g. a crisis in an area where they have formerly lived or where they had loved ones.” (Cobb et al. 2014, p894)

Most first responders in disaster situations have positive outcomes from their contribution (Alexander & Klein 2009). However researchers note that first responders from ethnic minorities may be more vulnerable to psychological damage because enduring discrimination exacerbates stress (Perilla et al. 2002; Alexander & Klein 2009). A known contributor to disaster resilience is social capital (Masten & Obradovic 2008). This suggests that supporting participants to feel part of a community of practice, rather than simply interacting in isolation with the technology that makes up GeoTag-X, could support resilience and wellbeing. Further research to understand motivations and support to broaden incentives for participation in citizen science initiatives could help to mitigate risks and address inequities in who may benefit from participation.

9.3 Project knowledge Knowledge in GeoTag-X can be framed by research describing a taxonomy of information management in disaster response (King 2005; Tatham and Spens 2011). This taxonomy is represented here in relation to GeoTag-X.

9.3.1 Data A collection of related facts usually organized in a format such as a table or database and gathered for a particular purpose. In this sense, GeoTag-X outputs image analyses as data tables associated with specific modules.

9.3.2 Information Data that has been interpreted, verbalized, translated or transformed to reveal the underlying meaning or context. In this sense, the images shown on the GeoTag-X platform are raw data, while volunteers’ image analyses create information that is saved as data tables.

9.3.3 Knowledge Internalization of information, data and experience. It is in this sense that GeoTag-X is a learning and training project rather than a platform for data analyses. Knowledge can be further sub-divided into two categories:

9.3.3.1 Tacit Knowledge Personal knowledge resident within the mind, behavior and perceptions of individual members of the organization. Transforming the tacit knowledge of core project participants into explicit knowledge for sharing is explored in further detail in this report.



9.3.3.2 Explicit Knowledge Formal, recorded or systematic knowledge that can easily be accessed transmitted or stored in computer files or hard copy. Explicit knowledge is what GeoTag-X aims to generate for humanitarian disaster response.

9.4 What participants learn It is difficult to isolate the impact of digital volunteers in humanitarian disaster response (UNOCHA 2011), leaving some volunteers wondering if their contributions make a difference (Cobb et al. 2014). Thus is it useful to understand what impact there is from digital volunteering and one measurable of this can be participants’ own learning. Analysis of participation in GeoTag-X is based on the model of participation in citizen science developed in Citizen Cyberlab report D2.1: Motivation factors, learning behaviour, creativity and social dynamics in citizen cyber science projects. The figure below, titled Participation in citizen science model, comes from page 123 of the D2.1 Citizen Cyberlab report where it is Figure 20: in this GeoTag-X report, it is Figure 16. This figure demonstrates feedback loops involved in participant learning from their involvement in a project. Figure 16. Participation in citizen science model



Motivations for GeoTag-X are typically humanitarian or linked to a personal connection or empathy with a particular disaster. A sense of belonging to community, a real-word community impacted by disaster, may result in involvement in a digital community responding to that disaster. Contributing to tasks as a digital volunteer involves several types of learning. Example learning objectives for the Geotag-X platform and different modules according to the six types of learning defined in Figure 16 are shared in Table 1. In the case of GeoTag-X, people using external resources as they engage with the project is almost certain, given that the GeoTag-X system design ingests media from external sources, and all projects are associated with events or problems about which further information exists online. As people develop experience with the platform and as digital volunteers, they may experience changes in confidence and a sense of belonging particularly as we develop a community of practice. Table 1: Learning types applied to GeoTag-X

Learning type GeoTag-X example

Project mechanics Successfully navigating a module interface

Pattern recognition Notice a recurring landmark or crop type in images

On-topic extra learning Learn about how emergency services respond to natural disasters

Scientific literacy Learn about how science helps us anticipate the likelihood of natural disasters

Off-topic skills Humanitarian thinking in interpreting news media

Personal development Sense of connection in contributing to humanitarian disaster response

The model described in Table 1 suggests that participants learn about project mechanics first. In the case of GeoTag-X, this is typified in participants choosing a module and clicking through it as they begin to conceptualise how data about images is collected from them based on their step-by-step image analysis. In coming to understand the interface they may note a current bug, in which the progress bar reaches up to 120%, or experiment with zooming in and out of the image, or clicking the “view image source” button, which opens a new window displaying the webpage from which the image was sourced. The next stage of learning in this model is pattern recognition. In the case of GeoTag-X, this happens when volunteers complete more than one task and become familiar with the patterns of questions asked within a particular module. They may also begin to notice patterns in the images. For example working within an image set from a particular location they may notice landmarks or typical animals. The third type of learning in this model is on-topic knowledge or skills. In the case of GeoTag-X, this may be participants searching the internet for further information about a particular disaster or a particular type of humanitarian response, such as how emergency services choose when to evacuate people during a flood event. This can involve science learning, if participants search to understand the different between for example crop or animal types, weather patterns or geography.



The next type explored in the model is science literacy. In the case of GeoTag-X, participants may explore scientific concepts and research related to natural disasters such as in hydrology, geology, meteorology or agricultural science, or learn about risk perception or behavioural psychology in how people respond during disasters. This is followed in the model by off-topic knowledge and skills. In the case of GeoTag-X, this may include improved web literacy, for example through clicking “view image source” and coming to understand the relationship between the GeoTag-X site and the site where the image was sourced. It may include greater critical thinking skills in understanding media, comparing how disasters are reported as news stories versus in humanitarian response networks. It may include greater geographical or cultural understanding, for example through greater interest in a particular ethnic group or location featured in a module. The final type of learning in the model is personal development. In the case of GeoTag-X, this may include a sense of connection or purpose in contributing to humanitarian disaster response, or greater confidence in one’s ability to usefully analyse images. The next phase of the project will include evaluation of learning from GeoTag-X by staff from the University of Geneva. To support this process, learning outcomes from specific modules were documented by Rosita Hadded and Eleanor Cervigni as documented in Appendix A: collaboration log of the Yamuna - Women for Sustainable Cities Project. Table 2 below shows learning objectives from the ‘Crop Identification for Drought’ module in project mechanics, pattern recognition and on-topic learning. Project mechanics learning involves interacting with the module interface. Pattern recognition includes identifying crops, water stress and agricultural activities. On-topic learning includes explaining drought and water stress and the context of particular drought situations. Describing agricultural activities and their relevance to disaster response are further topics of extra learning. Table 2: Specific learning objectives for the ‘Crop Identification for Drought’ module

Type Learning Objective

Project mechanics Successfully interact with module interface

Pattern Recognition Identify crops in a photos

Identify different types of agricultural activities in a photo

Identify signs of water stress in crop plants

On-topic extra learning Explain what is a drought

Explain the context and situation of this particular drought situation

Explain why information on agricultural activities is important to disaster response

Describe what water stress is

describe agricultural activities in the drought affected region



Table 3 documents specific learning objectives for the ‘Practice Geotagging’ project, which was explicitly developed to support volunteer learning about geotagging as described earlier in the report section titled ‘Developing modules on the platform’. Table 3: Specific learning objectives for the ‘Practice Geotagging’ project

Type Learning Objective

Project mechanics Successfully georeference a photo on a map in the module

Successfully zoom into the world map and circle an area

Pattern Recognition Match geography in the image with geography on the map

Identify specific world regions, countries or bodies of water in the photo

On-topic extra learning Explain what is geotagging or georeferencing

Explain why geotagged media is valuable in disaster response

Staff from the University of Geneva who are involved in evaluating learning from GeoTag-X will report in further detail in the upcoming Citizen Cyberlab report D6.3: Learning and Knowledge Acquisition Evaluation Report.

9.5 Monitoring and evaluation A mixed-method approach to monitoring and evaluation is being used in this project. Firstly, use of GeoTag-X is monitored with quantitative data generated through the system design and Google Analytics. Secondly, we take a practical approach in using existing communications with volunteers as sources for qualitative analysis and participatory evaluation. These include emails and an open issues reporting platform alongside the project source code GitHub. Thirdly is observational and survey-based usability testing at UCLIC. Fourthly, the Outcome Mapping approach of considering boundary partners is being used. Finally, surveys hosted by the University of Geneva will be embedded into the system, consistent with evaluation happening across the Citizen Cyberlab project.

9.5.1 Quantitative data and analytics GeoTag-X system design combined with Google Analytics allows collection of quantitative data about volunteers’ contributions, shown in the earlier section of this report titled ‘Participant data’. Additional analytics are being developed for all Citizen Cyberlab projects. These analytics will explore in more detail behaviours of users who opt-in to the research; for example monitoring their cursor activity to see whether participants change the scale of map resolution or image resolution to inform their analysis decisions, or attempt greater geotagging precision.

9.5.2 Communications with volunteers

Quotes from volunteers have been used in this report. Communication with volunteers is an ongoing feedback mechanism for developing GeoTax-X modules and the project overall, part of an iterative participatory evaluation approach. As well as email, Skype and face-to-face meetings at events, participants have been encouraged to reflect on their experiences online.



We hope at later stages of the project to be able to cite volunteers’ own stories of participation shared through digital storytelling.

9.5.3 Usability testing

The Geotag-X platform, modules, and tutorials have been through usability testing UCLIC during formative evaluations. Problems and suggested improvements have been addressed between each round of testing. Details of this formative evaluation and the usability studies are reported in the Citizen Cyberlab project deliverable D6.1 Evaluating the design of Citizen Cyberlab pilot projects and platforms (Jennet and Cox 2014).

9.5.4 Surveys

Participant surveys are being developed specifically for GeoTag-X (on-topic learning) as well as for all Citizen Cyberlab projects (general learning) in collaboration with the University of Geneva. On creating an account on the platform, a volunteer will be asked if they are willing to participate in Citizen Cyberlab research. If they select yes, after filling in a consent form, they will invited to complete a survey in which we ask their background and motivations for volunteering on Geotag-X. They will then be presented with pre- and post-participation surveys in which they are asked to rate their level of knowledge on topics relevant to disaster management and/or a specific event that they have been working on, and their competence in performing different tasks related to the modules. The University of Geneva will host surveys and participating volunteers will be asked on the GeoTag-X site to fill in the survey at the appropriate moment via a box appearing at the top of the page, similar to the ‘sign in’ reminder seen in Figure 17. Figure 17: screenshot showing a reminder in the orange button

Planned GeoTag-X surveys developed by Eleanor Cervigni in partnership with staff from the University of Geneva form Appendix A of this report. Further detail will be shared in the upcoming Citizen Cyberlab report D6.3: Learning and Knowledge Acquisition Evaluation Report.

9.5.5 Outcome Mapping

The concept of boundary partners, from the evaluation technique called Outcome Mapping (Carden et al. 2001; Smutylo 2005), has been useful for planning engagement with other organizations active in data and mapping for humanitarian disaster response. Interactions with organizations such as the Humanitarian Data Exchange, Humanitarian OpenStreetMap Team, Open Knowledge and Ushahidi are core for the projects success, to enrich our community of



practice, promote innovation and avoid duplication. The concept of boundary partners comes from systems thinking and refers to people and organisations outside of control but within sphere of influence. Boundary partners have been described as “reaching their hands across the boundary and agreeing to work together explicitly and systematically” (Olivari and Hearn 2013); though boundary partners can instead perceive themselves as opposition. Working collaboratively involves recognising the roles of other interventions and organizations in achieving outcomes, rather than attempting to attribute change to one intervention or technology in isolation. By virtue of being open-source, GeoTag-X is already technically interwoven with some of these. For example PyBossa, the crowdsourcing platform underpinning GeoTag-X, was developed with input from Open Knowledge developers. The geotagging function is built on OpenStreetMap. Recognizing collaborations and making time for positive engagement with people is part of monitoring and evaluation. For example, the Humanitarian OpenStreetMap Team advertised for a Crowdsourcing Platform Evaluation Expert to interview stakeholders and review available platforms for flood reporting and detection in Indonesia. We emailed to indicate our interest in GeoTag-X being involved in their evaluation process, which was received with thanks. This is an example of proactively engaging with related organizations to maximise existing efforts in crowdsourced humanitarian disaster mapping. Another example is a proposal for the 2014 Mozilla Festival, building on productive engagement at hackathons so far and our open-source technology. While the proposal was not accepted, creating it involved engaging new stakeholders from Ushahidi and the Humanitarian Data Exchange, which was a valuable outcome for our community of practice in itself.

10 Future development

10.1 Free tagging media versus structured analyses A different type of analysis for future experimentation is one in which volunteers freely tag images with metadata of their choice. This draws on the work of other researchers (Bishr & Kuhn 2007) and was discussed in the earlier Citizen Cyberlab report D4.2.1: System design for collaborative disaster mapping. However this may suit a later stage of GeoTag-X development, in which levels of participation are supported, allowing more experienced volunteers to freely tag images but encouraging new volunteers to engage in learning modules first. Researchers have suggested that allowing volunteers to freely tag images with metadata of their choice may be as productive as developing prescribed modules for generating useful information. Stewart Buckfield, a creator of Flickr, one of the photo services used in GeoTag-X, discussed classification schemes:

“I think the lack of hierarchy, synonym control, and semantic precision are precisely why it works. Free typing loose associations is just a lot easier than making a decision about the degree of match to a predefined category (especially hierarchical ones). It’s like 90% of the value of a proper taxonomy but 10 times simpler” (Bishr & Kuhn 2007, p18)

A future research project could explore information and learning outcomes comparing the step-by-step analytical approach of current GeoTag-X modules with free tagging of the same media.



A hypothesis could be that modular analyses support greater learning, through volunteers questioning the reasons and values behind the programming of module steps. The process of decisionmaking related to predefined categories described in the quote above involves more mental effort, which may be associated with more learning.

10.2 Analyses to develop humanitarian thinking Another value of guiding volunteers through step-by-step analyses was discussed during a GeoTag-X workshop during a conference on science communication and social inclusion, focused on young people. A primary discussion in this workshop is whether a ‘walled garden’ or closed platform should be developed, allowing levels of participation, in contrast to the entirely open platform now. This would support for example restricting potentially gory disaster modules to people of a certain age. While participants in the workshop were not in consensus about whether this should happen as the next stage of development or following more pilot projects, there was consensus it would be a useful future development for GeoTag-X to build teacher confidence in using it for learning with young people. The current open platform stores media URLs from a range of online news sources that young people can access without restriction regardless. The case against moving youth participation to a closed platform is that young people see these news stories and may face disasters in real life, so censoring this does not support their learning. Rather development should prioritize modules that teach participants to view such media from a humanitarian perspective, rather than as a passive media consumer. There was discussion about the potential for GeoTag-X to support learning about potential careers in humanitarian fields, as well as to support learning how to respond to potentially distressing situations as a proactive and helpful first responder. Engagement with experts in working with young people is continuing, with a view to developing a new project exploring these issues.

10.3 Engaging with Migrant Communities At the same conference Dr. Emily Dawson (2012; 2014) experimented with GeoTag-X and gave advice about plans for a future project engaging with migrant communities to geotag content from their home countries. Her research with migrant communities in London about non-participation in science communication informs plans for this future project. Experience so far with the projects focused in Yemen and India shows that participatory planning and codesign are an important strategy underpinning genuine engagement and project development.

10.4 Verifying volunteer contributions Discussions with cofounders of the humanitarian social network GoodWall, in which UNOSAT through GeoTag-X is a collaborative organisation, have focused on the practicalities of verifying volunteer contributions. This is planned so that for example volunteers can list their participation in GeoTag-X on their university or job applications and have some evidence of this. There are plans to work with an early pool of GoodWall volunteers to test out a verification feature, alongside these volunteers taking part in the University of Geneva study of learning from GeoTag-X. Implementing this experiment with verification is awaiting the launch of new capabilities of the GoodWall platform. Given this has taken longer than originally anticipated,



discussion about the potential of working with the developing Mozilla Open Badges community and technology has begun. This is supported by Francois Gray who is working on this idea across the Citizen Cyberlab project and will be explored at the upcoming Mozilla Festival.

10.5 Analyses to develop humanitarian thinking For all modules, volunteers are first presented with a polar analysis of whether media is “spam” or “not spam”. This has been noted as potentially confusing and disengaging in user testing (Jennet and Cox 2014), however remains in system design given other research supporting the need for critical analysis of content emerging from social media in disaster situations (Gupta et al. 2013). Testing variations of this question to support critical thinking among volunteers and filter irrelevant or misleading content would be valuable future research.

10.6 Integration with the growing PyBossa developer community Better integration of GeoTag-X developers into the growing community involved in the underlying software PyBossa will benefit both projects and is a priority for the final year of FP7 funding. Integration plans align with open software norms and sustainability plans for the broader Citizen Cyberlab initiative.

10.7 Documentation of tacit knowledge A priority for the final year of the project is to document tacit knowledge of the core project team so it can be disseminated as explicit knowledge, allowing new volunteers to begin participating more independently. Future activities may be targeted around particular pilot modules. For example as documented in Case Study 2: Yamuna - Women for Sustainable Cities Project, there is a community specifically interested in modules about monsoonal flooding of the Yamuna river in New Delhi, which involves discussion about translation into Hindi and Urdu, as well as developing a better mobile version of GeoTag-X for use in India. It is possible communities like this may spin off into their own versions GeoTag-X as ideas and practice matures – supporting such potential for innovation and community ownership is part of the dissemination plan.



11 Conclusion Geotag-X is now a live online open-source crowdsourcing platform to engage volunteers in analysing media about disasters. It is still at an early pilot stage, so beta testing has focused on engaging participants in ways that they can be supported by GeoTag-X staff. A repository of bugs and requests for enhancement has been established on GitHub to support iterative development and feedback. This report documents collaborative learning with participants who have expertise related to the project. Most participants so far have their own related expertise which they have shared through giving advice, making connections with others of relevance, or codesigning projects. Recognizing and developing a community of practice for GeoTag-X is a priority, to collaborate with people and organizations active in humanitarian disaster response and open technologies. Concepts of collaborative learning and legitimate peripheral participation inform reporting on the development of modules on media interpretation and disaster response data generation. Knowledge of website users is complemented by knowledge of participants from face-to-face events such as hackathons. Related Citizen Cyberlab reports give further detail about the creation of learning modules on media interpretation and disaster response data generation through GeoTag-X, particularly Jennet and Cox (2014) D6.1 Evaluating the design of Citizen Cyberlab pilot projects and platforms. Citizen Cyberlab. This report incorporates a collaboration log from the codesign process for Case Study 2: Yamuna - Women for Sustainable Cities, which involved usability analysis and iterative development and is the most developed set of learning modules on media interpretation so far. Reviewing documentation particularly from the ISCRAM research community about related platforms has informed development priorities, while the needs of specific communities has shaped priorities, such as for example supporting Hindi script. Alongside module development has been discussion and documentation about learning objectives, which will be tested in the next phase of the project and reported in the future Citizen Cyberlab deliverable D6.3: Learning and Knowledge Acquisition Evaluation Report.



12 List of acronyms

AAAS American Association for the Advancement of Science

ACTED Agency for Technical Cooperation and Development

CERN European Organization for Nuclear Research

ECHO European Commission's Humanitarian aid and Civil Protection department

FP7 European Commission 7th Framework Programme for Research and Technological Development

GIS Geographical Information Systems

ISCRAM Information Systems for Crisis Response and Management

UCLIC University College London Interaction Centre

UNHCR Office of the United Nations High Commissioner for Refugees

UNITAR United Nations Operational Satellite Applications Programme

UNOCHA United Nations Office for the Coordination of Humanitarian Affairs

UNOSAT United Nations Institute for Training and Research

URL Universal Resource Locator



13 References 2009 Victorian Bushfires Royal Commission (2009) Terms of Reference, accessed online 29/08/2014 at: http://www.royalcommission.vic.gov.au/getdoc/ad6c101e-c8a2-4bbd-8fbd-a8f31813f7bd/Royal_Commission_2009_Terms_of_Reference.pdf Aanensen, D. M., Huntley, D. M., Feil, E. J., & Spratt, B. G. (2009). EpiCollect: linking smartphones to web applications for epidemiology, ecology and community data collection. PloS ONE, 4(9), e6968. Alexander, D. A., & Klein, S. (2009). First responders after disasters: a review of stress reactions, at-risk, vulnerability, and resilience factors. Prehospital and disaster medicine, 24(02), 87-94. Bharosa, N., Lee, J., & Janssen, M. (2010). Challenges and obstacles in sharing and coordinating information during multi-agency disaster response: Propositions from field exercises. Information Systems Frontiers, 12(1), 49-65. Bishr, M., & Kuhn, W. (2007). Geospatial information bottom-up: A matter of trust and semantics. In The European information society (pp. 365-387). Springer Berlin Heidelberg. Blandford, A., & William Wong, B. L. (2004). Situation awareness in emergency medical dispatch. International Journal of Human-Computer Studies, 61(4), 421-452. Bromley, L. and Lombraña González, D. (2013) Citizen Cyberlab report D4.2.1: System design for collaborative disaster mapping. Carden, F., Smutylo, T., & Earl, S. (2001). Outcome mapping: Building learning and reflection into development programs. International Development Research Centre. Cobb, C., McCarthy, T., Perkins, A., Bharadwaj, A., Comis, J., Do, B., & Starbird, K. (2014, February). Designing for the deluge: understanding & supporting the distributed, collaborative work of crisis volunteers. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing (pp. 888-899). ACM. Comfort, L. K., Ko, K., & Zagorecki, A. (2004). Coordination in Rapidly Evolving Disaster Response Systems The Role of Information. American Behavioral Scientist, 48(3), 295-313. Dawson, E. (2012). Non-participation in public engagement with science: A study of four socio-economically disadvantaged, minority ethnic groups (Doctoral dissertation, King's College London).



Dawson, E. (2014). Reframing social exclusion from science communication: moving away from ‘barriers’ towards a more complex perspective. JCOM, 13(02), C02. Dillenbourg, P. (1999). What do you mean by collaborative learning?.Collaborative-learning: Cognitive and Computational Approaches., 1-19. Ellul, C., Gupta, S., Haklay, M. M., & Bryson, K. (2013). A Platform for Location Based App Development for Citizen Science and Community Mapping. In Progress in Location-Based Services (pp. 71-90). Springer Berlin Heidelberg. Elshaug, C., & Metzer, J. (2001). Personality attributes of volunteers and paid workers engaged in similar occupational tasks. The Journal of social psychology, 141(6), 752-763. Fransen, J., Kirschner, P. A., & Erkens, G. (2011). Mediating team effectiveness in the context of collaborative learning: The importance of team and task awareness. Computers in Human Behavior, 27(3), 1103-1113. Gupta, A., Lamba, H., Kumaraguru, P., & Joshi, A. (2013, May). Faking Sandy: characterizing and identifying fake images on Twitter during Hurricane Sandy. In Proceedings of the 22nd international conference on World Wide Web companion (pp. 729-736). International World Wide Web Conferences Steering Committee. Gurstein, M. (2003). Effective use: A community informatics strategy beyond the digital divide. First Monday, 8(12). Haklay, M. (2013). Citizen science and volunteered geographic information: Overview and typology of participation. In Crowdsourcing Geographic Knowledge (pp. 105-122). Springer Netherlands. Halfaker, A., Keyes, O., and Taraborelli, D. (2013). Making peripheral participation legitimate: Reader engagement experiments in Wikipedia. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work, CSCW '13, New York, NY, USA. ACM. pages 849-860. Haerlin, B., & Parr, D. (1999). How to restore public trust in science. Nature, 400(6744), 499-499. Jasanoff, S. (Ed.). (2004). States of knowledge: the co-production of science and the social order. Routledge. Jennet, C. and Cox, A. L. (2014) D6.1 Evaluating the design of Citizen Cyberlab pilot projects and platforms. Citizen Cyberlab.



Kapucu, N. (2006). Interagency communication networks during emergencies: Boundary spanners in multiagency coordination. American Review of Public Administration, 36(2), 207–225. King, D. J. (2005), “Humanitarian Knowledge Management”, Proceedings of the Second International ISCRAM Conference. Lave, Jean; Wenger, Etienne (1991), Situated Learning: Legitimate Peripheral Participation, Cambridge University Press, ISBN 0-521-42374-0 Longstaff, P. H., and S. Yang. (2008). Communication management and trust: their role in building resilience to “surprises” such as natural disasters, pandemic flu, and terrorism. Ecology and Society 13(1): 3. Madey, G. R., Szabo, G., & Barabási, A. L. (2006). WIPER: The integrated wireless phone based emergency response system. In Computational Science–ICCS 2006 (pp. 417-424). Springer Berlin Heidelberg. Olivari, D.R. & Hearn, S. (2013). Discussion Summary Topic: What is a boundary partner? Accessed online 1/9/14 at: http://www.outcomemapping.ca/download.php?file=/resource/files/BP%20discussion%202013.pdf Palen, L., Anderson, K. M., Mark, G., Martin, J., Sicker, D., Palmer, M., & Grunwald, D. (2010, April). A vision for technology-mediated support for public participation & assistance in mass emergencies & disasters. In Proceedings of the 2010 ACM-BCS Visions of Computer Science Conference (p. 8). British Computer Society. Palen, L., Vieweg, S., & Anderson, K. M. (2010). Supporting “everyday analysts” in safety-and time-critical situations. The Information Society, 27(1), 52-62. Perilla, J. L., Norris, F. H., & Lavizzo, E. A. (2002). Ethnicity, culture, and disaster response: Identifying and explaining ethnic differences in PTSD six months after Hurricane Andrew. Journal of social and clinical psychology, 21(1), 20-45. Ren, Y., Kiesler, S., & Fussell, S. R. (2008). Multiple group coordination in complex and dynamic task environments: Interruptions, coping mechansism, and technology recommendations. Journal of Management Information Systems, 25(1), 105–130. Schram, A., & Anderson, K. M. (2012, October). MySQL to NoSQL: data modeling challenges in supporting scalability. In Proceedings of the 3rd annual conference on Systems, programming, and applications: software for humanity (pp. 191-202). ACM.



Smutylo, T. (2005). Outcome mapping: A method for tracking behavioural changes in development programs. ILAC Brief, 7. Smyth, J. D., Dillman, D. A., Christian, L. M., & Stern, M. J. (2006). Comparing check-all and forced-choice question formats in web surveys. Public Opinion Quarterly, 70(1), 66-77. Starbird, K., Muzny, G., & Palen, L. (2012, April). Learning from the crowd: Collaborative filtering techniques for identifying on-the-ground Twitterers during mass disruptions. In Proc. 9th Int. Conf. Inf. Syst. Crisis Response Manag. ISCRAM. Tatham, P., & Spens, K. (2011). Towards a humanitarian logistics knowledge management system. Disaster Prevention and Management, 20(1), 6-26. UNOCHA (2011). OCHA Lessons Learned [Publicly Editable] - Collaboration with VTCs in Libya and Japan. Accessed online 26/08/2014 at: https://docs.google.com/document/d/1wut8oDRo9BYSlc0hQ34Ng8qQ-pLVGlRO95WOvR3MN78/edit?hl=en_US Vitos, M., Lewis, J., Stevens, M., & Haklay, M. (2013, January). Making local knowledge matter: supporting non-literate people to monitor poaching in Congo. In Proceedings of the 3rd ACM Symposium on Computing for Development (p. 1). ACM. Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge University Press. ISBN 978-0521663632 Westrope, C., Banick, R., and Levine, M. (2014). Groundtruthing OpenStreetMap building damage assessment. Procedia Engineering, 78:29-39. DOI: 10.1016/j.proeng.2014.07.035 in special issue Humanitarian Technology: Science, Systems and Global Impact 2014 Wilkinson, D. M. (2008, July). Strong regularities in online peer production. In Proceedings of the 9th ACM Conference on Electronic Commerce (pp. 302-309). ACM. Wynne, B. 1992. “Misunderstood misunderstanding: social identities and public uptake of science.” Public Understanding of Science 1:304, 281. Wynne, B. 1996. “May the sheep safely graze? A reflexive view of the expert-lay knowledge divide.” Risk, Environment and Modernity: Towards a New Ecology 44–83. Yin, J., Lampert, A., Cameron, M., Robinson, B., & Power, R. (2012). Using social media to enhance emergency situation awareness. IEEE Intelligent Systems, 27(6), 52-59.



14 Appendix A: collaboration log of the Yamuna - Women for Sustainable Cities Project

RH = Rosita Haddad; EC = Eleanor Cervigni; SN = Sylvia Nagl; CS = Cobi Smith

Date Collaborative action Purpose

13 January 2014 Paris, Citizen Cyberlab Meeting

RH suggests to EC collaboration between UniGe and Unosat on co-design of applications integrating learning aspects

16-28 January 2014 RH and EC decide on on-topic questions for the ‘Crop application’

EC implements PyBossa platform with Daniel Lombraña

5 February 2014 RH and EC iterate relevant on-topic questions

EC establishes first version of Pre-post questionnaire in GeotagX ‘Crop application’

7-12 February 2014 Skype, video calls, emails RH, EC

RH and EC establish framework of integrated questionnaire for use in all GeotagX applications. Template designed.

12 -17 February 2014 RH and EC decide on importance to integrate volonteers as sensors. SN contacts EC ahead of the Citizen Cyberscience Summit.

Paradigm shift in GeotagX Applications RH, EC and SN redirect conceptual re-design of applications from top-down to partnership of design.

17 February 2014 SN offers the Yamuna’s Daughters project in New Delhi for the Citizen Cyberscience Summit Hackathon. This is agreed.

21/22 February Citizen Cyberscience Summit

Yamuna’s Daughters project is used as the example project for the GeoTagX / UNITAR-UNOSAT-UniGe workshop. Further



collaboration is agreed between EC, RH, SN for the purpose of co-creating GeoTagX applications based on Yamuna flooding in Delhi and the Yamuna’s Daughters project.

24 February SN selects a large set of images from the Yamuna’s Daughters project in Delhi and annotates these in detail, images are sent electronically to EC and RH by WeTransfer.

SN donates images for the new GeoTagX project Yamuna Monsoon Flooding. GeoTagX project Yamuna Monsoon Flooding is officially inaugurated, the donated images are used for creation of two draft applications.

26 February - 13 March EC creates two draft applications using Yamuna’s Daughters image materials and sends links to SN and RH for review.

SN reviews the draft applications in-depth and provides detailed feedback. SN and EC begin the process of defining the questions in the applications. SN also performs a usability analysis of the two draft applications and provides detailed feedback and suggestions to EC.

26 February - 13 March Numerous email exchanges between EC, SN and RH about the draft applications and project scope and aims.

17 March - 20 March SN uses Firefox plugin to select images from the internet which document the 2013 Yamuna flooding for inclusion in the GeoTagX applications, making use of her local knowledge of Delhi.

19 March Skype video call, EC, RH, SN

Review of the draft applications, detailed



planning of next collaborative activities.

24 March - 26 March EC sends SN the first draft of the topics for the Yamuna flood application. SN creates first list of questions and sends these to EC in spreadsheet format. Email exchanges and two skype calls between EC and SN discussing the questions.

28 March RH and SN meet in London for half-day collaborative work on GeoTagX and proposal development

28 March - 15 April ongoing lively exchange of emails

31 March - 7 April SN is developing the proposal for the new collaborative project

7 April SN sends RH and EC first draft of the proposal

1 April - 4 April EC sends SN an updated list of applications, split into water and people, agriculture, shelter, accessibility. SN provides detailed feedback on the questions suggested by EC under these topics and further suggestions for refinement.

3 April Skype video call EC, RH, SN about the design of the applications and the new proposal

8 April - 16 April lively email exchange between EC and SN on designing and refining the



applications

16 April Skype video call SN with Francois Grey, EC and RH also participated

SN presents the Yamuna’s Daughters project and plans for new project to Francois Gray, slide presentation on Skype by SN

23 April - 30 April EC sends SN the animal application draft, SN provides feedback, ongoing email discussion on the design of the applications

30 April - 1 May EC finishes the first tutorial for the Yamuna applications, SN provides feedback

6 May Skype video call RH and SN

8 May Skype video call EC, RH, SN

13 - 15 May SN visit in Geneva Powerpoint presentation of the new proposal by SN at Unige, discussion of advanced version of the new project proposal written by SN, collaborative meetings for co-design of the GeoTagX applications

27 May - 4 June various email discussions

8 June Skype video call RH and SN

9 - 10 June RH and SN meet in London, two half-day meetings

17 June Skype video call with Einar Bjorgo, Lars Bromley, SN and RH, SN presents Yamuna’s Daughters project and plans for new project

28 June - 4 August various email exchanges between all collaborators, on co-design and on building the collaborative network of



the project

15 July Cobi Smith (CS) introduces herself to SN via email, CS and SN work together on Introduction for the Yamuna applications

3-4 August CS and RH meet and discuss project at CERN Summer Student Webfest, meet with S.P. Mohanty about Hindi translation of project introduction.

4-8 August CS and SN communicate by email with S.P. Mohanty for Hindi translation, as well as SN and CS finalising updated project description incorporating Hindi.

8 - 28 August SN visit in New Delhi Setting up the collaborative network for the new project with partners in New Delhi, including discussion of GeoTagX project and potential future applications of it in Delhi

15-17 September SN visit in Geneva, multiple collaborative meetings over three days



15 Appendix B: participant surveys developed in partnership with the University of Geneva

Citizen Cyberlab Survey 1: Background and Motivations 1) What is your age? a. 17 or under b. 18-24 c. 25-34 d. 35-44 e. 45-54 f. 55-64 g. 65-74 h. 75 or older 2) What is your gender? a. Male b. Female 3) What is your current occupation a. Voluntary work b. Student c. Full-time employed d. Retired e. Part-time employed f. Unemployed g. Other: 4) Do you have any previous knowledge of disaster management? a. I have studied disaster management b. I have no previous knowledge of disaster management c. My current occupation is related to disaster management d. I have read articles/books about disaster management e. I have studied disaster management at college f. Other: 5) How did you find out about the Geotag-X project? a. I was invited to join Geotag-X by a friend b. Via an internet search for disaster management c. I saw that my friend had joined the project and decided to join d. Other: 6) What do you hope to get out of participating in Geotag-X? a. I would like to make friends with other people interested in disaster management b. I would like to interact with disaster management workers and find out more about their work c. I would like to learn about disaster management and associated topics d. I would like to discuss ideas for crowdsourcing and disaster management e. Other:



7) Have you taken part in any similar projects before? a. No b. Yes (please describe) 8) What are your first impressions of the project? What do you like? What do you dislike? a. Free text comment box 9) Do you have any suggestions for how the Geotag-X website could be improved? a. Free text comment box 10) Thank you for filling in our survey! We will be selecting one person at random to win an Amazon gift voucher worth 10 Euros. If you would like to be included in our raffle, please enter your email address. Before and After Survey All participants are asked to “Please rate your level of knowledge and competency, where 1 is low, 2 is some, 3 is average, 4 is moderate, and 5 is high” unless otherwise stated. 1) Project Concepts and Mechanics: a. I understand how to use the applications on Geotag-X b. I understand how to find external resources to help me answer questions within the applications 2) Pattern Recognition a. I am able to recognise the relevant aspects within a photo that enable me to successfully answer the questions in the applications I have contributed to. 3) On-topic knowledge and skills a. I can explain what a disaster is b. I can explain what disaster management involves c. I am able to explain why the data that we are collecting in the different applications is important for disaster response d. I am able to describe the context and situation of at least on eof the disasters that I have worked on on Geotag-X 4) Scientific literacy a. I can list different types of data collected during field-work for disaster management b. I can explain why different types of data are collected during field-work for disaster management c. I can explain how scientific research is relevant to disaster management d. I have a better understanding of how scientists approach answering a research question 5) Off-topic knowledge and skills a. I am more skilled at using a computer b. I am able to use different Internet search tools 6) Personal development Please rate the extent to which you agree or disagree with the statements below on a scale of 1 to 5, where 1 is strongly disagree, 2 is disagree, 3 is neither agree nor disagree, 4 is agree, and 5 is strongly agree. a. I am confident in my ability to understand and categorise photos b. I am confident in my ability to contribute to disaster management and disaster response c. I have discovered other disaster-related topics that I am interested in



d. I find myself having new ideas about disaster-related topics e. I discuss disaster-related topics with other peope online/offline

Documents

Learning modules on media interpretation and disaster response data generation