AFFILIATIONS: Som de Cerff, van de vegte, BoerS, BrandSma, de Haij, van mooSel, noteBoom, Pagani, and van der SCHrier—Royal Netherlands Meteorological Institute, De Bilt, NetherlandsCORRESPONDING AUTHOR: Reinout Boers, reinout.boers@knmi.nl

The abstract for this article can be found in this issue, following the table of contents.DOI:10.1175/BAMS-D-17-0273.1

In final form 11 July 2018©2018 American Meteorological SocietyFor information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy.

The Agile Way of Working (AoW) was applied to three innovation projects at the

Royal Netherlands Meteorological Institute.

AGILE DEVELOPMENT IN METEOROLOGICAL R&DAchieving a Minimum Viable Product in a

Scrum Work Setting

Wim Som de Cerff, joHn van de vegte, reinout BoerS, tHeo BrandSma, marijn de Haij, Wim van mooSel, jan Willem noteBoom, giuliano andrea Pagani, and gerard van der SCHrier

R esearch and development (R&D) in the meteor- ological and climate sciences has until recently been organized along the preference and exper-

tise of the individual practitioner because scientific problems could be broken down into relatively small and isolated pieces, requiring only limited manpower to work on. Nowadays, such simple work practices can no longer be sustained. For instance, output from climate scenarios or early warnings of weather- and climate-related calamities need to be communicated to large and diverse interest groups with oftentimes

competing or even contradictory requirements. In brief, climate and weather R&D has evolved from a monodisciplinary exercise to a multidisciplinary enterprise. In such an enterprise, teams of scientists, software engineers, and science communicators work together with stakeholders so that products can be thoughtfully tailored for decision-makers and for the general public. This multidisciplinary work environ-ment requires innovative ways to organize the work practices around specific topics of interest.

The requirement to match the needs of interest groups and stakeholders to the output from science poses challenges to all participants involved (Briley et al. 2015), among others the development of a com-mon language and terminology. For example, the im-pact of climate change demands the development of climate services that in turn produce policy-relevant indicators (Goosen et al. 2014). Voinov and Bousquet (2010) give an overview of models that describe par-ticipation of stakeholders in collaborative decision-making. Traditional project management practices available to ensure project success are explained in that paper, although sometimes the differences between these management practices appear to be subtle. Their conclusions indicate that decisions are

2507AMERICAN METEOROLOGICAL SOCIETY |DECEMBER 2018Unauthenticated | Downloaded 04/01/22 12:47 PM UTC

more effectively implemented if the stakeholder bear-ing the consequences is involved from the start of the project. Early stakeholder involvement is particularly valuable in climate science and adaptation problems emerging from the interpretation of climate scenarios (Berkhout et al. 2014; Hegger et al. 2012). For example, Hegger and Dieperink (2014) describe a project where scientists, policy makers, and other stakeholders jointly worked on a plan to “climate proof” a deep polder in the western part of the Netherlands. In their paper they coined the term joint knowledge produc-tion to describe the early involvement of relevant interest groups other than the scientists.

Voinov and Bousquet (2010) also provide a ge-neric set of principles that ensure optimal project outcomes. They include a continuous awareness of and care for group dynamics, which is important in a setting involving people from diverse disciplines and backgrounds.

The purpose of this paper is to describe an in-novative way of working, called the Agile Way of Working (AoW) in a Scrum setting. In the AoW/Scrum, much attention is given to the manner in which projects are completed in a joint setting. Group dynamics are considered an important fac-tor in achieving successful project outcomes. There are important differences between Agile, or flexible project management (Wingate 2015), and traditional project management practices. In the AoW conven-tional project management tools are discarded and terminology and processes are different from tradi-tional management tools. For example, there are no Gantt charts, no time sheets, and no assignment of tasks to individuals, and there is a different project management approach in scope, resources, schedule, and quality. In Agile/Scrum development resources, schedule and quality are fixed. In a close relation with the stakeholder, the scope is adjusted as the product is developed. In traditional project management the project manager can trade between the constraints. However, Scrum processes are tightly choreographed and involve careful planning.

We demonstrate the usefulness of Agile/Scrum at a meteorological R&D department of the Royal Neth-erlands Meteorological Institute (KNMI). KNMI is a typical example of a national meteorological institute. It houses in one location R&D in weather and climate, operations of a network of automatic weather stations, an extensive department of software engineers and highly skilled technical staff, a weather room, a satel-lite department, and a communications department. Intensive exchange and interaction among staff is essential for meaningful weather and climate services.

The next section gives a short introduction of AoW. The following section describes the case studies on which the AoW was practiced for a short period at KNMI. Contrary to most papers describing R&D in meteorology, the final scientific results are only briefly addressed, since the emphasis of this study is on the process of going through AoW and its evalua-tion and not on the final scientific result. The method and experiences are then discussed in following sec-tions, respectively. Discussion and conclusions are dealt with in the final section.

THE AGILE WAY OF WORKING. The AoW presented here is an adaptation of the Scrum frame-work. Before explaining the AoW adaptation, Agile and Scrum as an Agile method will be explained. Agile as a concept was first described in the Mani-festo for Agile Software Development (Beck et al. 2001) and is based on 12 principles (Fowler and Highsmith 2001). The manifesto and principles originated from ideas and concepts arising in the 1990s as a reaction to more traditional approaches. These were regarded as bureaucratic, microman-aged, slow, and hindering creativity and effective-ness of software development. There are many Agile frameworks, but Scrum is the most commonly used. Scrum (Sutherland and Schwaber 1995) has a small set of clearly defined roles, processes, and artifacts as described in the Scrum guide (Schwaber 1997). Scrum is lightweight and simple to understand but difficult to master.

Scrum defines three roles (Schwaber 1997): 1) product owner (PO), responsible for the product vi-sion and setting priorities; 2) Scrum master (SM), to ensure that the Scrum process is optimally executed by the team and the PO; and 3) the team (three to nine people), responsible for performing the work and ensuring its quality. The team is granted full author-ity to do whatever it decides is necessary to achieve the goal (Schwaber and Beedle 2002). The PO creates a prioritized list of ideas for the product, called the product backlog. Ideas from the product backlog are broken down into smaller, manageable items by the team and PO. Development of the product backlog is a continuous process. During the sprint planning, the team pulls items from the top of the product backlog and decides how and how many items can be implemented in the next sprint: this becomes the sprint backlog. Together with the PO a sprint goal is defined. During the sprint the team meets every day in the daily Scrum. Within these stand-up meetings of at most 15 min, the team focus is on assessing prog-ress, asking three questions to each team member:

2508 | DECEMBER 2018Unauthenticated | Downloaded 04/01/22 12:47 PM UTC

Fig. 1. Scrum process: roles, meetings, and artifacts.

What have you done, what will you do today, and what are impediments to the work to be done? The SM keeps the team focused on its goal and removes impediments preventing progress. Sprints usually take between one and four weeks. Each sprint deliv-ers a deployable product, which is demonstrated to the stakeholders at the sprint review meeting. In ad-dition, a retrospective on the team’s work process is held. During a retrospective, interactions and tools are evaluated and discussed per theme: How well did the Scrum preparations go (Preparations); did the team act according to the Scrum rules (Scrum Conformity); how well did the team perform (Team Performance); and how well did the Scrum work for the project at hand (Apply Agile/Scrum)? Figure 1 summarizes the Scrum process, and Table 1 summarizes the specific terminology used throughout the document.

At KNMI Scrum is regularly and successfully ap-plied in software development projects. Therefore, it was decided to use an adaptation of Scrum for the implementation of the case studies to introduce this methodology to scientists. Adapting Scrum to a research environment has been done before (Hicks and Forster 2010; Ribeiro Lima et al. 2012; Barroca et al. 2015; Marchesi et al. 2007). In our adaptation of Scrum, we apply Scrum in a “pressure cooker” time frame of just two one-week sprints using truly multidisciplinary teams of scientists and software engineers, ignoring the common experience that teams usually take more than two sprint weeks to become effective. Within the two sprints the aim is to develop a minimal viable product (MVP), ready for use by the stakeholders and suitable for further development using a product backlog.

DESCRIPTION OF CAS E STUDIES . Preliminaries. At KNMI a small portion of time (two weeks per year per person) can be reserved for in-novative projects or research ideas. Following the observations strategy for KNMI (Boers et al. 2015), we decided to translate a number of themes from the strategy into exploratory case studies serving as road maps for future investigations. A list of case studies was composed in the year prior to the sprint weeks from which three were selected. The case studies re-quired integration of information/data from different parts of the organization.

To form the teams for the three case studies, three POs were appointed and asked to pitch the cases during a plenary session that was attended by the R&D department. After the three pitches, each department member could select the case of interest by adding his/her name to the case team. Sometimes a prospective team member was asked to switch teams because a team was lacking crucial skills. Below are short descriptions of each of the three case studies.

Drones for boundary layer applications (the drone team). KNMI’s observation strategy calls for ex-ploring the utility of drones in institutional R&D applications (Boers et al. 2015, their action 5). Deployment of drones could be useful for a) local thermodynamic profiling for nowcasting of extreme weather, b) siting classification of KNMI’s automatic weather stations, and c) probing the early morning boundary layer transition phase. The central aim of the sprint weeks was chosen as “f lying a drone to capture the boundary layer morning transition.” The two sprint weeks were broken down into a prepara-tion week (week 1), whereby a radiosonde was roped to a tethered balloon and its output directly relayed to the ground crew, and a final experiment week (week 2), in which a drone was f lown by a commer-cial operator at KNMI’s research station Cabauw (Monna and Bosveld 2013). In week 1, software was constructed to provide real-time transmission of output from the research tower and the radiosonde to ground crew and weather forecasters. This simu-lates a nowcasting situation in which forecasters would be able to make decisions based on incoming thermodynamic profile data. During week 2, the drone was f lown at Cabauw during a clear morn-ing, capturing the boundary layer morning transi-tion. From a technical standpoint, the Scrum was a success, as all objectives set out in the preparation weeks were accomplished. Figure 2 shows the drone f light at Cabauw.

2509AMERICAN METEOROLOGICAL SOCIETY |DECEMBER 2018Unauthenticated | Downloaded 04/01/22 12:47 PM UTC

Table 1. Terminology of AoW [abbreviated from the Scrum guide (Sutherland and Schwaber 2017) and Wikipedia]. The cartoons from Fig. 1 have been included when appropriate.

Term Explanation

Definition of Done (DoD)Scrum team members must have a shared understanding of what it means for work to be complete, to ensure transparency. This is written down in the DoD.

Deployable product A deployable product is the product that is delivered at the end of a sprint.

Focus, Stable Team values

MVPAn MVP has just enough features to satisfy early customers and to provide feedback for future product development.

Keep, Improve, Discuss

Structured method for a retrospective. Participants write down on Post-it notes what went well (Keep), what needs improvement (Improve), and what needs to be discussed (Discuss) about the Scrum process in the sprint. Lessons learned are written down, and one to three of them will receive focus for improvement in the next sprint.

Product backlog

The product backlog is an ordered list of everything that is known to be needed in the product. It is the single source of requirements for any changes to be made to the product.

Pair programming Programming by two people working together at a single computer.

POThe PO is responsible for maximizing the value of the product resulting from work of the development team and is responsible for the product backlog.

Retrospective At the sprint retrospective, the Scrum team inspects itself and creates a plan for improve-ments to be enacted during the next sprint.

ScrumScrum is a simple yet powerful set of principles and practices that help teams deliver prod-ucts in short cycles, enabling fast feedback, continual improvement, and rapid adaptation to change.

Scrum evaluation tools: Preparations, Scrum Con-formity, Team Performance, Apply Agile/Scrum

Themes to be discussed during a retrospective in order to gauge the quality of the complet-ed Scrum in relation to project goals, achievements, and team performance.

SM The SM is responsible for promoting and supporting Scrum as defined in the Scrum guide.

Sprint The heart of Scrum is a sprint, a time box of one month or less during which a Done, usable, and potentially deployable product is created. A new sprint starts immediately after the conclusion of the previous sprint.

Sprint backlog

The sprint backlog is the subset of product backlog items selected for the sprint, plus a plan for delivering the deployable product and realizing the sprint goal. The sprint backlog is a forecast by the development team about what functionality will be in the next deployable product and the work needed to deliver that functionality into a Done deployable product.

Sprint planning A sprint planning is a team session in which items from the top of the product backlog are discussed. During this session, decisions are made on how and how many items can be implemented in the next sprint; it results in a sprint backlog and a clear sprint goal.

Stand-up/daily Scrum The stand-up or daily Scrum is a 15-min time-boxed event held every day of the sprint at which the development team plans the work to be carried out in the next 24 h.

Story map

Story mapping consists of ordering user stories along two independent dimensions. The map arranges user activities along the horizontal axis in rough order of priority (or the order in which you would describe activities to explain the behavior of the system). Down the verti-cal axis, it represents increasing sophistication of the implementation.

Sprint review A sprint review is a meeting held at the end of the sprint to inspect the deployable product and adapt the product backlog if needed.

TeamThe (development) team consists of professionals who do the work of delivering a poten-tially deployable product at the end of each sprint.

User story A user story is an informal, natural-language description of a proposed functionality.

2510 | DECEMBER 2018Unauthenticated | Downloaded 04/01/22 12:47 PM UTC

Quality of manual rain gauge network (the quality team). KNMI operates a dense network of 325 manual rain gauges. The rain gauges provide daily sums of pre-cipitation made by volunteers. The network acts as a reference in the Netherlands, so it is necessary—in addition to the regular quality control—to moni-tor its long-term homogeneity with respect to other sources. In the Netherlands two other sources are 1) precipitation sums from the network of automatic weather stations (AWSs) and 2) sums derived from KNMI’s rainfall radars. For the sprint weeks the aim was to develop a tool to compare long-term monthly and annual precipitation sums from the three rain-fall sources using their mutual difference series. During the sprints, several kinds of functionalities were implemented. First, the precipitation sums from the AWS network and the radar data had to be aggregated to the same accumulation interval in use by the manual gauges. Thereafter, methods to apply statistical tests to detect divergences were added.

The sprints resulted in a system with working functionalities. However, owing to a lack of time, a fully automated chain of data aggregation and inho-mogeneity detection was not achieved.

A merging platform for divergent data streams (the data-wrangling team). Thanks to cheap sensing capabilities, companies and institutions are now able to quanti-tatively monitor (i.e., by sensing) the performance of their business processes. In particular, they are increasingly interested in combining their sensed data with high-resolution meteo-rological data. Data wran-gling is defined as “a process of iterative data exploration and transformation that enables analysis” (Kandel et al. 2011). To facilitate data wrangling of meteorologi-cal and nonmeteorological data, KNMI decided to de-velop a prototype platform that 1) enables nonmeteo-rological parties to make better use of the KNMI meteorological observa-tions, 2) eases the handling of complex meteorological data formats (for users out-side the meteorological do-main), and 3) speeds up the data preparation process.

A two-week work breakdown was organized with the goal of wrangling a dataset in plain text [comma-separated values (CSV) formatted] containing car and bike accident data with information about location and time together, with precipitation from radar ob-servation and temperature registered by the KNMI AWS network.

The two-week effort resulted in a web-based ap-plication able to successfully wrangle precipitation from radar in gridded format [Network Common Data Form (netCDF)], traffic accident data (CSV), and temperature data from the KNMI observation network (CSV).

APPLICATION OF THE AGILE WAY OF WORKING TO THE METEOROLOGICAL CASE STUDIES. Preparation weeks. Six to eight weeks prior to the sprint weeks, POs, none of whom had any experience with AoW, received a basic and general training from the experienced SMs on the principles and methodologies of AoW. The training focused on a) how to write project activities and goals in terms of story maps and user stories (see Table 1 for descriptive information of these terms), b) the Agile mentality, c) the role and tasks of the SM, and d) the role and tasks of the PO. A user story for a project is a statement of intent. It identifies a) in what capacity the user makes the statement of intent, b) what the purpose of the case study is, and c) what the activity is going to be. Table 2 shows the user story for each of

Fig. 2. Drone flight at the Cabauw Experimental Site of Atmospheric Re-search. Photo is taken from the top of the tower (210 m). A radiosonde is tethered at the end of a nylon cord. Note the curvature of the nylon cord due to the strong low-level jet in the early morning.

2511AMERICAN METEOROLOGICAL SOCIETY |DECEMBER 2018Unauthenticated | Downloaded 04/01/22 12:47 PM UTC

the three teams. Next, for each of the three case stud-ies, the PO, SM, and team worked together in several sessions to plan their programs of work by breaking them down into small tasks that could be executed by individual sprint team members. These sessions were essential for the PO to keep sharpening and refining the project ideas and prepare the user stories. In ad-dition, they assisted the teams to get a clear picture of the work involved in the case studies. Thus, resources could be planned and the feasibility of the wishes of the PO could be assessed well ahead of time.

Each team planned its work product backlogs, expected to be feasible in two one-week sprints, ac-cording to its particular case study.

The drone team faced limitations placed on flying drones and balloons over the Netherlands in civil and military air space. A full week was deemed necessary for a grand rehearsal of the drones experiment by us-ing a tethered balloon flown on-site at KNMI. Other tasks involved planning of software construction necessary to access the radiosonde profiles in real time, breaking down the protocol for each experiment (balloon and drone), securing manpower support for balloon and radiosonde launching, securing of an on-site radiosonde ground station, surveying the field of commercial drones operators, securing financial support for procuring their services and submitting flight plans to the selected drones operator, and air traffic control.

The quality team had identified the precipitation validation team of KNMI as the end user of their product (i.e., the stakeholder). They joined the quality team in several sessions to communicate their wishes relating to the graphical user interface (GUI) of the software and provide feedback on the proposed work-ing prototype. At these consultations it became clear that the original ideas of the POs were too ambitious. In breaking down the project into smaller tasks, the

POs tried to align the activities as much as possible with existing skills of the team members, which included reusing earlier work. Attention was also given to a common framework, such as a common scripting language.

The central task for the data-wrangling team was to construct architecture to be used as a skeleton for the wrangling platform; this included the diverse datasets to ingest, establishing approved function-alities (e.g., wrangle precipitation and temperature meteorological data only; support of just CSV for-mat), securing user work space, and developing user authentication protocols. Story mapping [Patton and Economy (2014), see their Table 1 for descriptive infor mation] and a synthetic biography of a facti-tious user of the product (Haikara 2007) were used to design the platform from the user perspective (e.g., upload data, selection of wrangling variables). User stories were refined and put on the story map to show priority of implementation for each phase.

The end result of the preparation weeks for each team was a product/sprint backlog consisting of a whiteboard with Post-its that described in detail the tasks to be executed during the first week of the sprint sessions (see Fig. 3 for one of these backlogs).

Sprint weeks. During the sprint weeks, work centered around the Scrum board in the project room (see Fig. 4 for an example). As Agile/Scrum does not de-fine how a Scrum board is organized, it is up to the team to set it up. The Scrum board shown in Fig. 4 was set up by one of the teams as a whiteboard with five columns: To Do, In Progress, Review, Test/Veri-fication, and Done. The To Do column contained as Post-its the list of tasks to be performed by individu-al team members as derived from the sprint backlog. The In Progress column was filled by Post-its shifted from the To Do column by individual team members

Table 2. Examples of user stories used in the sprint weeks.

Team As To We want

Drone team A boundary layer researcher

Understand the development of the bound-ary layer in the early morning (the so-called morning transition); we would like to obtain temperature–humidity profiles with high time resolution

A drone to fly several hours just after sunrise to probe the first 300–700 m of the developing boundary for temperature, mois-ture, and wind

Quality team A validation expert Monitor the long-term quality of the various precipitation-measuring systems

An automated tool that alerts when estimates of one precipitation-measuring system starts to deviate too strongly from the others

Data-wrangling team

A data scientist/analyst

Wrangle data from different data sources and formats

A platform/software application that wrangles data once sources and wrangle criteria are specified

2512 | DECEMBER 2018Unauthenticated | Downloaded 04/01/22 12:47 PM UTC

when they picked up their task. Once a team member deemed the task completed, the Post-its were placed onto the Review column, where they remained until the results were checked by a team member on qual-ity of the solution. Once reviewed, the Post-it shifted over to the Test/Verification column, where it re-mained until the PO verified whether the completed task conformed to the required functionality in the project. After verification by the PO, the Post-it was shifted to the final column, Done.

Each morning (typically 0900 local time) the team would gather around the Scrum board for a daily stand-up. Thereafter (most) team members reverted to work on their own tasks. In the event of blockages, part of the team continued discussing until problems were resolved or a clear avenue toward a solution could be mapped out.

Even though each team had been allotted a work space that could seat all team members, there were some team members who would rather work from behind their own desks, ar-guing that they could better focus (owing to the nature

of the tasks: writing, thinking) and combine the proj-ect better with other tasks in a more isolated setting.

At the end of the week, a sprint review meeting was organized during which a project demonstra-tion was given to the management and to the other teams. The project demonstration reviewed either the construction and workings of the deployable product (at the end of the first week) or the minimum viable product (at the end of the second week). Although the format of the sprint review varied from team to

Fig. 4. Example of a Scrum board, as in use during a sprint week.

Fig. 3. Example of a backlog in use, as developed during the preparation weeks.

2513AMERICAN METEOROLOGICAL SOCIETY |DECEMBER 2018Unauthenticated | Downloaded 04/01/22 12:47 PM UTC

team, it consisted mostly of a presentation containing a condensed flow of the work, preparation, measure-ments, visualizations, and conclusions.

Two of the three teams conducted the sprint review through a “Keep, Improve, Discuss” retrospective board allowing everybody to ref lect on the sprint using Post-it notes to be put in these three categories. Positive remarks were designated as Keep, such as knowledge sharing and new results. Impediments caused by routine work from other projects or lack of agreement whether tasks were actually completed to the satisfaction of the PO were noted as Improve. Designated as Discuss were 1) the main complaint that a full day was “lost” to demonstration, retro-spective, and planning for the next sprint and 2) the optional use for pair programming. After the sprint review meeting, at the end of week 1 the product backlog for the next week was prepared (sprint plan-ning), and at the end of week 2 a joint retrospective was organized in which all three teams participated. The joint retrospective contained a questionnaire inquiring team members on their experience and opinions about AoW.

The team room of the drone team was often manned by just one or two team members because many tasks required team members to be elsewhere on-site, or even off-site, preparing the various stages of the balloon and drone experiments. The absence of team members was on occasion an impediment to the work of the team members left in the team room.

Work in both the quality team and the data-wrangling team proceeded more in a way resembling a traditional Agile/Scrum work flow. Team members were mostly seated in the team room. The quality team on occasion required pair programming (see Table 1), where two team members were seated to-gether at a single workstation exchanging program-ming details, working toward a solution. Also, work procedures common to the experienced software engineers, such as using source code repositories, database protocols, and development tools for col-laborating on software code and making data avail-able for multiple usages and defining transferable data formats, were shared among the inexperienced data scientists.

Both teams spent a significant amount of time during these two weeks on the challenge to find common ground on the usage of tools and technical solutions and learning to make the ad hoc team work within the scope and in the interest of the project. The data-wrangling team had set up a chat channel using Slack (which is a cloud-based set of proprietary team collaboration tools and services), where links, ideas,

or questions could be posted when team members were working remotely.

EVALUATION OF THE SCRUM WORK PROCESS. Level of commitment of Scrum contribu-tors. The level of commitment of contributors in a Scrum project is one of the essential success factors. Even though team members committed themselves to one of the three case studies, it was difficult to secure their full commitment for the entire dura-tion of the sprints. Many had previous engagements that could not be postponed, so some tasks needed to be transferred to others in the middle of a sprint. Also, a few persons did not participate at all during the Scrums even though they had subscribed in the preliminary phase.

For the drone team it was often unclear to team members when and where their colleagues were pres-ent. Because of the many logistical arrangements and technical preparations in the laboratory/field, large parts of the work had to be performed outside the team room. Furthermore, team members apparently felt the need to get back to their routine activities whenever tasks were completed.

A limited level of commitment sometimes posed a problem in assuring the completion of tasks on time and within required specifications. Furthermore, it was hindering team building. During retrospectives, the flawed commitment was often heard as a high-priority point of improvement.

Multidisciplinary work environment and AoW. Forced multidisciplinary teamwork in a department more oriented to research poses a challenge. Scientists are by nature and training used to working independently. Therefore, a multidisciplinary AoW requires consider-able adaptation for scientists. For some of them it may perhaps be suitable only after a transition period or after adaptation to the Scrum rules. This is an impor-tant point to reflect upon for broad (i.e., company- or institution-wide) applicability. Nevertheless, close lines of communication facilitated by a joint presence in the team room were important to exchanging ideas and taking advantage of the skills of other team members.

Each team had its own multidisciplinary dynam-ics, summarized as follows:

• In the drone team the work and tasks were very precisely divided and could be executed indepen-dently; therefore, the interaction and the team dynamics were limited.

• In the quality team, the interdisciplinary as-pect of the case study, curiosity, new ideas, and

2514 | DECEMBER 2018Unauthenticated | Downloaded 04/01/22 12:47 PM UTC

discussions of the many scientists in the project were obstacles in the execution of the project. Often, discussions and points of view emerged on different approaches (e.g., database structure design) or on perfecting the software application (e.g., continuous integration tooling). The discus-sions helped the spread of new ideas between the scientists but posed a threat to the tight schedule of completion of the project.

• In the data-wrangling team the multidisciplinary aspect of the tasks was limited, since almost all the members were software developers. However, since they came from different work groups, the exchange between team members focused on the tools both to use and reuse previous knowledge acquired in projects that the members had par-ticipated in before.

Management and communication. The idea behind a Scrum and sprints is that a project team becomes self-directing. In other words, the PO is reduced to a one-man oversight committee, ensuring that the big-picture aims are being met, while the team members decide on the work partitioning and carry out the bulk of the work. The role of the SM is thus important to guarantee that the efforts of the team match the intentions of the PO and are channeled toward achieving an MVP. However, individual teams sometimes struggled with implementing the framework of AoW because of the inexperience of the POs with AoW and their interaction with the SMs.

For the drone team the PO took a directive stance, since the success of the experiment was dependent on the timely assistance of collaborators other than the sprint team members (i.e., commercial drone com-pany, military air traffic control, regulatory authori-ties, in-house radiosonde expert, balloon launchers, and Cabauw site manager and technicians).

The quality team determined that a more direc-tive stance of the PO would have been more useful to the success of the sprints, including handing down tips and tricks to junior researchers and guarding the scope of some of the software elements. Also, it was felt that too much time had been spent on fruit-less discussions between team members.

The data-wrangling team was the most success-ful in following the self-organizing principle of the Scrum method. This is perhaps not surprising, as most members of this team were experienced in AoW.

Ambitions vs realism. The extent to which project aims are being met depends to a large extent on whether

the required tasks are sufficiently well thought out and timed to match the specified project ambition. The articulation of realistic project ambitions by a PO experienced in an individualistic research environ-ment where success is measured by papers written and citations gained is difficult.

For the drone team, the aim was clear: “Fly a drone with real-time output available to ground personnel and the weather room.” Provided that adverse weather conditions would not prevent the experiment, this aim is focused enough to assign clear tasks and anticipate a satisfactory outcome. However, for the two other case studies, formulat-ing suitable tasks that could match the articulated ambitions was more difficult.

Achieving all the goals of the quality team project in two weeks’ time was not a realistic ambition. The scope of work was too big and lacked a clear break-down of activities and products.

For the data-wrangling team, wrangling data from different domains in different formats, resolu-tions, and criteria (quality) was simply not feasible in a two-week period. However, in this team the ambition was adjusted in the first week to provide a proof of concept and MVP that supported a limited set of data and formats to demonstrate the potential of a wrangling tool to speed up data preparation for analysis. The experience of team members working in a Scrum setting was instrumental in achieving the change of plans and ambitions.

Team retrospective results. After showing the team re-sults at the final sprint review, an all-team retrospec-tive was conducted to capture and share important lessons learned. Results were obtained using an online voting system accessible by mobile phone. After each question, the results were shown to the participants and discussed. The results are shown in Table 3.

Questions dealt with their opinions about the achievements of the Scrums, the preparation, whether the Scrum process was used, the value of the PO and SM, whether they liked working in a multidis-ciplinary environment, and whether they would like to do it again.

Preparations could be improved. The preparation of the user stories and Agile training was mostly done by the POs and SMs. The team members were involved at a later stage, 50% of whom had no Agile/Scrum experience. So it was felt by the teams that improved preparation (Agile training, team story refinements) should have resulted in improved esti-mation of the work to be done, leading to improved priorities and an improved MVP.

2515AMERICAN METEOROLOGICAL SOCIETY |DECEMBER 2018Unauthenticated | Downloaded 04/01/22 12:47 PM UTC

Regarding Scrum Conformity, and despite the unfamiliarity of team members with Agile/Scrum, the main Scrum roles, rituals, and artifacts were mostly used. Agile principles (Fowler and High-smith 2001) of “Focus” and “Stable teams” were experienced as important, as violations of these values were mentioned as significant impediments to success. The role of the PO was valued highly by more than 90%. Interestingly, while the role of the SM was also valued highly (>70%), there was a siz-able proportion of participants (~20%) that was not sure about the value of the SM. This could not be further investigated, as no differentiation of answers according to the teams was done.

Team Performance was regarded as high by the teams. They liked working in teams, and results obtained by the teams ex-ceeded expectations. Note that the team consisted of volunteers, so this result was expected.

The gained experience in applying Agile/Scrum was regarded as very positive by the teams. More than 80% of participants thought the achievements of the Scrums were good to excellent. Also,

66% of participants thought that AoW made their work more effective, although 33% of participants answered only “somewhat” to this question. All participants indicated they would participate again if a new Scrum session were organized the next year.

As a final feedback, participants could leave a Post-it note on the Keep and Improve f lip-over boards. The clustered results of the flip-overs are shown in Table 4. Suggestions for Improve were, among oth-ers, a) selecting team members based on their com-petencies and b) improving communication about an individual’s commitment and availability. Also, the loss of effective working time due to the morn-ing stand-ups, retrospectives, evaluation time, and demonstrations (in particular at the end of a sprint on Friday) was mentioned.

Table 3. All-team retrospective results: general experience. Note that the percentages do not necessarily add up to 100, as team members did not answer all questions.

Question Reply

Preparations

Experience with Agile/Scrum? 50% had none or limited (less than a year) previous experience

Preparation before sprints? 68% replied OK

Scrum Conformity

Followed the Scrum rituals? 31% fully, 37% best effort, 25% tried but made adjustments

Used the Scrum artifacts? 71% used them; 19% did not

Role of PO 100% agreement this role is needed

Role of SM 82% agreement this role is needed

Impediments? (Focus) 30% had trouble focusing because of “business as usual”

Team Performance

Sitting together as a team? 55% managed to sit together in the team room

Working in a team? 89% found this inspiring or nice

Team results? 60% replied very good, 18% exceeded expectations

Apply Agile/Scrum

Use it again in this setting? 44% yes, 39% with adjustments (6% no)

Use it in your normal work? 63% yes, 21% not sure, 10% no

Agile made you more effective? 66% yes, 33% somewhat

Scrum weeks in 2018? 100% yes, of which 40% wanted to expand to 3 weeks

Table 4. All-team retrospective results: what to keep, what to improve.

Keep Improve

Individual projects Team composition based on skills

Cooperation working on one goal Project diversity

Agile/Scrum approach More merging of the different disciplines

Focus Commit to availability

Teamwork All in or all out (no part-time team members)

R&D innovation weeks! Dedicate people to participate

Team rooms Preparation of the user stories

2516 | DECEMBER 2018Unauthenticated | Downloaded 04/01/22 12:47 PM UTC

DISCUSSION AND CONCLUSIONS. As work at KNMI takes on a more multidisciplinary approach, we examined AoW as a method to facilitate collabo-ration and coordination among scientists, software engineers, and other relevant personnel. To this end, three case studies were defined derived from KNMI’s observations strategy. The suitability of AoW for each project was tested in two sprints of one week each, and a number of observations can be made based on the outcome of this experiment.

AoW works best if the proposed work is to be achieved in a joint environment where the success of the project is dependent upon frequent exchange and interaction between team members. This was the case for the quality team and the data-wrangling team. For the drone team, most members had to work on indi-vidually assigned tasks that needed to be accomplished independently from other tasks and team members. However, the drone team did experience much ben-efit from the meticulous planning of the drone-flying experiment in the weeks prior to the sprints.

While AoW is often profitable in software devel-opment, its success in an R&D environment depends on the institutional ability to fuse the typical work practice of a researcher with that of a software en-gineer. Many researchers work individually. In fact, they have been educated and trained to do so, and an adjustment to work in a joint setting is not always beneficial to their productivity. During the sprints, a number of researchers felt it necessary to withdraw occasionally to the confines of their office to think and resume their traditional work practice. Therefore, a relaxation of AoW to a more flexible work environ-ment (rather than a joint setting) could be beneficial to institutions interested in engaging their personnel in AoW. Exploration of adapting Scrum or applying methods such as Kanban (Ahmad et al. 2013) may be considered.

In an R&D environment, the PO is often a re-searcher or research manager with little or no experi-ence in AoW. Therefore, in an R&D AoW setting, an experienced SM is needed to channel the efforts of both the PO and the team toward an MVP. During the sprints at KNMI, the experience and flexibility of the SMs were essential in adjusting ambitions of the POs to obtain an MVP in the appointed time.

Commitment from all team members is of vital importance to the success of the project. Team mem-bers need to feel engaged in the process, and frequent absences due to prior commitments or other pressing tasks are detrimental to the other team members. Lack of commitment was identified as the largest obstacle to team building. No doubt, efforts to ensure

commitment in the preparatory phase of the Scrum could be improved.

The answers to the questionnaire by the partici-pants showed that they rated the experience and the results of the AoW highly and valued the joint team-work toward a common goal. Nevertheless, they also felt that some aspects of the AoW, namely, the stand-ups, group demonstrations, and retrospectives, took too much time and could have a negative impact on the results of their team. However, these group processes are essential to the success of the Agile/Scrum method, and it is suspected that more exposure to Agile/Scrum would improve the understanding of the relevance of these processes to the success of projects.

The Scrums that were held at KNMI are but a relatively small effort involving a single department consisting of researchers, software engineers, and technicians (50+ people). Plans are under way to extend the AoW to projects involving more depart-ments for tests in a more complex institutional and programmatic setting. At the moment it is clear that no single method will suit our aims. Fortunately, Agile methods can be continuously improved and adapted to diverse teams, their varying tasks, and ever-changing circumstances.

ACKNOWLEDGMENTS. The case studies would not have been possible without the dedicated assistance of a number of people. We specifically mention Marc Allaart, who was essential in orchestrating the launching of the radiosondes; Marcel Brinkenberg for facilitating the drone launch at the research station of Cabauw; Netty Huisman, Frits van de Peppel, Jelis van der Berg, Jos van Dun, René van Aken, Hans Olminkhof, Karin Tukker, and Jeroen van Zomeren for arranging computer hardware and facilitating other infrastructure; Gijsbert Boon for facilitating the all-team retrospective; Fred Bosveld for his advice on organizing measurements to probe the boundary layer morning transi-tion; and Albert Klein Tank for his support for organizing the Scrum. Finally, we thank the team members who contributed to the sprint sessions: Jurian Baas, Else van den Besselaar, Cisco de Bruijn, Marieke Dirksen, Siebren de Haan, Henk Klein Baltink, Stephen Knoop, Susanne Kok, Michal Koutek, Hidde Leijnse, Jonas Matser, Andrej Mihajlovski, Ian van der Neut, Cor van Oort, Corné Oudshoorn, Maarten Plieger, Reinder Ronda, Andrew Stepek, Antonello Squintu, Maarten Terpstra, Rob Tjalma, Hans Verhoef, Ernst de Vreede, Lotte de Vos, Saskia Wagenaar, Wiel Wauben, and Ine Wijnant.

REFERENCESAhmad, M. O., J. Markkula, and M. Oivo, 2013: Kanban

in software development: A systematic literature

2517AMERICAN METEOROLOGICAL SOCIETY |DECEMBER 2018Unauthenticated | Downloaded 04/01/22 12:47 PM UTC

review. Proc. 39th Euromicro Conf. on Software Engineering and Advanced Applications, Santander, Spain, IEEE, 9–16.

Barroca, L, H. Sharp, D. Sala, K. J. Taylor, and P. Gregory, 2015: Bridging the gap between research and agile practice: An evolutionary model. Int. J. Syst. Assur. Eng. Manage., 9, 323–334, https://doi.org/10.1007 /s13198-015-0355-5.

Beck, K., and Coauthors, 2001: Manifesto for agile soft-ware development. www.agilemanifesto.org.

Berkhout, F., B. van den Hurk, J. Bessembinder, J. de Boer, B. Bregman, and M. van Drunen, 2014: Fram-ing climate uncertainty: Socio-economic and climate scenarios in vulnerability and adaptation assess-ments. Reg. Environ. Change, 14, 879–893, https://doi.org/10.1007/s10113-013-0519-2.

Boers, R., A. Klein Tank, T. Brandsma, G. Lenderink, J. F. Meirink, R. Sluijter, R. Sluiter, and W. Wauben, 2015: Observations strategy KNMI: 2015–2024. KNMI Publ. 233, 38 pp., http://bibliotheek.knmi.nl /knmipubmetnummer/knmipub233.pdf.

Briley, L., D. Brown, and S. E. Kalafatis, 2015: Overcom-ing barriers during the co-production of climate information for decision-making. Climate Risk Manage., 9, 41–49, https://doi.org/10.1016/j.crm .2015.04.004.

Fowler, M., and J. Highsmith, 2001: The Agile manifesto. Software Dev., 9, 28–32.

Goosen, H., and Coauthors, 2014: Climate adapta-tion services for the Netherlands: An operational approach to support spatial adaptation planning. Reg. Environ. Change, 14, 1035–1048, https://doi.org /10.1007/s10113-013-0513-8.

Haikara, J., 2007: Usability in Agile software develop-ment: Extending the interaction design process with personas approach. Agile Processes in Software Engineering and Extreme Programming, G. Concas et al., Eds., Springer, 153–156.

Hegger, D., and C. Dieperink, 2014: Toward success-ful joint knowledge production for climate change adaptation: Lessons from six regional projects in the Netherlands. Ecol. Soc., 19, 34, https://doi .org/10.5751/ES-06453-190234.

—, M. Lamers, A. Van Zeijl-Rozema, and C. Dieperink, 2012: Conceptualising joint knowledge production in

regional climate change adaptation projects: Suc-cess conditions and levers for action. Environ. Sci. Policy, 18, 52–65, https://doi.org/10.1016/j.envsci .2012.01.002.

Hicks, M., and J. S. Forster, 2010: Adapting Scrum to managing a research group. University of Maryland Dept. of Computer Science Tech. Rep. CS-TR-4966, 9 pp., www.cs.umd.edu/~mwh/papers/score.pdf.

Kandel, S., and Coauthors, 2011: Research directions in data wrangling: Visualizations and transformations for usable and credible data. Inf. Visualization, 10, 271–288, https://doi.org/10.1177/1473871611415994.

Marchesi, M., K. Mannaro, S. Uras, and M. Locci, 2007: Distributed Scrum in research project management. Agile Processes in Software Engineering and Extreme Programming, G. Concas et al., Eds., Springer, 240–244.

Monna, W., and F. Bosveld, 2013: In higher spheres: 40 years of observations at the Cabauw site. KNMI Publ. 232, 56 pp., http://bibliotheek.knmi.nl/knmipub metnummer/knmipub232.pdf.

Patton, J., and P. Economy, 2014: User Story Mapping: Discover the Whole Story, Build the Right Product. O’Reilly Media, Inc., 324 pp.

Ribeiro Lima, I., T. de Castro Freire, and H. Costa, 2012: Adapting and using Scrum in a software research and development laboratory. Revista de Sistemas de Informação da FSMA, No. 9, 16–23, www.fsma.edu.br /si/edicao9/FSMA_SI_2012_1_Principal_2_en.pdf.

Schwaber, K., 1997: SCRUM development process. Busi-ness Object Design and Implementation, J. Sutherland et al., Eds., Springer, 117–134.

—, and M. Beedle, 2002: Agile Software Development with Scrum. Prentice Hall, 158 pp.

Sutherland, J., and K. Schwaber, 1995: Business Object Design and Implementation. Springer-Verlag, 167 pp.

—, and —, 2017: The Scrum guide: The defini-tive guide to Scrum; the rules of the game. www .scrumguides.org/docs/scrumguide/v2017/2017 -Scrum-Guide-US.pdf.

Voinov, A., and F. Bousquet, 2010: Modelling with stake-holders. Environ. Modell. Software, 25, 1268–1281, https://doi.org/10.1016/j.envsoft.2010.03.007.

Wingate, L., 2015: Project Management for Research and Development. CRC Press, 517 pp.

2518 | DECEMBER 2018Unauthenticated | Downloaded 04/01/22 12:47 PM UTC