Embed Size (px)
Citation preview
Creativity in Software Engineering: A systematic
literature review
University of Oulu
Department of Information Processing
Science
Master’s Thesis
Ahmed Lasisi
May, 2016.
2
Abstract
The competitive nature of industries and emergence of newer technologies demands an
improving and creative approach in the development of software solutions. Conventional
approaches to software development are characterized by sequential stiffened procedures
with rigid management practices. These approaches limit the potential of alternative
creative processes by impeding the adoption of one's intuition, ability to explore,
knowledge sharing and collaboration among developers. Software Engineering (SE) is a
complex domain that requires intense creative activities for its sustainability; hence a firm
understanding of creativity is in need. Creativity is a cognitive activity that is influenced
by people, process, product and place.
The study identified 89 primary studies through systematic literature review conducted to
analyze, synthesize and aggregate evidences regarding the state-of-art on research of
creativity in software engineering with the aim of addressing the techniques, tools, factors
and measurement of creativity in software engineering domain.
The findings revealed that the requirement and design phases of software development
are more important stages in the development process where design decisions are made
and creativity is much emphasized. Factors categorized as extrinsic and intrinsic factors
are said to influence creative practices in SE domain. The adoption of creativity
techniques such as collaboration, knowledge sharing and brainstorming with appropriate
lightweight creativity supporting tools were identified to influence creative practices in
software engineering domain, while its evaluation is difficult and subjective.
In conclusion, the study advocates the adoption of creative practices with flexible
management style in software development towards the building of effective and timely
software. However, assertive adherence to traditional practices in software development
could hamper creative practices. There exists no ultimate solution to creativity, beyond
the harnessing of human cognitive capability. Hence, the management of software project
should seek the development of the creative mind (engineers) with an embracing
management practices for continuous deliverance of effective and novel solutions. The
findings could impacts decision making of stakeholders in software engineering domain
towards harnessing of creative practices.
Keywords: Creativity, software engineering
Supervisor: Rahul Mohanani.
Co-Supervisor: Prof. Burak Turhan
3
Foreword
All gratitude is due to almighty Allah. My sincere appreciation goes to my family, friends
and especially my wife for her patience and understanding during my study. The support,
guidance, critics and commitment from Rahul Mohanani and Prof. Burak Turan was
appreciative and commendable. I say a big thank you to the University of Oulu, for the
wonderful platform provided in actualizing this research work.
4
Contents
Abstract ............................................................................................................................. 2
Foreword ........................................................................................................................... 3 1. Introduction .................................................................................................................. 5 2. Background Studies ...................................................................................................... 9
2.1 Software engineering ........................................................................................... 9 2.2 Creativity and software engineering .................................................................. 10
3. Systematic Literature review ...................................................................................... 15 3.1 SLR… ................................................................................................................ 15 3.2 Planning Review ................................................................................................ 16
3.3 Conducting the Review ...................................................................................... 16 3.3.1 Search strategy and data sources ............................................................ 16 3.3.2 Selection of primary studies ................................................................... 19 3.3.3 Study selection procedure ....................................................................... 20
3.3.4 Pilot Testing ............................................................................................ 21 3.3.5 Quality Assessment ................................................................................ 22 3.3.6 Data Extraction ....................................................................................... 24 3.3.7 Data Synthesis ........................................................................................ 25
4. Results ........................................................................................................................ 26 4.1 To what extents has creativity been studied in SE (RQ1)? ................................ 26
4.1.1 Publication Distribution per year ............................................................ 26
4.1.2 Creativity Definition ............................................................................... 27
4.1.3 Analysis and Discussion ......................................................................... 28 4.1.4 Study Context ......................................................................................... 29 4.1.5 Creativity distribution in SE phases ....................................................... 30
4.1.6 Analysis and Discussion ......................................................................... 31 4.2 What are the techniques used in introducing and enhancing creativity in SE
(RQ 2)? .............................................................................................................. 33 4.2.1 Results 33 4.2.2 Analysis and Discussion ......................................................................... 35
4.3 What are tools used in introducing/enhancing creativity in SE? ....................... 37
4.3.1 Analysis and Discussion ......................................................................... 39 4.4 What are the factors that influence creativity in SE (RQ 3)? ............................ 41
4.4.1 Results 41 4.4.2 Analysis and Discussion ......................................................................... 42
4.5 What are the ways of measuring creativity in SE (RQ4)? ................................. 45 4.5.1 Results 46 4.5.2 Analysis and discussion .......................................................................... 47
5. Discussion .................................................................................................................. 48 6. Conclusion .................................................................................................................. 51
6.1 Research implication .......................................................................................... 52 6.2 Validity threats and Limitations of study .......................................................... 53
References ....................................................................................................................... 54
Appendix A: Lists of Accepted Primary studies ............................................................. 61
5
1. Introduction
Human cognition and information are basis for creativity (Amabile, 1983; Amabile and
Gryskiewicz, 1989). Creativity is endowed in every being which can be developing
through social collaboration and practices, and forming relationship between unconnected
entities and events (Henry, 2006). The steady inventions in different disciplines are all
dividend of human creativeness in technological manifestation. Human encounter diverse
challenges throughout their life and they endeavor in resolving them through diverse
thinking and collaboration, in minimizing the problem gap. In similar approach, software
engineers’ seeks creative approaches in the development of quality and cost effective
software solutions (Rus and Lindvall, 2002.)
According to Amabile (1983), creativity is the production of novel useful ideas. Creativity
is observed as the basic processes relating to production of novel and functional ideas by
an individual or group of individuals working together (Amabile, 1996;
Csikszenthmihalyi, 1999). Several research theories have attributed creativity largely to
novelty and useful ideas generation towards a problem solution (Amabile et.al, 2005).
Novelty is seen as the distinguishing factor for creativity (Amabile, 1983) which depends
on cognitive variation within the creative mind, hence creativity might increases as
cognitive elements varies (Nguyen and Shanks, 2008; Amabile, 2005.)
Creativity could be of diverse complex forms that exist within a variety of contexts such
as social and educational contexts (Kozbelt, Beghetto, Runco, 2010; Adams, 2005).
Creativity has been a theme of research in disciplines such as; management (Crawford
et al., 2012), human cognition (Hayes, 1989), Social and organizational psychology,
(Amabile, 1983; 1988; Amabile & Gryskiewicz, 1989), arts (Zimmerman, 2010) and
pedagogy (Arbaoui et al., 1999) longer than in software engineering (Maiden, Jones,
Karlsen, Neill, Zachos and Milne, 2010). Creative idea is a product of divergent
thinking but excess thinking without conservation prevents relevant thoughts and
promotes shifts from useful subjective focus (Kozbelt, Beghetto, Runco, 2010).
Creativity is a cognitive activity and it’s influenced by processes like emotion and mood
(Amabile, Barsade, Mueller and Staw, 2005). It is regarded as an essential approach in
managing the challenges of software development problems (Highsmith and Cockburn,
2001.)
Software engineering is a field that requires in-depth process understanding (Crawford
and Barra, 2007) that is determined by the input of human creativeness, comprehension
6
and reasoning (Florida, Knudsen, Stolarick and Youl Lee, 2006). Software engineering is
a collaborative task, which encourages knowledge synergy from diverse discipline (Walz,
Elam and Curtis, 1993; Tiwana and Keil, 2004). The sporadic changes in Information
technology field, demands the creativeness of software engineers (Fischer, Boogaard and
Huysman, 1994) in providing quality and cost effective solutions. Creativity is recognized
as an important factor in software engineering (Mich, Anesi and Berry, 2004; Graziotin,
2013). The human knowledge and her creative potentials are key factors for software
development (Amin, Basri, Hassan and Rehman, 2011). Agile software development
methodologies such as Extreme encoding (XP), scrum practices do foster creativity in
software development (Sutling, Mansor, Widyarto, Letchmunan, and Arshad 2014). In
management phase of software development, the project manager is encouraged to adopt
agile methods in addressing development process challenges and to foster creativity
(Sutling, Mansor and Widyarto, 2014). In software testing, exploratory testing has been
identified to depend on human cognition and supports creativity (Shoaib, Nadeem, and
Akbar, 2009). Several researches have investigated creativity in requirement engineering,
with emphasis on framing requirement as a creative solving process (Maiden et, al., 2010)
and in software design (Akin, 1990; Ball, Onarheim & Christensen, 2010; Díaz et al.,
2014). Requirement engineering encourages the adoption of creative techniques as an
important tools for user’s requirement content clarity and implementation (Saha, Selvi,
Buyukcan and Mohymen, 2012; Maiden, Jones, Karlsen, Neil, Zachos and Milne, 2010;
Crawford, Barra, Sot and Monfroy, 2012; Nguyen and Shanks, 2008; Lemos, Alves,
Duboc and Rodrigues, 2012). Software engineering practitioners and researchers have
considered creativity as a key success factor (Maiden, Jones, Karlsen, Neil, Zachos and
Milne, 2010; Graziotin, 2013), in which the consequences has been beneficial at large in
terms of cost, quality and satisfaction.
Software’s are complex systems, essentially developed by human for human benefits
(Crawford et.al, 2012). Human needs are reflected in engineering requirement that is
translated into satisfactory products through creativity (Hoffman, Cropley, Cropley,
Nguyen and Swatman, 2007). Attempt to comprehend creativity requires a round
knowledge of it, which will provide an improved process of software development for
effective solutions to complex issues (Graziotin, 2013).
The study decipher creativity from innovation, as creativity is described as the generation
of novel and useful idea(Amabile, 1983; 1988) while innovation is regarded as the
application of creative outcome (Maiden, et. al., 2010). The research focuses more on
7
creativity in software development with emphasis on its introduction, impacts, factors and
measurement. In accordance to the notion that software engineering challenges demands
creative thinking in providing effective solutions (Fuggetta, 2000; Crawford, Barra, Soto
and Monfroy, 2012; Gu and Tong, 2004), there has been no adequate empirical evidences
regarding software development as a creative activity (Maiden and Gizikis, 2004;
Graziotin, 2013). Recently, there has been several researches focused towards the
essential of creativity in requirement engineering(Robertson, 2002; Maiden and Gizkins,
2001; Lemos, Alves, Duboc and Rodrigues, 2012) and software design (Akin, 1990; Ball,
Onarheim & Christensen, 2010; Díaz et al., 2014) but there is need to understand what
creativity is in software development from the viewpoint of software engineering (Wen,
Zhang, Liu and Yang, 2010; Graziotin, 2013; Adams and Trapp, 2015.)
Having acknowledged the importance of creativity in software engineering (Henry,
2006; Graziotin, 2013; Maiden and Gizkins, 2001), this motivates the present research
in conducting a systematic literature review to understand the current state-of-art
research on creativity in software engineering domain. A systematic literature review
was adopted according to (Kitchenham, Brereton, Budgen, Turner, Bailey, and
Linkman, 2009) guidelines, to analyze, evaluates, synthesize and classify scientific
literature regarding creativity in software engineering. The summarized evidences from
the literature review will be used to answer the following research questions:
R1: To identify the extent in which creativity has been studied in Software
Engineering?
R2: What are the techniques used in introducing and enhancing creativity in
software engineering?
R3: What are the tools used in introducing and enhancing creativity in software
engineering?
R4: What are the factors that influence creativity in software development?
R5: What are the ways of measuring creativity in software engineering?
The deductions from the literature reviewed could be of importance to practitioners
willing to adopt creative practices in development of novel and useful software solutions
that could measure up with the emerging technology. Also, it could provide insights to
researchers on new research areas and opportunities for exploration.
This study was organized as follows: In chapter 2, the background review of the study
was discussed. Chapter 3 describes the adopted method used in systematic literature
review, this entails the review protocols, inclusion and exclusion criteria, quality
8
assessment and data extraction from the selected primary studies. Chapter 4, gives the
results, analysis and discussion of the synthesized themes resulting from study research
questions. Chapter 5 was dedicated for discussion in comparing key findings with
background review and finally chapter 6 presents the conclusion, research implication
and validity threats and limitations of study.
9
2. Background Studies
Software engineering has been recognized in Software Engineering Body of Knowledge
SWEBOK, which provides content agreement on software engineering discipline
(SWEBOK, 2004). The following terms were defined according to the IEEE Standard
glossary of software engineering terminology (1990.)
Software engineering: “(1) the application of a systematic, disciplined, quantifiable
approach to the development, operation, and maintenance of software; that is, the
application of engineering to software. (2) the study of approaches as in (1)” (IEEE.
1990.)
Creativity According to Amabile (1983), “creativity is production of novel useful ideas”.
Innovation is regarded as the breakthrough in creative solution (Adams, 2005.)
The above definitions were considered to make a clear distinction in the present research
that the research work is focused towards software engineering with emphasis on
creativity and not innovation.
2.1 Software engineering
Software development comprises of activities that involve collaboration of different
individual at every phase of its development (Kusumasari, Supriana, Surendro, and
Sastramihardja, 2011).The tasks of software engineers is large and complex (Ackerman,
2014), such that it requires the collaboration and contributions of varied expertise from
different background(Gandomani, Zulzalil, Ghani, Sultan, and Sharif, 2014). Software
systems are essential, such that parts of its failure in critical systems can results in life
threatening situation, though various software failure has been recorded in the past
(Baxter and Sommerville, 2011). Software has been found influential in all phases of our
daily lives and also to larger extent become vital component of many industries (Herbsleb
and Moitra, 2001). The basis of software engineering entails the development of software
in addressing the issue of producing viable solutions to sets of functional requirement
(Cowling, 2004.) Software systems are complex abstract systems with high failure rate
(Kusumasari, Supriana, Surendro, and Sastramihardja, 2011). Diverse methodologies,
approaches and techniques have been adopted to facilitate the development of software
solutions, in which some has been less widely accepted than expected (Bourque, Dupuis,
Abran, Moore, Tripp and Wolff, 2002).
10
Software development entails development of software in a planned and organized
approach. The development of software does gradually emerges from the classical
waterfall model (Royce, 1970) of stable non overlapping requirement to the present
evoking agile practices that encourages collaboration among stakeholders(Crawford et
al., 2012; Hollis & Maiden, 2013; Sutling et al., 2014) in the development of creative
software. Boehm, 1988, proposes the Spiral model of software development to
compensate and improve the inefficiency of the traditional waterfall model. Many
software engineering projects has resulted in failure (Kusumasari, Supriana, Surendro,
and Sastramihardja, 2011) and design creativity are hindered due to fixation in
requirements (Mohanani, Ralph, and Shreeve, 2014), higher outcome expectation and
lack of creative process in the design process (Baxter and Sommerville, 2011). Hence,
the demand for flexible collaborative approaches in software development could be
embraced (Fraser et.al, 2013.)
2.2 Creativity and software engineering
According to Amabile (1983), creativity is production of novel useful ideas. Novelty is
seen as the distinguishing factor for creativity (Amabile, 1983) which depends on
cognitive variation within the creative mind (Amabile, 2005). According to (Mumford
and Gustafson, 1998) the dimensions of creativity require the generation of novel and
innovative products, and such products must be of social relevance. Creativity was
considered to be an output of an unplanned or sudden occurrence resulting in valuable
output (Boden, 2004; Nguyen and Shanks, 2009). Also, creativity does not necessarily
depicts something new but could results from the combination of existing product or
process (Crawford et al., 2012). It is defined as an internal process of exploration and
transformation of the conceptual space in an individual mind (Boden, 1991; 1998).
Creativity is describe as persuasion, such that person could be persuaded to be adopt
creative practices (Amabile, 1990). Creativity may be understood using a set of
dimensions including creativity elements (product, process, people, domain and socio-
organisation), creativity levels, and creativity loci (Nguyen and Shanks, 2009.)
Contextual creativity is related to the interaction of four elements (process, product,
person and place) that are referred to as "four P's" (Kozbelt, Begheuo and Runco, 1990;
Amabile, 1983; 1989). Categorical classification of creative phenomenon can limit and
complicate creativity quality (Kozebelt, Begheuo and Runco, 1990). Creative
environment does focus on fostering creative behavior such that the norms and value of
such climate promotes creativity (Couger, Higgins and McIntyre, 1993) hence
11
environmental variation thus influence creative output. Creativity occurs when the
associative barriers that exist between disciplines and knowledge are broken down, hence
giving rooms for intersection of both (Simonton, 1999; Simonton, 2003). Establishing
consistency and developing an in-depth knowledge of a certain subject domain could
form a basis for creativity in such domain (Amabile, 1996, 1999; Adams, 2006;
Johansson, p.104, 2004). Creativity output could increase with active time spent on a
given domain (Adams, 2006). According to Sternberg (Handbook of creativity, p. 256,
1999), key aspects of intelligence associated with creativity are synthetic, analytical and
practical, which were basis for creative thinking processes.
Cognition is an important factor for divergent thinking, which is a great construct for
creativity (Kozebelt, Begheuo and Runco, 1990; Couger, Higgins and McIntyre, 1993;
Butler and Scherer, 2003). Creative process is an internal process that is difficult to
understand, hence if such process could be understood, describe and model; it will
enhance and facilitate creativity (Boden, 1991). Creative thought could be stimulated by
directing one’s thought towards more desirable solution by strengthen one’s internal
positive thought , weaken an existing negative approach and focusing on adding new
positive concepts (Couger, Higgins and McIntyre, 1993). An approach in redefining
problems, seeking for unique idea, consistency and focus, and improving one’s intrinsic
motivation does enhance creativity (Akin, 1990; Adams, 2006). Creativity could be
stimulated by encouraging individual of their inborn creative knowledge (Amabile,
1983). Intrinsic motivation such as primary interest, satisfaction and challenges enhances
creativity unlike external pressures which are extrinsic motivation that may impede
creativity (Amabile, 1996). Motivating individual also improve their creativity (extrinsic)
(Amabile, 1989; Couger, Higgins and McIntyre, 1993). Nurturing creative potentials by
engaging in targeted program and approach can enhance creativity. Individual’s state of
mind and environment does influence one’s creative potentials (Amabile, 1983; 2005). A
positive mind improves one’s cognitive ability; focus the mind towards a cognitive
context and increases divergent thinking that alleviate positive performance leading to
creativity (Amabile, 2005; Simonton, 1999). A positive mental activity enhances creative
thinking. The paradox about mood is that people tends to show more commitment when
having a negative feeling in trying to solve a task, conversely, they stop working when
they experience positive mood. Emotional state of human, both negative and positive may
affect task performance as people tends to focus on emotion inducing situation which
12
reduces innovative cognitive thinking (Frijda, 1986; Weiss and Cropanzano, 1996;
Martindale, 1999.)
Adopting creativity approaches in the development of solutions that is novel and useful,
requires human cognition and social collaboration within a designated domain and
context (Nguyen and Shanks, 2009). Human are relevant in creativity because people
have diverse knowledge in different domains through which idea is could be generated
(Kozebelt, Begheuo and Runco, 1990; Tardif and Sternberg, 1997). The combination of
different information could results in Creative Problem solution (CPS) techniques, which
could reveal different idea that are essential views of creativity (Kozbelt, Begheuo and
Runco, 1990; McFadzean, 1998). Brainstorming among other techniques is widely
considered to induce creativity (Floyd, Mehl, Reisin, Schmidt and Wolf, 1989; Maiden,
Manning, Robertson and Greenwood, 2004; Schumann, Shih, Redmiles and Horton,
2012). (Amabile 1983; 1989; Csikszentmihalyi, 1988), deduced that the perception of
creativity emerges from three interacting components; domain knowledge, individual and
field of expertise. Human knowledge and domain expertise are said to have greater impact
on person’s creative potential (Amabile, 1996). In design task within a designated
domain, in-depth domain expertise (Mahaux, Nguyen, Mich & Mavin, 2014), including
creative thinking and intense knowledge (Aleven, Ashley, Lynch & Pinkwart, 2008) thus
foster creativity. In contrast, domain expertise could also inhibit one creative potential by
limiting their exploration capability and settling for prior knowledge (Kilgour, 2006.)
Creativity is a unique concept that cannot be undermined in software engineering (Glass
and Vessey, 1994). In software engineering, it has been identified as the potential driver
for market sustenance; however research addressing the subject is still minimal in
compensating for the challenges of the emerging competitive market (Amin, Basri,
Hassan, and Rehman, 2011). Industries are focusing more on creativity research to enable
them explore new approaches to software development, which might results in creating
unique and higher quality software products (Gu and Tong, 2004). The development of
quality and efficient software in achieving desired cost, quality and time is becoming
expensive and more challenging (Elberzhager, Munch and Nha, 2012). Domain
knowledge and skills are required in developing successful software; hence software
engineers should strive to possess firm domain skills and creativity (Graziotin, Wang and
Abrahamsson, 2014).
13
Software development are complex and knowledge intensive task (Crawford et.al, 2012;
Hedge and Walia, 2014; Ackerman, 2014), that require developers communication;
planning and strategies in generating creative and innovative ideas since no single
developer possess absolute knowledge in completing a task (Crawford, Barra, Soto, and
Monfroy, 2012). Collaboration of different individual could results in creative idea
generation (Kozbelt, Begheuo and Runco, 1990). Collaboration techniques have been
adopted in software engineering in achieving convergence of single architectural design
and in establishing project scope (Whitehead, 2007; Lanubile, Ebert, Prikladnicki, and
Vizcaíno, 2010). Open communication resulting from collaboration between employer
and employee could enhance employee motivational level (Tolfo & Wazlawick, 2008;
Amabile, 1996). Collaborative effort within team of varied interests and interdisciplinary
study promotes creativity (Adams, 2006). Factors such as collaboration , communication,
knowledge sharing, divergent and convergent thinking and brainstorming are said to
induce creativity (Aleven et al., 2008).Creativity is an essential concept for the
exploration, construction and expansion of the problem domain (Amabile, 1999). Team
of software designer’s with cross-cultural dimension is regarded as a potential for
creativity due to the differences in people's perspective and knowledge coming together
as knowledge synergy (Nonaka, and Takeuchi, 1995; Dalberg, Angelvik and Fossberg,
2006; Petrillo, Pimenta, Trindale and Dietrich, 2008). Traditional software development
process such as waterfall model focus majorly on developer’s view and separates
development into phases ranging from requirement to testing in which, each phase has a
unique objectives and are performed iteratively(Royce, 1970; Ramamoorthy and Tsai,
1996). Traditional approach of software development, which entails repeated and
mechanical procedure does inhibit developer’s avenue to explore and collaborate, unlike
the second generation method which doesn’t indicate definite pattern, process or
framework of addressing development issue, thus foster designer creativity (Fallman,
2003). Agile methodologies; eXtreme Programming, Scrum of software development
emphasizes communication and collaboration among stakeholders during software
development and has proven to be cost and time effective (Letchmunan and Arshad,
2014). One of the strong feature of agile software development process is enabling open
communication, which promotes knowledge sharing and enhance creative practices
(Tanaka, Nakamura and Takemata, 2009; Letchmunan and Arshad, 2014).
Communication aids clarification of concepts among stakeholders. Agile method of
software development encourages stakeholders in facilitating periodical meeting in
14
addressing requirement modifications and solution proposal in shorter developmental
cycle hence inducing creativity (Maiden et.al, 2012).
According to Amabile, 1983, quantifying creative output is a subjective concept. One of
the proposed ways of creativity measurement is to subject individual to certain experience
(Smith, Bird and Zimmerman, 2015). Creativity evaluation is complex and difficult to
attain in shorter time frame (Weiley and Edmonds, 2011). One of the metric that could be
used to ascertain creativity measurement in software project, is by observing the quantity
of features or functions added per each release (Paulson, Succi and Eberlein, 2004)...
Working as a team foster higher creativity in software development project rather than as
a single individual thus; there is improvement in performance as a team over individual
(Salas, Sims and Burke, 2005; Fagerholm, Kettunen, Munch, Roto and Abrahamsson-
2014). Empirical study conducted in requirement engineering points that creativity in
industries in associated with group collaboration in developing task that cannot be created
by single member (Mahaux, Nguyen, Mich and Mavin, 2014).
Despite of the overview of published literatures on creativity and in relation to software
engineering, there is still need for broad overviews of creativity in software engineering
domain. The motivation for this research was to give a more thorough and broader
overview by conducting a systematic literature review and gather findings in answering
the proposed research questions.
15
3. Systematic Literature review
This chapter entails the adoption of a systematic literature review based on the guidelines
proposed by Kitchenham (2004) with emphasis on search strategy, selection process, data
extraction, synthesis, and reporting.
3.1 SLR
The adopted approach is used to identify, evaluate and aggregate all available evidences
relevant to the study research questions. The implication of the findings, could be adopted
by practitioners in making creative decisions towards improving software engineering
domain. The aim of the study aligned with the goal of evidence-based software
engineering; “To provide the means by which current best evidence from research can be
integrated with practical experience and human values in the decision making process
regarding the development and maintenance of software” (Kitchenham et al., 2004;
Dyba, Kitchenham and Jorgensen, 2005). The below figure 1, represents the outlines of
the research process that has been adopted in this research:
Need for Systematic Review
Define research questions
Develop review Protocol
Pilot selection and Extraction
Primary study selection
Data extraction
Study Quality Assessment
Data synthesis
Thesis documentation
Plan
nini
ngCo
nduc
ting
Repo
rt
Figure 1. Systematic review outlines (Kitchenham & Charters, 2007)
16
3.2 Planning Review
The planning phase entails stating the motivation for engaging in SLR, which facilitate
the review protocol. The approaches used to conduct systematic review are specified by
the review protocol and thus decreases the possibility of research bias (Kitchenham,
2004). A review protocol contains the need for the systematic review, research questions,
search strategy including search terms and data sources, inclusion and exclusion criteria,
quality assessment procedures, data extraction strategy and data synthesis strategy. The
conducted background study on literatures of creativity does widen the researcher
knowledge as regards creativity and thus help to define the review protocol component.
Literature authored by renowned psychologist Amabile et al., (1983; 1993; 1996),
management (Crawford et al., 2012) and human cognition (Hayes, 1989) were reviewed
for wider insight on creativity.
3.3 Conducting the Review
This phase is a follow-up phase from the agreed protocol above and it entails stages that
precede one another, with the aim of identifying relevant primary studies related to the
proposed research questions. It is a stepwise implementation of the review protocol.
3.3.1 Search strategy and data sources
The time period for the searches was documented to compensate for variation in
replication results and to allow research replication. The selected electronic databases for
the study are:
● Scopus
● IEEE Xplore
● ACM
● Science Direct
The justification for the selected databases results from the deduction that they are widely
use in software engineering domain and allows advance query search (Barney, Petersen,
Svahnberg, Aurum, and Barney, 201). Also, most SLR conducted in software engineering
domain is considering top reference databases such as; IEEE Xplore, ACM, Scopus,
ProQuest, Science direct for data retrieval. Publications submitted to these databases are
subjected to rigorous review before being accepted, which is an indication for quality. In
similar manner, the researcher consulted the University of Oulu library in order to choose
17
relevant databases and for the know-how on keywords combination and strategies for
making meaningful searches towards the current study. The constructed search string
created by combination of keywords deduced from the study research questions was
used in probing the selected software engineering databases. The search string was
created based on the procedure proposed by (Kitchenham & Charters, 2007). Also, for
broader and precise searches among similar terms, the use of Boolean function was
employed. The Boolean AND operator is to link findings of two search terms to be more
specific while the Boolean OR operator is for wider search among common terms. The
search string structure according to (Kitchenham & Charters, 2007) in terms of population
and intervention were specified (Population and intervention) see table 1 below.
Table 1: Search keywords
Population Intervention
“software engineering” Creativity
The search string is a combination of search terms that represent population and
intervention. The constructed keywords were combined with Boolean AND operator to
probe the selected databases except for ACM database that differs in search approach.
The query functionalities and operation for reference retrieval varies for each database,
though same search strings was used across all the selected databases. The figure 2 below
depicts the search strategy process that was followed in the current study in the
identification of papers in table 2 below.
18
Figure 2: search strategy, adapted from (Kitchenham, 2004)
The table 2 below presented retrieved papers and corresponding databases including
date and time of retrieval. The constructed search string returned 8872 papers that were
managed with help of a reference manager. A Refworks management tool has the
potential of integrating and synchronizing references from databases. The search results
from individual searched databases were exported directly to refworks, where folder
was created for search results from each database and later combined in single folder for
easy management. Exporting from ACM database to refworks was really tedious and
rigorous. Also, the selected databases do have varied exporting functionalities.
19
Table 2: Selected databases, retrieved papers, date and time.
Database Search Query Date/Time Papers
Scopus “creativity AND software
engineering”
4.10.2015 @17:29 2695
IEEE Xplore “creativity AND software
engineering”
27.9.2015 @ 02:15 3268
ACM “creativity AND software
engineering”
27.9.2015 @ 18:24 1385
Science
Direct
“creativity AND software
engineering”
26.9.2015 @ 19:59 1524
Total 8872
3.3.2 Selection of primary studies
A set of inclusion and exclusion criteria were specified based on the review objectives to
ascertain the quality of the retrieved papers after duplicates removal, see table 3 below.
The I/E criteria is to ensure that relevant paper were not discarded and to reduce research
bias.
Table 3: Study inclusion and exclusion criteria
Inclusion criteria Exclusion criteria
Publication language should be in English Any language except English
Publication should be from a
journal/conference/workshop and peer
reviewed
Publication that is not part of
any Journal /conference
/workshop. Any gray literature
(viz. thesis, magazines, posters,
notes, training materials, etc.)
Publication should be related to software
engineering domain.
Any domain but software
engineering
The main focus should be creativity
20
3.3.3 Study selection procedure
The approach to primary study selection was carried out in phases. The retrieved data
from each databases were exported to refworks, where exact duplicates search on each
and combine folder was carried out, see table 4 below;
Table 4. Retrieved Publications duplicate removal in REFWORKS
Database Before
Duplicate
After
Duplicate
Duplicate
Scopus 2695 2465 230
IEEE Xplore 3268 3152 116
ACM 1385 1302 83
Science Direct 1524 1489 35
Total 8872 8408 464
The next figure 3 below gives an overview of primary study selection processes. In figure
3 below, 8872 papers was retrieved and duplicates publications were removed from
individual and combined database folders. Also, result from refworks was exported to an
excel sheet where the data were arranged in alphabetical order and duplicates removed.
Applying the exclusion and inclusion criteria reduced the papers to 108 relevant
publications, that were further filtered and 19 was discarded after full-text reading. The
remaining 89 papers were selected as primary studies after couples of iterative review,
followed by quality assessment phase and finally exported to Nvivo 10 for data extraction.
Nvivo 10 is a standard software application for qualitative data analysis.
21
Figure 3: Primary study selection Process
3.3.4 Pilot Testing
The validation of the search strategy is important, which is established by conducting a
pilot run according to Kitchenham (2004). The pilot run is to reduce the risk of flawed
processes and procedure in the review before conducting the actual search (Kitchenham,
2004). This will help to reduce biases in the study selection and also modify the review
protocol, such that relevant studies corresponding to the research focus could be identify.
The review procedure was piloted to validate the selection criteria and it was carried out
by two master’s thesis student independently. They are working on similar research topic
but adopting different method with different context. The protocol for the inclusion was
represented in figure 4 below. Thirty papers were randomly selected for piloting the
inclusion and exclusion criteria in aiding the acceptance or rejection decision on each
publication. This tasks was carried out independently and the results was correlated and
Fleiss Kappa (Carletta, 1996) value was calculated to ascertain the degree of
understanding and agreement among the student regarding the task. The second phase of
the pilot review was conducted because there was an indication of procedural error and
bias.
22
Table 5: Inclusion/Exclusion piloting
n No of selected papers Kappa Value(K) Interpretation
1 30 0.7 fair
2 20 1.00 accepted
The selected primary studies in this phase were considered for further quality assessment
viz applying quality checklist criteria proposed by (Kitchenham & Charters, 2007), before
the data extraction phase.
Figure 4: Publication Inclusion/Exclusion protocol
3.3.5 Quality Assessment
It is essential to assess and analyze the quality of selected primary studies to avoid biases
and ensure their relevances and strength for the current study (Staples and Niazi, 2008).
The selected primary studies were peer reviewed papers and indexed in a recognized
engineering database. Reference to Kitchenham (2007), there exist no concise definition
for quality, hence some quality assessment criteria checklist proposed by Kitchenham
23
(2007) was applied in reducing study bias. The selected primary studies were categorized
into three paper types see table 7 below; Empirical, Theoretical and Opinion papers. The
categorization was done independently in two iterations and results was presented below
(table 6) with calculate Kappa value of (1.00).
Table 6: Publication categorization piloting
n No of selected papers Kappa Value(K) Interpretation
1 12 0.57 fair
2 20 1.00 accepted
During quality assessment exercise, the proposed questions in table 8 below were
answered for each selected publication. The checklist questions was addressed based on
paper categorization since not all questions is applicable to all paper type except empirical
paper. Also, at this phase pilot test was conducted with 9 randomly selected papers
consisting of (3 empirical, 3 theoretical and 3 opinion) and the quality assessment
checklist questions was applied. This was done individually, to ascertain common
understanding and minimize biases in the study, hence agreement level was calculated
with Kappa value (0.769), which was an indication for strong significant. The essence of
the quality assessment was not to drop or rank any paper. The selected studies were peer
reviewed before publication in a recognized journals and conferences, hence papers are
not subjected to ranking or graded over one another by the author. This phase help in
ascertaining the quality and strength of findings generated from the reviews in generating
valid study conclusion. The checklist questions could be appropriate for any well-
structured study.
Table 7: Classification of primary studies into study type
Study type Paper_ID
Empirical [S2], [S4-S17], [S22-S29], [S31-S35], [S38-S41], [S43-S47], [S51-
S53],[S55-S57], [S59-S60], [S62], [S64-S65], [S67], [S69-S70], [S72],
[S85-S89]
Theoretical [S3], [S18], [S20-S21], [S30], [S36-S37], [S42], [S48-S50], [S54],
[S58],[S61], [S66], [S68], [S71], [S73], [S84]
Opinion [S1], [S63]
24
The table 7 above shows the distribution of primary studies into empirical, theoretical and
opinion papers after the piloting phase.
Table 8: Quality assessment checklist
ID Quality Assessment Question Paper Type
1 Does the paper provide rationale for the study? E,T,O
2 Does the paper states the study aims (goals, purposes, problems,
motivations, objectives)? E,T,O
3 Does the paper describe the study context in which study was carried
out (SE, academia, industry)?
E,T
4 Does the paper states the metrics (methods, design, measures) used
in the study?
E,T
5 Are the variables/metrics/methods/design considered by the study
measured?
E,T
6 Does the study provide description of the data/samples? E
7 Does the study provide description of data analysis? E
8 Does the study present its findings? E,T
9 Does the study relates its findings (result/data) to study aims? E,T
10 Does the study discuss any problems (limitations, threats) that might
affect the validity (reliability) of results/findings?
E,T
11 Does the paper mention the relationship between participants and
researchers?
E,O
12 Does the study discuss/mention its relevance to academy/industry? E,T,O
3.3.6 Data Extraction
In this phase, data was extracted from each of the 89 primary studies with the aid of a
standardize application called Nvivo 10. Nvivo10 is an application concisely designated
for systematic literature review, it was an effective tool for organizing and structuring
data for effective decision making. The application contains classification sheet that
stores publication information of all the imported primary studies (Reference ID, Title,
Author, Year, Source, Subject category, Reference type). The extraction process was
piloted before actual the data extraction was carried out. A total number of 89 primary
studies were imported to Nvivo10 and data were extracted based on the created nodes.
Nodes were created to answer the study research question and the application allows
25
researcher to code directly into existing nodes or creates new nodes as paper is being read.
A junk of texts, paragraph could be coded into corresponding single node or different
nodes depending on how relevant the texts are to the nodes. The application allows one
to generate summary report regarding a particular node.
3.3.7 Data Synthesis
The objective of this phase in the review is to synthesize data in reporting information
related to primary studies and also in answering the study research questions. With the
aid of Nvivo10, data was extracted from the primary studies and coded directly under
corresponding nodes (existing nodes, new nodes). The nodes and sub-nodes are further
aggregated into themes, in which reports are generated with the aid of Nvivo10 in
answering the study research questions. The aim is to structure the extracted data such
that it could be adopted in reporting the state-of-the-art research of the study. Information
related to specific research question are extracted on separate nodes, in this manner, it
was easier for the researcher to collate primary studies pointing at certain aspect of the
research question.
Figure 5: Data synthesis illustration.
In figure 5 above, relevant texts, labels, paragraphs from the primary studies were coded
into related nodes which are further combined in addressing the study research questions.
26
4. Results
A total of 89[S1-S89] primary studies were identified addressing different area of the
research questions, see Appendix A. This section presents an overview of the identified
primary studies with the aim of addressing the study research questions.
4.1 To what extents has creativity been studied in SE (RQ1)?
With respect to RQ1, the below sub-questions were considered;
To identify the distribution of study publication per year?
To explore how creativity has been defined in the study?
To establish the study context?
To find out the stages in software development as defined in the SWEBOK, where
creativity is much emphasized/skipped and the possible reason?
4.1.1 Publication Distribution per year
The distribution of primary studies for the current review was between 1992 and 2015,
see figure 6 below, with 73 (82%) studies published in peer reviewed Journal articles and
16 (18%) published in conference proceedings as illustrated in figure 7 below.
Figure 6: Publication distribution per year
In general, the trend of creativity research in software engineering via publication
distribution per year represents an epileptic behavior as illustrated in figure 6 above,
indicating a non-steady developmental trend. Hence, it might be an advantage to improve
creativity research in software engineering domain to allow for better understanding of
creative concepts and its adoption. The illustration in figure 7 below depicts the frequency
1992 1998 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015
no 1 1 2 4 4 2 2 4 11 6 5 8 6 14 12 7
Freq
uen
cy
27
distribution of reference source involving 73(82%) and 16(18%) of the primary studies
published in journals and conference proceedings respectively.
Figure 7: Frequency distribution of Reference source
The primary studies reflected substantial amount of the research based on
experimentation or observation; Empirical research 69(77.5%), theoretical 18 (20.2%)
research and opinion papers 2 (2.3%), see figure 8 below.
Figure 8: Relational distribution of Research Type
4.1.2 Creativity Definition
The knowledge of definitions ascribed to creativity in the selected primary studies could
be of importance in the present research. The table 9 below represent the summary of
publications (19) where creativity has been defined.
Journal Conference proceedings
Frequency 73 16
0
10
20
30
40
50
60
70
80
Freq
uen
cy
studies source
69
18
2
Empirical Theoretical Opinion
28
Table 9: Publications defining creativity
Creativity Paper_ID Frequency
Creativity
meaning/Definition
[S46][S47][S27][S14][710][S15]
[S51][S35][S88][S81][S78]
[S73][S72][S24][S70][S66][S62]
[S61][S22][S50]
19
4.1.3 Analysis and Discussion
The term creativity is the ability to generate creative idea or solution [S50]. Generating a
creative idea could be on individual or group level effort, hence such output is subjective
and is described by three key parameters: new, valuable and unexpected/surprisingly
[S50]. An idea could be considered creative if its address the proposed design problem
and its novel to the person who brings up the idea [S46]. In this case one can deduced that
creativity is subjective. Also, if an idea in software design poses surprise by deviating
from the proposed outcome patterns, such an idea could be considered to be creative
([S47]). This infers that creativity is an unplanned concept, it could occur surprisingly or
unexpectedly. According to ([S14], [S51], [S22], [S73], [S24], [S70], [S62], and [S88]),
Creativity is defined as “the ability to produce work that is both novel (i.e original,
unexpected) and appropriate (i.e useful, adaptive concerning task constraint)”.
Generating of novel and valuable solutions to a problem or defect is ascribed as creativity
([S27], [S22], [S62], [S70], [S24]). In similar manner, creativity does not necessarily
means an idea that is totally new; it could result from the combination of existing ideas
or novel thoughts in forming new ideas ([S61]). Also, adding value to an existing idea
through the creation original idea is termed creativity ([S81]). Value creation through the
transformation of existing products or ideas could be regarded as creativity relative to the
source of the product or idea ([S35]).In project involving diverse contribution like an
open source project, creativity could be termed as the number of feature developed among
the developers’ team ([S15]). Creativity is a concept that could be exhibited as a group or
an individual quality producing idea that is both novel and valuable ([S66], [S70], [S24],
[S73], and [S22]). Creativity is also considered a process of idea generation ([S72]). The
definition of Creativity affirms that generating creative idea is the product of
unconventional thinking that entails constant modification of divergent thoughts,
resulting in novelty and value creation ([S78]).
29
4.1.4 Study Context
The study subject categories were categorized as engineers and academic professionals
including students (graduate/undergraduate) see table10 below. Figure 9 below represent
the frequency distribution of study subject category. The distribution represented in figure
9 below depicts 57(82.6%) and 12(17.4%) studies in academic and industrial settings
respectively.
Table 10: Distribution of study context
Subject Category Domain Frequency percent
Engineers Industry 12 17.4
Professional/students Academics 57 82.6
Figure 9: Frequency distribution for study subject category
The observation in figure 9 above involving 12 (17.4%) publications in industries was an
indication that creative practices could be encourage in software industries ([S35],[S36]),
in developing novel and cost effective solution. The result was in agreement with ([S66]),
focusing on the need to understand creativity in fostering innovation, in order to remain
competitive in the industry.
0
10
20
30
40
50
60
Industry Academics
Engineers Professional/students
Freq
uen
cy
Domain/subject category
Frequency percent
30
4.1.5 Creativity distribution in SE phases
The table 11a below depicts the frequency distribution of studies where creativity has
been emphasized and table 11b indicates the frequency distribution of studies which give
reasons for adopting creativity at different phases in software engineering.
Table 11a: Creativity distribution in software engineering phases
SE phases Paper_ID Frequen
cy
Requirements [S69], [S78], [S47], [S37], [S14], [S58], [S51], [S40],
[S43], [S10], [S64], [S67], [S70], [S71], [519] [S82],
[S85], [S88], [S44], [S9],[S2], [S35], [S36]
24
Design [S46], [S27], [S56], [S28], [S18], [S57], [S49], [S38],
[S3], [S31], [S59], [S32], [S43], [S60], [S23], [S65],
[S75], [S76], [S77], [S44], [S78], [S69]
23
Management [S7], [S15], [S1], [S41], [S4], [S61], [S24], [S83], [S86],
[S54], [S36], [S25]
12
Testing [S44], [S78], [S53] 3
Total 62
Table 11b: Reason for adopting creativity at early phases of software development.
SW Engineering
phases Paper_ID Reason Frequency
Requirements [S2] Decisions made at the early
stages of requirement does
impact the remainder phase and
allows ease of later software
modification
1
Requirement and Design [S40], [S35],
[S36]
This phases are the foundation
for software construction 3
Total 4
Four stages of software development was identified as creative dominant see figure 10
below, consisting requirement phases 24(38.7%), design 23 (37.1%), management 12
(19.4%) and testing phase 3 (4.8%).
31
Figure 10: Frequency distribution of creativity in Software development phases.
4.1.6 Analysis and Discussion
From the figure 10 above, one could deduce that the dominant phase in software
development where creativity has been much emphasized were the requirement ([S69],
[S78], [S47], [S37]) and design phases ([S18], [S57], [S49]). The requirement and design
phase 24(38.7%) and 23 (37.1%) respectively indicates that creative practices at this
phases of software development is important.
There is need for adequate planning before and during software development to avoid
project delay and failure. Decisions made at the earlier phases of software development
(requirement and design phase) does impact the generality of the whole phase and also
allow easy modification at later stages ([S2]). The requirement and design phases of
software design are the foundation for software construction ([S35], [S36]). At this
phases, activities such as brainstorming ([S3], [S56]), collaboration and knowledge
sharing ([S31], [S13], [S65]), and sketching ([S76], [S40], [S77]) which are creativity
techniques are exhibited.
The management phase of software development reflects an appreciable amount of
creative practices during software development ([S41], [S4], [S61], [S24], [S83]) with 12
(19.1%) publications, see (figure10 above) emphasizing creativity at this stage. This is an
indication that management decisions in software organization thus influence
organisation creative practices. Certain extrinsic and intrinsic factors such salary, status
of working environment ([S42], [S52], [S62], [S86], [S46]) and emotional state of
developer ([S74], [S7], [S58], [S39], [S51], [S35]) respectively, does influenced
Requirement Design Management Testing
Frequency 24 23 12 3
0
5
10
15
20
25
30
32
creativity during software development. The management of software development
organisation must provide an enabling environment for software developer to allow them
showcase their potentials and express their creativity ([S1]). According to ([S83]),
software organisation culture should embrace opinion from developers, this in turn
enhance their motivation in working creatively. Also, a form of rewarding system could
influence an employee in working creatively but in similar manner excessive hourly
working could hinder creative practice of employee ([S83]). Management practice in
software organisation should endeavor to document history of costly project failure,
which in future my aid creative practice ([S25]). Proper planning and management of
software development process could serves as guide in building creative solutions ([S4],
[S24]).
The testing phase of software development (figure10) above depicts minimal creative
practices during testing. Four papers from the primary studies discussed creativity at this
phase 4 (6.4%). The widely adoption and flexibility in exploratory testing ET approach;
supports creativity during testing phase in software development ([S53] [S78], [S44]). In
this case no specific test case is made and more defects are detected ([S78], [S53], [44].)
In the current research there was no reported study pointing precisely at any stages in
software development as defined by SWEBOK where creativity has been skipped. In an
empirical study conducted to established which software development task was more
creative, revealed that requirement identification and analysis , modeling, data flow
analysis shows higher rate of creativity adoption unlike test case development which
indicates an intermediate usage of creativity([S78]). Making inference from (figure 10)
above, one could deduced that creativity has been adopted more in the earlier phases of
software development because at this phase, planning and decision are made
([S35][S36][S2]). The investigation regarding the importance of creativity across
software engineering practices revealed that requirement and design phase ([S35], [S36])
has received much attention until later stages. The finding was in agreement with work
of (Akin, 1990; Ball, Onarheim & Christensen, 2010; Díaz et al., 2014) on the importance
of creativity in requirements engineering and software design. A creative plan and design
at the initial stage will serves as a foundation ([S40]) for software development process.
33
4.2 What are the techniques used in introducing and enhancing creativity in SE (RQ 2)?
The motivation for the above research question was to find out ways in which creativity
is been introduced and enhanced during software development. Also, to find out the tools
used in this perspective.
4.2.1 Results
The (table 12) below represent the summary of aggregated studies reporting creativity
method/techniques adopted in SE and the frequency distribution plot is illustrated in
(figure11) below. Brainstorming, Collaborative thinking and knowledge sharing were
considered to be the most widely adopted techniques used to induce creativity in Software
engineering.
Table 12: Distribution of creativity techniques/methods
Creativity Method/Techniques Ref_ID Frequency
Brainstorming [S3] [S20] [S13] [S71] [S80] [S17] [S56] [S28]
[S14] [S49] [S58] [S40] [S31] [S22] [S10]
[S65] [S66] [S24] [S23] [S72] [S74] [S7] [S82]
[S84] [S88] [S89] [S35] [S36] [S9] [S25]
30
Collaboration/knowledge sharing [S7] [S61] [S23] [S62] [S47] [S27] [S3] [S30]
[S31] [S13], [S65] [S66] [S24] [S79] [S89]
[S54] [S35] [S36] [S33] [S9] [S55] [S70] [S3]
[S75]
24
Agile Method [S69] [48] [S4] [S13] [S73] [S83] [S84] [S54]
[S35][S36][S7] [S30] [S4] [S83] [S84] [S28]
[S49] [S10]
16
Sketching [S76] [S18] [S40] [S30] [S59] [S32] [S60]
[S65] [S76] [S77] [S56] [S49] [S38] [S58]
[S23], [S75]
16
OSS [S42] [S22] [S75] [S87] [S11] [S27] [S31]
[S84] [S86] [S36] [S11]
11
Communication [S2] [S30] [S4] [S39] [S31] [S23] [S54] [S35]
[S65] [S88] [S36]
11
Workshop [S62] [S71] [S58] [S30] [S80] [S88] [S89]
[S70] [S10] [S72]
10
Creative thinking [S27][S31][S10] [S52] [S14] [S80] [S88] [199] 8
Flexibility [S63] [S1] [S29] [S30] [S61] [S65] [S23] 7
Scenarios [S71] [S59] [S47] [S62] [S23] [S35] [S36] 7
34
Play [519] [S30] [S81] [S79] [S80] [S84] [S49] 7
Motivation [S52] [S61] [S63] [S79] [S81] [S83] [S54] 7
Creative Problem Solving CPS [S10] [S70] [[S6] [S71] [S39] [S33] 6
Unfamiliar connection [S47] [S61] [S33] [S58] [S74] [S88] 6
Design [S49] [S60] [S18] [1806] [S76] [S33] 6
Games [3] [S30] [S80] [S84] [S89] 5
Exploratory Techniques [S12] [S16] [S53] [S58] [S88] 5
Analogy [S33] [S14] [S10] [S23] [S72] 5
Domain Knowledge [S71] [S67] [S66] [S74][S75] 5
Constraint/Conflicts removal [S39][S88][S23][S72] 4
MindMaps [S34] [S85] [S43] 3
Crowdsourcing [S56] [S86] [S44] 3
Divide and trade [S62] [S33] 2
Transformational approach [S58] [S23] [S88] 3
Prototyping [S71][S49] 2
Model driven Techniques [S57] [S85] 2
Visualization [S35] [S36] 2
Documentation [21] [S64] 2
Creative computing [S50] 1
Pattern design [S37] 1
Decision making [S23] 1
Informal Space [S79] 1
Risk taking [S36] 1
Persistence [S49] 1
Hall of Fame [S58] 1
Groupware [S66] 1
Organisation ethics [S8] 1
35
Figure 11: Frequency Distribution of Creativity Techniques
4.2.2 Analysis and Discussion
It was represented in figure 11 above that brainstorming ([S3], [S20], [S13], [S71], [S80],
[S17], [S56]) was the most popular techniques used in inducing creativity in software
engineering. It is an effective method used among team in generating creative ideas and
clearer perspective of project requirement (S65] [S66] [S24]). It is a concept that is much
adopted among team rather than individual brainstorming which seems to be less effective
([S20]). Collaboration ([S27][S3][S30][S35][S65]) among developers’ does result in
brainstorming and knowledge sharing ([S30][S31][S13]) which could results in
generating creative ideas. The current research identified various studies that supports the
0 5 10 15 20 25 30 35
BrainstormingCollaborationAgile Method
SketchingOSS
CommunicationWorkshop
Creative thinkingFlexibilityscenarios
PlayMotivation
CPSunfamiliar connection
DesignGames
Exploratory TechniquesAnalogy
Domain knowledgeConstraint removal
Mind MapsCrowsourcing
divide and tradetransformational approach
PrototypingModel driven techniques
Visualization and simulationVE
Creative computingPattern Design
DocumentationDecision making
informal spaceRisk TakingPersistence
Hall of FameTrainning
GroupwareOrganisation practices
36
use of creativity workshops ([S62][S71][S80][S88][S89][S70][S10]) as a technique used
in software organizations to foster creativity among employee. The nature of open source
software development approaches, allowing open contribution from diverse source does
promotes creative output ([S42], [S22], [S75], [S87], [S11]). The model driven
architectural concept, which is regarded as the development of software in series of model
transformations, whereby requirements are translated into sketches and, sketches into
codes, has been adapted to software development to resolve abstraction and complexities
in software design and also to foster creativity in software development ( [S57][85]).
Agile software development approach; XP does allows interaction and exploration in
bringing out their creative potentials among developers ([S69], [S48], [S4],[S13], [S73]).
In early stages of software design, adoption of sketching model in manifestation of idea
is creative process that fosters creativity in software architectural design ([S76], [S18],
[S40]). Sketching, a source of creativity (VanDijk and WillemVos, 2011), could
supplement for poorly written conventional source code comments (Baltes and Diehl,
2014) and in software design, sketches compensate for short-term memory and augment
information processing (Suwa, Gero, Purcell, 2000; Gross and Do, 1996). Baltes et.al
2014, concluded that in software development, sketches aids better understanding of
source codes and enhance development work flow. In software architecture, the use of
conventional software diagramming tools results in rigid design that is difficult to modify
and does not adapt to other flexible sketching approaches, thus restrain development
creativity (Grundy and Hosking, 2007). The adoption of flexible approach to software
construction encourages developer’s self-organization and team coordination with
positive potential of fostering creativity among team ([S63], [S1], [S29], [S30], [S61],
[S65], and [S23]). The diagrammatic representation of an idea accords developer’s an
opportunity to convey pictorial representation of their thoughts and conception([S76],
[S18], [S40], [S30], [S59], [S32], [S60], [S65], [S77], [S56], [S49], [S38], [S58], [S23],
[S75]) hence, promoting communication([S2], [S30], [S4], [S39], [S31], [S23], [S54],
[S35], [S65], [S88], [S36]) among developers, which could results in creative problem
solving approach ([S10], [S70], [S6], [S71], [S39], [S33].)
Scenarios ([S71], [S59], [S47], [S62], [S23], [S35], [S36]), visual aid ([S35], [S36]) and
storyboarding ([S49], [S10], [S35], [S36]) are creativity techniques that does articulates
developers creative skills (Katterfeldt and Schelhowe, 2008; Riedl, Rowe and Elson,
2008).Software development models that allow randomly combination of ideas ([S47],
[S61], [S33], [S58], [S74], [S88]) and the uses of analogical reasoning ([S33], [S14],
37
[S10], [S23], [S72]) could generate new and useful ideas ([S73]). Games have also been
adopted in software development as an inspirational source for developers in creating
creative software solution ([3], [S30], [S80], [S84], [S89]). Selected creativity techniques
like; transformational([S58], [S23], [S88]), combinatory ([S47], [S61], [S33], [S58],
[S74], [S88]) and exploratory ([S12], [S16], [S53], [S58], [S88]) have been adapted to
system goals modeling in the creation of a well-structured creative requirement, which
could serves as requirements for the development of large complex critical system in
software engineering (Jennifer Horkoff, Neil Maiden, James Lockerbie, 2015). Team
work improve performance and foster higher creativity in software development project
rather than as a single individual ([S66], [S24], [S79], [S83], [S47], [S3], [S15], [S58],
[S39], [S51], [S2], [S30], [S31], [S86], [S35], [S36], [S7], [S54], [S44], [S10]). The
adoption of work and playful techniques during collaborative requirement elicitation does
foster creativity among software developer team ([S30], [S81], [S79], [S80], [S84],
[S49]). Motivation in terms of incentive and good working condition for software
developers has also been recognized as an assets in software development in enhancing
developers creativity potential ([S52], [S61], [S63], [S79], [S81], [S83], [S54]). The
encouragement of good organizational practices could stimulate good working practices
and in turn motivates developer to work creatively ([S8]). Among other creativity
techniques identified in the current study in introducing creativity during software
development are persistence([S49]), risk taking([S36]), delay in decision making ([S23]),
utilization of well documented best practices ([S21],[S64]), adoption of creative
computing ([S50]), prototyping ([S71], [S49]), groupware ([S66]), adopting mindset of
famous personas and asking them how they will behave in using certain approach([S58]),
crowdsourcing ([S56], [S86], [S44]), adoption of divide and trade concept among group
during requirement elicitation phase of software development([S62],[S33].)
4.3 What are tools used in introducing/enhancing creativity in SE?
The summary in table 13 below represents the reported tools in the primary studies used
in introducing/enhancing creativity during software development. It was reflected that
most pronounced tools, were tools that facilitates interaction and collaboration among
engineers ([S65],[S84], [S71], [S34], [S84], [S57], [S59]) during software development
since no single person possess the absolute knowledge in completing whole task. This
was in agreement with the notion that creative software development activities embraces
collaboration among team to aid better communication and promotes knowledge sharing
38
([13][55][75].)
Table 13: Distribution of creativity supporting tools
Creativity Tools Ref_ID Frequency
Wiki, MikiWiki [S65] [S84] [S31] [S75] 4
MindMap, Wikimap [S34] [S43] [S84] [S85] 4
AgroUML, UML [S18] [S57] [S65] 3
Calico [S56] [S18] [S77] 3
IdeaVIS, Pen and paper [S56] [S57] [S75] 3
Whiteboard [S57] [S38] [S77] 3
Game (jigsaw puzzle, LEGO bricks, Toy) [S80] [519] [S79] 3
FLEXISKETCH, CSCW Sketching tool [S59] [S76] [S77] 3
Scribbler [S38] [S40] 2
Groove [S3] [S6] 2
CSCT Computer supported cooperative Training [S3] [S77] 2
Brainstorming [S18] 1
Cards(DIXIT and PLEX) [S46] 1
Commune(prototyping tool) [S56] 1
Storyboarding [S14] 1
CEL, an iPad application [S57] 1
CoDICE (COdesigning DIgital Cultural
Encounters) [S49] 1
Groupware, Rose, DCSS, Lotus Notes, SOLVEIT,
CoNex, We-Met, Web-CCAT [S3] 1
Syzygy Surfer [S50] 1
SILK, DENIM, InkKit [S40] 1
i-require, Requirement presenter [S71] 1
TopCoder (TC) [S86] 1
Crowd Lang [S55] 1
The illustration in figure 12 below depicts frequency distribution plot of tools identified
in the study for fostering creativity during software development.
39
Figure 12: Frequency distribution of creativity supporting tools
4.3.1 Analysis and Discussion
The most prevailing tool identified in the current studies is Wiki ([S65], [S84], [S31],
[S75], [S84]) which possess an open structure and foster social creativity among
developers. The use of mind mapping tools such Wikimindmap is used in creative
knowledge search ([S34], [S43], [S84], [S85]). Also brainstorming was identified as a
tool used in enhancing creativity during software development ([S18]) while in most
studies brainstorming was identified as a technique in inducing creativity ([3][14][17].)
Different sketching tools have also been adopted in software engineering to foster
creativity. Adoption of tool such as Groove ([3]) allows collaborative activity like
brainstorming among teams. Sketching tools such as Flexisketch Team([S59], [S76],
[S77]) supports collocated and multi-display collaborative sketching and modeling, SILK
([S40]) supports creative process, Pen and Paper ([S75]), IdeaVis, which is a digital pen
for improving traditional paper sketching ([S56], [S57]), Scribbler ([S38], [S40])domain
specific modeling tool and InkKit ([S40]) were reported in the current study as creativity
supporting tools. Computer supported cooperative training CSCT which is a synchronous
collaborative system has been adopted by diverse geographically located developer in
fostering creativity in large programming tasks ([S38], [S40]). Cards; PLEX and DIXIT
has been used during software design in inspiring creative process and generating
intriguing ideas to foster creativity ([S46]). A game (Jigsaw puzzle) is also reported in
0
1
2
3
4
5
6
Wik
i
Min
d M
ap
UM
L
Cal
ico
Pen
an
d P
ape
r
Wh
ite
bo
ard
Flex
iske
tch
tea
m
Scri
bb
ler
Gro
ove
CSC
T
Bra
inst
orm
ing
Insp
irat
ion
car
ds
Co
mm
un
e
Sto
ryb
oar
d
CEL
Co
DIC
E
Gro
up
war
e
RO
SE
DC
SS
Lotu
s N
ote
s
SOLV
EIT
Co
NeX
We-
Met
Web
-CC
AT
Syzy
gy S
urf
er
SILK
DEN
IM
Ink
KIT
i-re
qu
ire
LEG
O b
rick
s
Toy
gam
e
Top
Co
de
r
Co
wd
Lan
g
40
the current studies to induce fun and promotes creativity at the inception of elicitation
meetings ([S80]). TopCoder was identified as the largest software development platform
that embraces crowdsourcing ([S86]), this in turn facilitates contribution from diverse
knowledge in developing creative solution. Adoption of traditional tools has been
reported to limits developer creativity ([S6]), such tools impose a non-flexible approach
to software development processes. The transition from one software configuration
toolkit to another, denied programmers’ of their most esteem tools, forces him to adopt
some divergent working principles which might affects his creativity (Mahler and
Lampen, 1988). In contrast, Melis and Buechley, 2001 argues that the adoption of flexible
development tools support creativity and proffers more dynamic functions in software
design. Adoption of computational tools supports creativity ([S77], [S3]) by allowing
rapid transformation of rough idea into interactive simulations to aid communication
(Gross and Do, 1996). The identified creativity tools such as Wiki, mind map, sketching
tools and storyboarding does possess the tendency to influence developer’s traits and
impacts their motivational potential towards collaboration and knowledge sharing.
According to Amabile, 1996, motivation and creativity skills does influence person’s
creative potential. The adoption of the above reported tools could positively impacts and
enhances developers thinking in the construction of creative architectural software
design. Different reported creativity supporting tools could support or adapt to certain
creativity techniques or methods in fostering creativity, see figure 13 below.
Figure 13: Thematic mapping of the creativity supporting tools to creativity techniques
41
4.4 What are the factors that influence creativity in SE (RQ 3)?
The rationale for the above research question was to find out the reported factors that
could influence creativity in software engineering.
4.4.1 Results
The table 14 below gives a summary of reported factors influencing creativity in software
engineering.
Table 14: Distribution of Creativity Factors
Creativity Factors Paper_ID Frequency
Communication and social
interaction [S69], [S5], [S23], [S66], [S79], [S81], [S82],
[S28], [S15], [S19], [S39], [S51], [S2], [S30],
[S4], [S31], [S86], [S35], [S36], [S54], [S44],
[S5]
22
Collaboration and
coordination [S66], [S24], [S79], [S83], [S47], [S3], [S15],
[S58], [S39], [S51], [S2],[S30], [S31], [S86],
[S35], [S36], [S7], [S54], [S44], [S10]
20
Domain knowledge [S22], [S66], [S67], [S24], [S79], [S27], [S3],
[S50], [S58], [S51], [S31], [S52], [S10], [S62],
[S86], [S29], [S74], [S82], [S9]
19
Intrinsic motivation
Affective state, Beginner's
Mind
[S22], [S68], [S24], [S47], [S51], [S41], [S42],
[S52], [S62], [S86],[S46], [S74], [S51], [S30],
[S54], [S69], [S25]
17
Time [S70], [S72], [S83], [S28], [S6], [S39], [S16],
[S51], [S10], [S62], [S35],[S86], [S36], [S25] 14
Extrinsic motivation [S83], [S28],[S1], 1806, [S51], [S30], [S4],
[S42], [S52], [S86], [S35], [S36], [S37],[S44] 14
Flexibility [S70], [S69], [S50], [S51], [S40], [S30], [S4],
[S21], [S61], [S63], [S29] 11
Stress and pressure [S70], [S79], [S66], [S79], [S80], [S1], [S36] 7
Relationship and trust [S74], [S7], [S58], [S39],[S51], [S35], [S54] 7
Constraints/Conflicts
management [S28], [S80], [S7], [S39], [S35], [S36] 6
Control/guidance [S50], [S58], [S39], [S42], [S61] 5
Cultural diversity [S19], [S39], [S51], [S42], [S13] 5
Task/practice problem [S68], [S24], [S6], [S7] 4
42
Access to resources [S79], [S28], [S10], [S35] 4
Informal space [S79], [S81], [S30] 3
Game [S80], [S30], [S44] 3
Play [S81], [S51], [S30] 3
Informal combination [S22], [S56] 2
Free hand sketching [S23] [S30] 2
Visual aid [S46] [S79] 2
Use of Analogies [S80] 1
Test case [S16] 1
Competition/Challenge [S51] 1
Fixation [S33] 1
The figure 14 below depicts frequency distribution of factors influencing creativity in
software engineering.
Figure 14: Frequency distribution of Creativity Factors
4.4.2 Analysis and Discussion
As represented in the figure14 above, the prevalent factors reported in the selected
primary studies influencing creativity are communication ([S69][S5][S23]), collaboration
([S66][S24][S79]), domain knowledge ([S22][S66][S24][36]) and intrinsic motivation
2220
1917
14 14
11
7 76
5 54 4
3 3 32 2 2
1 1 1 1
0
5
10
15
20
25
43
([S22], [S68], [S24], [S47]) respectively. Communication resulting from geographically
distribution of developers has been identified to directly/indirectly influence creativity of
software development ([S66]). Communication facilitated due to social gathering, relaxed
atmosphere, where engineers get into informal discussion could improve their creative
potential ([S79]). Communication among groups will results in creative synergy ([S15],
[S19], [S12]) unlike in individual with lesser amount of creative practice, hence combined
group are much effective in terms of creativity and quality ([S2]), and in decision making
([S28]). Diverse communication pattern within developers group will enhance their
performance and creativity ([S19]), unlike a centralized communication that might
impede creativity ([S15]). Communication gap and innovate idea does exhibit inverse
relationship, that is as communication gap between developers get central, it hinders the
influx of creative ideas and vice versa but in contrast, centrality was an absolute
determinant for developers group performance([S15]). A good working condition and
environment does stimulate creative communication among member
([S4]).Communication and knowledge sharing within people of diverse background is of
good practices for solving complex issues and source of creative idea but could be
difficult in terms of language variation that might obstruct common understanding, hence
hindering creative potential ([S31]). Communication has aided conflicts resolution,
constraints removal, improve collaboration, minimize pressure, encourage learning
among team and promote organizational structure, which are important factors for
creativity facilitation within group ([S35], [S36], [S54].)
Collaboration among team can improve their individual creative potential via knowledge
sharing ([S66]), though for effective collaboration one need to have an appropriate
knowledge of the subject domain and capacity ([S27]). Collaborating among team with
trust and openness are potential for creative performance ([S54], [S35). Appropriate
domain knowledge and expertise could stimulate creativity ([S66]) since creativity in
software design depends on this knowledge to be applied during software development
([S67]). Existing domain knowledge and expertise is needed for creative collaboration
([S27], [S66], [S67]). Knowledge sharing does directly/indirectly influence creativity and
creativity relies on knowledge ([S66]). Knowledge diversity could be an essential factor
in generating useful and valuables ideas ([S24]) and organisation depends solely on
employee knowledge for task actualization ([S81]). Also the unanticipated combinations
of knowledge from various discipline might results in creative ideas ([S50], [S29]) and
the manipulation of diverse knowledge might results in creative solution ([S10], [S22].)
44
In software development task, giving room for time will promote creative thinking ([S6],
[S10], [S62], [S86]) and in contrast extra working time might result in accumulated stress
and hinders developers creativity ([S83]). Working with specified deadline could hinders
developers’ ability to explore and in return obstruct task creativity ([S28], [S39], [S51],
[S10], [S72]) but exploration without time constraint will encourage creative possibilities
([S16]). Resources such as time is a fundamental issue emphasized in creativity hence its
availability will enhance group performance and creativity ([S51], [S35], [S36], [S10],
[S28]). Inadequate time management could hinder developer’s creative potential ([S25].)
Factors such as salary increase, reward system (money-wise or not), training
, relaxed atmosphere, good working environment, access to resources could influence
developers creative potential and allow worker to shoulder risk ([S83], [S52], [S86],
[S37]). Establishing flexible good relationship among developers and stakeholders by
allowing customers active involvement in development process, embracing developers’
opinion does enhance their motivation to work creatively ([S83],[S51]). Management of
software organisation could embrace a working ethics that allows creative expression and
flexible practices in boosting creativity ([S1]). Trust ([S51]) and participation in
workshops are factors that could enhance creative practices among team ([S1]). Task
oriented and positive minded team results in better creative output unlike relationship
oriented team which might yield lesser performance ([S1], [S39]). Organization policy
should support employees to perform as desirably in other to foster creativity ([S83],
[S36]).
Allowing play at work [70] can leads to employee freedom and creation of new ideas
unlike formal practices that are strict and could result in creative idleness ([S30], [S36]).
Developers’ affective states; moods and emotion ([S74], [S51], [S41], [S30], [S54]) are
essential intrinsic factor that might induce developers’ creativity and performance ([S52],
[S62], [S42], [S41], [S24], [S22]). Positive affective states enhances creative performance
unlike negative affective that impedes creativity ([S52]) but in contrast, positive affective
states does facilitate weak effect on creativity compared to negative affective state
([S52]). In similar sense, Fong (2006), concluded that there exist no specified relationship
between creativity and affective state ([S52]).
Happy software developers were observed to solve problem better and perform better
analytical reasoning ([S68]). Game could be used to enhance developer’s cognitive
potential which is essential factor creativity ([S44], [S30], [S36]). Developers’ flexibility
45
in working plan, procedure, and tools adoption could enhance developer’s creative
potential ([S70], [S69], [S50], [S51], [S40], [S30], [S4], [S21], [S61], [S63], [S29]).
Intrinsic task motivation, self-direction, one’s mental state are critical factor that might
inhibits developer's creativity ([S22], [S68], [S24], [S47], [S51], [S41], [S42], [S52],
[S62], [S86],[S46]). Stress generally inhibits one’s creative potential, distort ability to
think divergently, foster occupational incapacity and above all impedes developers’
creativity potential ([S66], [S79], [S80], [S1], [S36], [S70]).
Software startups companies possess the potential of being creative because they possess
strong motivation towards success and with their fresh partial mind in business, without
any recognized limitation in creative exploration ([S69], [S25]). Another factor such as
manipulating of freehand sketch could be essential in generating creative idea ([S23],
[S30]). Adoption of predesigned test cases does hinder the ability to explore but testing
without existing test cases allow the detection of more defects and this support
exploratory testing which makes use of Tester’s creativity ([S16]). Fixation in
development procedure inhibits creative possibilities ([S33].)
A good conflict resolution practices could promotes a relaxed environment that might
enhance cooperation among developers and in turn could improves employee creativity
([S80], [S7], [S68], [S39]). The occurrence of conflicts among team might results in team
division leading to poor performance and impedes information flow unlike positive
relation resulting in enhanced creativity ([S7]). Control and creativity are essential factors
in software development but aspect of computing need to be flexible for easy adaptation
and management ([S50]).
Software developers could be encourage to explore and express their creative potential
rather than following rigid practices that might hinders their cognitive ability ([S50]).
Developers’ performance is best noticed when they are less stressed ([S68], [S24], [S6],
[S7]). Tasks in form of challenge or competition could foster creativity among
developers’ due the attached reward ([S44]).
4.5 What are the ways of measuring creativity in SE (RQ4)?
The aim of the above research question was to gather reported evidences from selected
primary studies on how creativity is been measured. The interest is towards understanding
the ways in which creativity is quantified/ evaluated in software engineering domain.
46
4.5.1 Results
The table 15 below illustrates the summary of primary studies against corresponding
creativity metric/measurement. The frequency distribution plot is represented in figure 15
below.
Table 15: Distribution of Creativity Metrics/ Measurement
Creativity Metrics/Measurement Ref_ID Frequency
Numbers of added new features [S15][S87]
[S11][S89] 4
Williams creativity Assessment test [S82] [S9] 2
Number of redundant idea/ product idea
rating/information mobility and stability [S22] 1
consensus Value, novelty , open discussion,
diverse stakeholders opinion [199] 1
ratio of number of enhancements integrated in time
period to the number of bugs resolved in the same
time period
[S15] 1
fluctuations in quality as codes evolves [S42] 1
consensual Assessment Techniques [S52] 1
evaluation of creative process outcome [S52] 1
From figure15 below, added numbers of new features and William’s creativity
assessment task among other metric has been identified as the most significant metric for
quantifying creativity in software development.
47
Figure 15: Frequency distribution of Creativity metrics/measurement
4.5.2 Analysis and discussion
One of the metric used in evaluating software project creativity is by observing the
quantity of features or functions added per each release ([S15],[S68], [S87], [S11], [S89]).
Creativity level can be determined by measuring the mean score of the Williams Test
Score ([S82], [S9]). The present studies reported that measure of creativity could be
deduced by number of redundant ideas, ratings of product quality and by information
mobility and stability ([S22]). It could be measured in terms of the number of possible
solutions realized in response to a problem ([S15]) and also in the originality of the
solution in terms of its uniqueness from another ([51]). Although, some creativity
measure is dependent on factors such as originality of generated idea ([S51]).Creativity
process could be controlled, an instance is in free and open source software task in which
one can dynamically track the fluctuations in evolving codes([S42]). Creative
performance could be quantified in terms of process outcome resulting in creative output
([S52]). In evaluating creativity, there is need to review result of creative process or task
(Adams, 2005). In this perspective, the measurement of creativity through the adoption
of the Consensual Assessment Technique ([S51]), was based on the subjective judgement
of domain experts in field of creativity ([S52], [S89]). Creativity evaluation is complex
and difficult to attain in shorter time frame (Weiley and Edmonds, 2011) and still remain
a difficult approach in evaluating one’s creativity (Storme, Myszkowksi, Celik & Lubart,
2014.)
0 0,5 1 1,5 2 2,5 3 3,5 4 4,5
Added features
Williams CA Test
product quality
Stakeholders Opinion
enhnacements per bugs
codes fluntautions
CATs
Creative evaluation
Frequency
Cre
ativ
ity
mea
sure
men
tan
d m
etri
cs
Addedfeatures
WilliamsCA Test
productquality
Stakeholders
Opinion
enhnacements per
bugs
codesfluntautio
nsCATs
Creativeevaluatio
n
Frequency 4 2 1 1 1 1 1 1
48
5. Discussion
This section compared the findings from the primary studies to the existing knowledge of
software engineering regarding the research topic. The objective is to clearly provide a
rigid and exhaustive summary of reported evidences regarding the state of art of the
research on creativity in software engineering.
Having considered the various definitions and meanings assigned to creativity in the
primary studies ([27][62][22]), it's evident that the reported meanings was in agreement
with creativity definition by (Amabile, 1988; 1996; Sternberg and Lubart, 1999). The key
terms in the definition are; novelty and valuable. The term novelty and valuable as
described in literature by Amabile, 1988; Sternberg and Lubart, 1999, refers to the fact
that an idea or product must be original or unexpected and appropriate or useful or
adaptable to problem to be term creative. The process of generating such an idea is termed
creativity ([S72]) which affirmed Amabile, 1988 work describing creativity as a process.
The outcome of such creative process should be of value. Creativity is referred to as a
process (Amabile, 1988; [S72]) while being creative is an attributes associated with
people (Rhode, 1961; [S22];[S66]) and the outcome of such process should be of value
to man(Amabile, 1988 ; [S70],[S22]). The reported studies mirrored the attributes
considered in (Rhodes, 1991) ‘4ps model in describing the concept of creativity. Creative
idea is been created and used by Man, hence one could conclude that creativity starts and
end with man. Another important factor is that creativity has been soughed as being
contextual, which implies that it varies across domain (Kozbelt, Beghetto and Runco,
1990; Adams, 2006; Nguyen and Shanks, 2009; [S35]). The findings revealed that
creativity does not necessarily needs to be something new (idea, process or product) hence
it could be as a result of the combination of existing idea or solution or product,
transformation of existing knowledge or idea in forming new ones and exploring space
for newer idea (Maiden et.al, 2010; Boden, 2004; [58]).
In literature there has been vast numbers of methods, practices and techniques reported
to support creativity (Crawford et al., 2012), this also aligned with the reported finding
([S20][S61][S69][S76][S84]). Brainstorming has been identified as the most significant
technique in fostering creativity ([S3], [S71], [S80]; Aleven et al., 2008; Aragon, Poon,
Hernåndez and Aragon, 2009; Silva, Aureliano and Barbosa, 2006; VanDijk and
WillemVos, 2011; Ocnarescu, Pain, Bouchard, Aoussat and Sciamma, 2011). The need
to encourage creative and collaborative thinking in software development was reported
49
to induce creativity during software development ([S7], [S89], and [S70]; Aleven,
Ashley, Lynch and Pinkwart, 2008). Agile method of software development (XP) does
support creative thinking through effective communication among team which might
result in novel solution (Maiden et al., 2012; [S69][S48][S23][S10]). The findings from
the current study agree with the fact that, adopting open source software development
model could results in creative output ([S22], [S42], [S15]; Paulson, Succi and Eberlein,
2004). A significant number of studies is in support of using workshop in fostering
creativity ([S62], [S71], and [S58]; Shneiderman et. al., 2006). Workshop allows the
opportunity for stakeholders and developers from diverse background to meet and
informally interact, which might result in exchange and generation of new creative idea
([S70], [S10]). Thinking divergently allows developers to decipher issues from actual
solution, hence proffering better understanding (Basadur, Ellspermann and Evans, 1994;
([S52], [S27]). Literature have it that domain expertise is a function of divergent thinking
in that domain (Reiter-Pamon et al., 2009), which might results in one’s creative potential
in such domain (Schraagen, 1993; Silvia, Kauffman and Pretz, 2009; [S67]). An
interesting observation of the study finding was the use of technique referred to as Hall
of fame to induce creativity, in which an expertise of a particular domain is ask ;why and
how questions([S58]).
Software development organisation norms that consider the interest and welfare of its
employee will welcome trust, smooth and cordial relationship among staff. Such a
management style that provide access to good working tools([S65]), efficient reward
system([S28]), workshops and training([S88],[S58]), flexible leadership style([S23],
[S61]), enhance conflicts and constraints resolution approach ([S72], [S23]) is liable to
witness more creative success. These claims have been revealed also in the work of
(Amabile, 1996.)
Factors categorized as extrinsic and intrinsic motivation has contributed to developers
motivation. Findings from the present studies reported quite a number of factors
influencing creativity in software development. Intrinsic factors such as developer’s
personal traits, affective state (mood, knowledge, mental state, cognitive potentials, stress
and pressure) are said to be motivated by the external factors such as reward system,
infrastructure, culture, environment, play, occupation stress and leadership. The majority
of the identified factors in the primary studies was agreement with the work of (Amabile,
1983; 1996; 1997; Gu and Tong, 2004;[S22][S68][S24][S83][S28][52] ). Team of
developers need to establish good relationship to allow efficient work flow. In this
50
manner, geographically located team of software developers relies on trust and
relationship as essential factor in facilitating creativity and the overall project success
(Humayun, Gang and Masood, 2013; [S75][S54]).
Besides the identified creativity supporting tools; brainstorming ([S18]), wiki ([S84]) in
the current findings which are consistent with tools used in theory (Nguyen and Cybulski,
2008; Amabile, 1996), there exist some improvement to sketching tools, use in the
manifestation of thoughts. Tools such as pen and paper that is being used during
requirement elicitation or at the design phase of software design has been augmented with
tools such as CEL([S57]), CoDICE([S49]) and CaLICO([S18]), these enhanced tools has
the capability of supporting lightweight informal sketching, which provide avenue for
design exploration. CEL is an IPad application that support visual language for creating
and manipulating object oriented models while CoDice support high flexibility and
creativity expression
In the context of creativity measurement/evaluation, it is essential to review the creative
output (Adams, 2005). Creativity evaluation is complex and difficult to attain in shorter
time frame (Weiley and Edmonds, 2011; [S28], [S16]). The most widely accepted metrics
for creativity measurement was to evaluate the quantity of features or functions added per
each release (LeCompte, 2000; Paulson, Succi and Eberlein, 2004; [S11], [S89]). Judging
creative output is dependent on domain knowledge i.e the level of knowledge acquired in
such focused domain ([S10]). Amabile, 1996; 1983, reflects that domain knowledge is
required in evaluating creative output and in her opinion best approach of measuring
creativity is by judging a creative output. This was in line with findings from the present
study that judging creative output is subjective ([S15], [S9], [51], [S52].)
In essence, the research findings have shown much agreement to the existing knowledge
in terms of creativity definition, adopted tools, techniques and, evaluation and
measurement perspective. It will be an interesting task to find out how software industry
validates the reported evidences in the current study. Moreover, an empirical study could
be conducted to validate how the proposed improved creativity supporting tools will
behave over time in fostering creativity in software engineering domain.
51
6. Conclusion
This section describes the concluding part of the present research work. The validity
threats and limitations of the study were presented and the insights from the systematic
review were discussed. The research implication was also presented in this section.
The research identified 89 primary studies within (1992-2015) describing
methods/techniques/process, factors, tools and measurements adopted in software
engineering for emphasizing creativity. The definitions gathered from the study regarding
creativity conformed to the widely adopted definitions from literatures emphasizing that
an idea, product, or process should be novel and useful to be termed creative. The study
consisted of (77.5%) empirical, theoretical (20.2%) and opinion (2.3%) papers, this
reflects that more experimental and observational studies are been conducted in
enhancing and promoting creative practices in software engineering domain. The findings
reported the adoption of creative practices at the requirement elicitation (38.1%), design
(36.5%) and management (19.1%) phases of software development because this stage is
refers to the foundation for software development, where project planning and decisions
are made among the stakeholders and developers. The study context reflected (82%) and
(18%) studies conducted in the academic and industrial settings respectively. This
indicates that software development organisation could embrace more flexible
approaches/methods during development. The study findings revealed that creativity
measurement is subjective and can’t be benchmark to a single approach. The identified
creativity techniques and factors does have significant impact on software development
creative process. Augmented lightweight tools that support creative thinking and
expression was also reported. Further empirical research could be conducted to deduce
the influence of creativity techniques/method on creative problem solving in software
development. The reported creativity supporting tools could also be subjected to
industrial validation.
The findings from study does not specified or generalized any absolute
method/techniques/practice or approach to creativity, this implies that creativity is
subjective and varies with people and domain. The question to ask is that is there any
absolute new creation? In my opinion, every creation is an evolution and not
revolutionary; this implies that the creation started from modification of something in
existence. The creativity to the person that created it, is the feature that make it unique
from the existing one. Software engineering could embrace the adoption of alternative
52
flexible approaches to prevent limiting human imagination with specified development
model. The management of software organisation could embrace effective practices in
supporting the right creative mind, which are essential asset for creativity sustainability
in software engineering domain. The creative mind are expertise with enhance domain
knowledge and cognitive potential.
The systematic literature review present findings from literature, a colleague has
conducted an industrial questionnaires and survey related to same topic but there is need
to compare findings in future in order to identify the state-of-the practice of creativity in
software engineering.
6.1 Research implication
The reported findings in the current studies do possess some essential implications for the
adoption of creative practices in the development of software in SE domain for both
academic and industry. In the perspective of researchers and practitioners, the following
implications could be of relevance:
● The findings could contribute to an existing knowledge in software engineering
regarding creativity adoption. It revealed that creativity could be encouraged and
emphasized at requirement and design phases where planning and decisions
regarding development are made and also this phases are regarded as the
foundation for software development.
● The findings advocate the adoption of good management practices and flexible
leadership style in software organisation for continuous deployment of novel and
useful solutions.
● The study emphasizes the encouragement of knowledge sharing and collaboration
among developers. This findings has an important implication as it reflects that
researchers and practitioners could share ideas, help each other to see new
viewpoints, channels thought to trigger creative ideas.
● The study reaffirms the definition of creativity by emphasizing the key term;
novelty and usefulness as an important factor in describing creativity. This implies
that an idea, product, or process will be regarded as being creative if it does satisfy
the key terms.
● In terms of creativity supporting tools, the findings encourage the adoption of
tools that will explicitly support creative expression and foster creative thinking.
53
This implication could be of important to practitioners and researchers in making
decision regarding right tools to support creative thinking.
6.2 Validity threats and Limitations of study
The significant threats in the present study are resulting from the selection bias and in
data extraction. The data extraction was done by single researcher and this was a deviation
from kitchenham, (2004) guideline. The bias was reduced to minimal by adherence to the
outlined review protocol and repetitive iterative pilot test carried out multiple times at
each review phase. However, the supervisor was contacted for clarity in unsure situations.
Some relevant studies might have been omitted in the process of excluding studies related
to innovation which is a very close term to creativity, though distinction was made about
the two terms.
The searching phase of the study were subjected to certain limitations, an instance is the
searches carried out in ACM database, it was challenging importing search results to
reference manager due to number of searches. However, this was overcome by repeated
exportation with extreme caution.
The notion that creativity is a subjective concept that varies across knowledge and with
person, make the generalization about the findings on creativity in SE difficult.
The searched results from each probed databases was documented with corresponding
date and time such that it could allow study replication.
54
References
Adam, S., & Trapp, M. (2015). Success Factors for Creativity Workshops in RE.
Adams, K. (2005). The Sources of Innovation and Creativity. National Center on
Education and the Economy (NJ1).
Akin, Ö. (1990). Necessary conditions for design expertise and creativity. Design Studies,
11(2),107-113.
Amabile, T. (1996). Creativity in context. Westview press.
Amabile, T. M. (1983). The social psychology of creativity: A componential
conceptualization. Journal of personality and social psychology, 45(2), 357.
Amabile, T. M. (1993). Motivational synergy: Toward new conceptualizations of intrinsic
and extrinsic motivation in the workplace. Human resource management review,
3(3), 185-201.
Amabile, T. M., & Gryskiewicz, N. D. (1989). The creative environment scales: Work
environment inventory. Creativity research journal, 2(4), 231–253.
Amabile, T. M., Barsade, S. G., Mueller, J. S., & Staw, B. M. (2005). Affect and creativity
at work. Administrative science quarterly, 50(3), 367-403.
Amin, A., Basri, S., Hassan, M. F., & Rehman, M. (2011, September). Software
engineering occupational stress and knowledge sharing in the context of global
software development. In National Postgraduate Conference (NPC), 2011 (pp. 1-4).
IEEE.
Arbaoui, S., Lonchamp, J., & Montangero, C. (1999). The human dimension of the
software process. Software Process: Principles, Methodology and Technology,
LNCS, 1500, 165-196.
Baker, S. W. (2006, July). Formalizing agility, part 2: How an agile organization
embraced the CMMI. In Agile Conference, 2006 (pp. 8-pp). IEEE.
Barney, S., Petersen, K., Svahnberg, M., Aurum, A., & Barney, H. (2012). Software
quality trade-offs: A systematic map. Information and Software Technology, 54(7),
651-662.
Baxter, G., & Sommerville, I. (2011). Socio-technical systems: From design methods to
systems engineering. Interacting with computers, 23(1), 4-17.
Beck, K. (1999). Embracing change with extreme programming. Computer, 32(10), 70–
77.
Beghetto, R. A., & Kaufman, J. C. (2007). Toward a broader conception of creativity: A
case for" mini-c" creativity. Psychology of Aesthetics, Creativity, and the Arts, 1(2),
73.
55
Ben Hadj Salem Mhamdia, A. (2013). Performance measurement practices in software
ecosystem. International Journal of Productivity and Performance Management,
62(5), 514-533.
Bhowmik, T. (2014, August). Stakeholders' social interaction in requirements
engineering of open source software. In Requirements Engineering Conference (RE),
2014 IEEE 22nd International (pp. 467-472). IEEE.
Boden, M. A. (2004). The creative mind: Myths and mechanisms. Psychology Press.
Bourque, P., Dupuis, R., Abran, A., Moore, J. W., Tripp, L., & Wolff, S. (2002).
Fundamental principles of software engineering–a journey. Journal of Systems and
Software, 62(1), 59-70.
Buhr, R. J., Karam, G. M., Hayes, C. J., & Woodside, C. M. (1989). Software CAD: a
revolutionary approach. Software Engineering, IEEE Transactions on, 15(3), 235-
249.
Butler, A. B., Scherer, L. L., & Reiter-Palmon, R. (2003). Effects of solution elicitation
aids and need for cognition on the generation of solutions to ill-structured problems.
Creativity Research Journal, 15(2-3), 235–244.
Cao, J., Riche, Y., Wiedenbeck, S., Burnett, M., & Grigoreanu, V. (2010, April). End-
user mashup programming: through the design lens. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (pp. 1009-1018). ACM.
Carletta, J. (1996). Assessing agreement on classification tasks: the kappa statistic.
Computational linguistics, 22(2), 249–254.
Carmel, E., George, J. F., & Nunamaker, J. F. (1992, January). Supporting joint
application development (JAD) and electronic meeting systems: moving the CASE
concept into new areas of software development. In System Sciences, 1992.
Proceedings of the Twenty-Fifth Hawaii International Conference on (Vol. 3, pp.
331-342). IEEE.
Couger, J. D., Higgins, L. F., & McIntyre, S. C. (1993). (Un)structured creativity in
information systems organizations. MIS Quarterly, 375-397.
Cowling, A. J. (2004, March). The crossover project as an introduction to software
engineering. In Software Engineering Education and Training, 2004. Proceedings.
17th Conference on (pp. 12-17). IEEE.
Crawford, B., & De La Barra, C. L. (2007). Enhancing creativity in agile software teams.
In Agile Processes in Software Engineering and Extreme Programming (pp. 161-
162). Springer Berlin Heidelberg.
Crawford, B., de la Barra, C. L., Soto, R., & Monfroy, E. (2012, June). Agile software
engineering as creative work. In Proceedings of the 5th International Workshop on
Co-operative and Human Aspects of Software Engineering (pp. 20-26). IEEE Press.
Csikszentmihalyi, M. (1999). 16 Implications of a Systems Perspective for the Study of
Creativity. In Handbook of creativity (pp. 313–335). Cambridge University Press.
56
Díaz, P., Aedo, I., Rosson, M. B., & Carroll, J. M. (2010, May). A visual tool for using
design patterns as pattern languages. In Proceedings of the International Conference
on Advanced Visual Interfaces (pp. 67-74). ACM.
Dybä, T., Kitchenham, B. A., & Jorgensen, M. (2005). Evidence-based software
engineering for practitioners. Software, IEEE, 22(1), 58–65.
Eckert, C., Blackwell, A., Stacey, M., Earl, C., & Church, L. (2012). Sketching across
design domains: Roles and formalities. Artificial Intelligence for Engineering
Design, Analysis and Manufacturing, 26(03), 245-266.
Elberzhager, F., Münch, J., & Nha, V. T. N. (2012). A systematic mapping study on the
combination of static and dynamic quality assurance techniques. Information and
Software Technology, 54(1), 1-15.
Fischer, S., Boogaard, M., & Huysman, M. (1994, January). Innovation and creativity
versus control assessing the paradox in IT organizations. In System Sciences, 1994.
Proceedings of the Twenty-Seventh Hawaii International Conference on (Vol. 4, pp.
367-376). IEEE.
Florida, R., Knudsen, B., Stolarick, K., & Youl Lee, S. (2006). Talent and Creativity in
the Software Industry.
Floyd, C., Mehl, W. M., Reisin, F. M., Schmidt, G., & Wolf, G. (1989). Out of
Scandinavia: Alternative approaches to software design and system development.
Human-Computer Interaction, 4(4), 253-350.
Fong, C. T. (2006). The effects of emotional ambivalence on creativity. Academy of
Management Journal, 49(5), 1016–1030.
Frijda, N. H. (1986). The emotions: Studies in emotion and social interaction. Edition de
la.
Fuggetta, A. (2000, May). Software process: a roadmap. In Proceedings of the
Conference on the Future of Software Engineering (pp. 25-34). ACM.
Gandomani, T. J., Zulzalil, H., Ghani, A. A., Sultan, A. B. M., & Sharif, K. Y. (2014).
How human aspects impress agile software development transition and adoption.
International Journal of Software Engineering and its Applications, 8(1), 129–148.
Graziotin, D. (2013). The Dynamics of Creativity in Software Development. arXiv
preprint arXiv:1305.6045.
Gu, M., & Tong, X. (2004). Towards hypotheses on creativity in software development.
In Product Focused Software Process Improvement (pp. 47-61). Springer Berlin
Heidelberg.
Hegde, R., & Walia, G. S. (2014). How to Enhance the Creativity of Software Developers:
A Systematic Literature Review. In SEKE (pp. 229-234).
Henry, J. (Ed.). (2006). Creative management and development. Sage.
Herbsleb, J. D., & Moitra, D. (2001). Global software development. Software, IEEE,
18(2), 16–20.
57
Highsmith, J., & Cockburn, A. (2001). Agile software development: The business of
innovation. Computer, 34(9), 120–127.
Hili, N., Laurillau, Y., Dupuy-Chessa, S., & Calvary, G. (2015, June). Innovative key
features for mastering model complexity: flexilab, a multimodel editor illustrated on
task modeling. In Proceedings of the 7th ACM SIGCHI Symposium on Engineering
Interactive Computing Systems (pp. 234-237). ACM.
Hoffmann, O., Cropley, D., Cropley, A., Nguyen, L., & Swatman, P. (2007). Creativity,
requirements and perspectives. Australasian journal of information systems, 13(1).
Hollis, B., & Maiden, N. (2013). Extending Agile Processes with Creativity Techniques.
Software, IEEE, 30(5), 78–84.
Kilgour, M. (2006). Improving the creative process: Analysis of the effects of divergent
thinking techniques and domain specific knowledge on creativity. International
journal of business and society, 7(2), 79.
Kitchenham, B. (2004). Procedures for performing systematic reviews. Keele, UK, Keele
University, 33(2004), 1-26.
Kitchenham, B. A., Dyba, T., & Jorgensen, M. (2004, May). Evidence-based software
engineering. In Proceedings of the 26th international conference on software
engineering (pp. 273-281). IEEE Computer Society.
Kitchenham, B., & Charters, S. (2007). Guidelines for performing systematic literature
reviews in software engineering. Tech. Rep. EBSE 2007-001, Keele University and
Durham University Joint Report.
Kitchenham, B., Brereton, O. P., Budgen, D., Turner, M., Bailey, J., & Linkman, S.
(2009). Systematic literature reviews in software engineering–a systematic literature
review. Information and software technology, 51(1), 7-15.
Kitchenham, B., Mendes, E., & Travassos, G. H. (2006, April). A systematic review of
cross-vs. within-company cost estimation studies. In Proceedings of the 10th
international conference on Evaluation and Assessment in Software Engineering (pp.
81-90). British Computer Society.
Kohls, C. (2014, July). Dream teams at the right place. In Proceedings of the 19th
European Conference on Pattern Languages of Programs (p. 3). ACM.
Kozbelt, A., Beghetto, R. A., & Runco, M. A. (2010). Theories of creativity. The
Cambridge handbook of creativity, 20-47.
Kusumasari, T. F., Supriana, I., Surendro, K., & Sastramihardja, H. (2011, July).
Collaboration model of software development. In Electrical Engineering and
Informatics (ICEEI), 2011 International Conference on (pp. 1-6). IEEE.
Lanubile, F., Ebert, C., Prikladnicki, R., & Vizcaíno, A. (2010). Collaboration tools for
global software engineering. IEEE software, 27(2), 52.
Lemos, J., Alves, C., Duboc, L., & Rodrigues, G. N. (2012, March). A systematic
mapping study on creativity in requirements engineering. In Proceedings of the 27th
Annual ACM Symposium on Applied Computing (pp. 1083-1088). ACM.
58
Lubart, T. I. (2001). Models of the creative process: Past, present and future. Creativity
Research Journal, 13(3-4), 295-308.
Macasaet, R., Chung, L., Garrido, J. L., Noguera, M., & Rodríguez, M. L. (2011, June).
An agile requirements elicitation approach based on NFRs and business process
models for micro-businesses. In Proceedings of the 12th International Conference
on Product Focused Software Development and Process Improvement (pp. 50-56).
ACM.
Maiden, N. (2001). Where do requirements come from? Software, IEEE, 18(5), 10-12.
Maiden, N., & Hollis, B. (2013, June). Creativity on a shoestring: concept generating in
agile development. In Proceedings of the 5th ACM SIGCHI symposium on
Engineering interactive computing systems (pp. 333-334). ACM.
Maiden, N., & Robertson, S. (2005, May). Developing use cases and scenarios in the
requirements process. In Proceedings of the 27th international conference on
Software engineering (pp. 561-570). ACM.
Maiden, N., Gizikis, A., & Robertson, S. (2004). Provoking creativity: Imagine what your
requirements could be like. Software, IEEE, 21(5), 68-75.
Maiden, N., Jones, S., Karlsen, K., Neill, R., Zachos, K., & Milne, A. (2010, September).
Requirements engineering as creative problem solving: A research agenda for idea
finding. In Requirements Engineering Conference (RE), 2010 18th IEEE
International (pp. 57-66). IEEE.
Marakas, G. M., & Elam, J. J. (1997). Creativity enhancement in problem solving:
through software or process? Management Science, 43(8), 1136–1146.
McFadzean, E. (1998). Enhancing creative thinking within organisations. Management
Decision, 36(5), 309–315.
Mich, L., Anesi, C., & Berry, D. M. (2004). Requirements engineering and creativity: An
innovative approach based on a model of the pragmatics of communication. In Proc.
REFSQ (pp. 3-922602).
Mohanani, R., Ralph, P., & Shreeve, B. (2014, May). Requirements fixation. In
Proceedings of the 36th International Conference on Software Engineering (pp. 895-
906). ACM.
Mumford, M. D. (2000). Managing creative people: Strategies and tactics for innovation.
Human resource management review, 10(3), 313-351.
Munoz, R., Barcelos, T., Noel, R., & Kreisel, S. (2012, November). Development of
Software that Supports the Improvement of the Empathy in Children with Autism
Spectrum Disorder. In Chilean Computer Science Society (SCCC), 2012 31st
International Conference of the (pp. 223-228). IEEE.
Nguyen, L., & Shanks, G. (2009). A framework for understanding creativity in
requirements engineering. Information and software technology, 51(3), 655-662.
Obrenović, Z. (2013). Software Sketchifying: Bringing Innovation into Software
Development. Software, IEEE, 30(3), 80–86.
59
Onarheim, B., & Wiltschnig, S. (2010, August). Opening and constraining: constraints
and their role in creative processes. In Proceedings of the 1st DESIRE Network
Conference on Creativity and Innovation in Design (pp. 83-89). Desire Network.
Piffer, D. (2012). Can creativity be measured? An attempt to clarify the notion of
creativity and general directions for future research. Thinking Skills and Creativity,
7(3), 258-264.
Reiter-Palmon, R., Illies, M. Y., Cross, L. K., Buboltz, C., & Nimps, T. (2009). Creativity
and domain specificity: The effect of task type on multiple indexes of creative
problem-solving. Psychology of Aesthetics, Creativity, and the Arts, 3(2), 73.
Robertson, J. (2002). Eureka! why analysts should invent requirements. IEEE Software,
19(4), 20-22.
Robillard, P. N. (2005). Opportunistic problem solving in software engineering. Software,
IEEE, 22(6), 60-67.
Royce, W. W. (1970, August). Managing the development of large software systems. In
Proceedings of IEEE WESCON (Vol. 26, No. 8, pp. 1-9).
Rus, I., & Lindvall, M. (2002). Knowledge management in software engineering. IEEE
software, 19(3), 26.
Saha, S. K., Selvi, M., Buyukcan, G., & Mohymen, M. (2012, May). A systematic review
on creativity techniques for requirements engineering. In Informatics, Electronics &
Vision (ICIEV), 2012 International Conference on (pp. 34-39). IEEE.
Salas, E., Sims, D. E., & Burke, C. S. (2005). Is there a “Big Five” in teamwork? Small
group research, 36(5), 555-599.
Seth, F. P., Mustonen-Ollila, E., Taipale, O., & Smolander, K. Software quality
construction in 11 companies: an empirical study using the grounded theory.
Software Quality Journal, 1-34.
Shneiderman, B. (2002, October). Creativity support tools: a tutorial overview. In
Proceedings of the 4th conference on Creativity & cognition (pp. 1-2). ACM.
Shoaib, L., Nadeem, A., & Akbar, A. (2009, December). An empirical evaluation of the
influence of human personality on exploratory software testing. In Multitopic
Conference, 2009. INMIC 2009. IEEE 13th International (pp. 1-6). IEEE
Shull, F., Basili, V., Carver, J., Maldonado, J. C., Travassos, G. H., Mendonça, M., &
Fabbri, S. (2002). Replicating software engineering experiments: addressing the tacit
knowledge problem. In Empirical Software Engineering, 2002. Proceedings. 2002
International Symposium n (pp. 7-16). IEEE.
Simonton, D. K. (1999). Origins of genius: Darwinian perspectives on creativity. Oxford
University Press.
Simonton, D. K. (2003). Scientific creativity as constrained stochastic behavior: the
integration of product, person, and process perspectives. Psychological bulletin,
129(4), 475.
60
Singer, L., Seyff, N., & Fricker, S. A. (2011, September). Online social networks as a
catalyst for software and IT innovation. In Proceedings of the 4th international
workshop on Social software engineering (pp. 1-5). ACM.
Staples, M., & Niazi, M. (2008). Systematic review of organizational motivations for
adopting CMM-based SPI. Information and software technology, 50(7), 605–620.
Stein, M. I. (1953). Creativity and culture. The journal of psychology, 36(2), 311-322.
Sternberg, R. J., & Lubart, T. I. (1999). The concept of creativity: Prospects and
paradigms. Handbook of creativity, 1, 3-15.
Sutling, K., Mansor, Z., Widyarto, S., Letchmunan, S., & Arshad, N. H. (2014,
September). Agile project manager behavior: The taxonomy. In Software
Engineering Conference (MySEC), 2014 8th Malaysian (pp. 234-239). IEEE.
Tiwana, A., & Keil, M. (2004). The one-minute risk assessment tool. Communications of
the ACM, 47(11), 73–77.
Tolfo, C., & Wazlawick, R. S. (2008). The influence of organizational culture on the
adoption of extreme programming. Journal of systems and software, 81(11), 1955–
1967.
Van der Veer, G. C., & Vyas, D. (2011, August). Non-formal techniques for requirements
elicitation, modeling, and early assessment for services. In Proceedings of the 29th
Annual European Conference on Cognitive Ergonomics (pp. 285-286). ACM.
Walz, D. B., & Wynekoop, J. (1994, October). Creativity and software design: is formal
training helping or hurting? In Systems, Man, and Cybernetics, 1994. Humans,
Information and Technology., 1994 IEEE International Conference on (Vol. 1, pp.
842-846). IEEE.
Weiss, H. M., & Cropanzano, R. (1996). Affective events theory: A theoretical discussion
of the structure, causes and consequences of affective experiences at work.
Wen, Y., Zhang, H., Liu, L., & Yang, H. (2010, September). One bridge, two gaps-
beyond an engineering approach: Creativity in requirements elicitation. In
Requirements Engineering Visualization (REV), 2010 Fifth International Workshop
on (pp. 40-42). IEEE.
Whitehead, J. (2007, May). Collaboration in software engineering: A roadmap. In 2007
Future of Software Engineering (pp. 214–225). IEEE Computer Society.
Zimmerman, E. (2010). Creativity and art education: A personal journey in four acts. Art
Education, 63(5), 84–92.
61
Appendix A: Lists of Accepted Primary studies
Ref_ID REFERENCES
[S1] Whalley, J. (1993). Creativity and discipline quality management in
software. IEE Review, 39(6), 263–265.
[S2] Ocker, R., Fjermestad, J., Hiltz, S. R., & Turoff, M. (1997, January). An
exploratory comparison of four modes of communication for determining
requirements: results on creativity, quality and satisfaction. In System
Sciences, 1997, Proceedings of the Thirtieth Hawaii International
Conference on (Vol. 2, pp. 568-577). IEEE.
[S3] DeFranco-Tommarello, J., & Deek, F. P. (2002, January). Collaborative
software development: a discussion of problem solving models and
groupware technologies. In System Sciences, 2002. HICSS. Proceedings of
the 35th Annual Hawaii International Conference on (pp. 568-577). IEEE.
[S4] Knublauch, H. (2002, July). Extreme programming of multi-agent systems.
In Proceedings of the first international joint conference on Autonomous
agents and multiagent systems: part 2 (pp. 704-711). ACM.
[S5] Campbell, C. L., & Van De Walle, B. (2003, August). Asynchronous
requirements engineering: enhancing distributed software development. In
Information Technology: Research and Education, 2003. Proceedings.
ITRE2003. International Conference on (pp. 133-136). IEEE.
[S6] DeFranco-Tommarello, J., Hiltz, S. R., Deek, F. P., Perez, C., & Keenan, J.
P. (2003, January). Collaborative software development: experimental
results. In System Sciences, 2003. Proceedings of the 36th Annual Hawaii
International Conference on (pp. 10-pp). IEEE.
[S7] Domino, M. A., Collins, R. W., Hevner, A. R., & Cohen, C. F. (2003, April).
Conflict in collaborative software development. In Proceedings of the 2003
SIGMIS conference on Computer personnel research: Freedom in
Philadelphia--leveraging differences and diversity in the IT workforce (pp.
44-51). ACM.
[S8] Dybå, T. (2003). Factors of software process improvement success in small
and large organizations: an empirical study in the Scandinavian context.
ACM SIGSOFT Software Engineering Notes, 28(5), 148–157.
[S9] Damian, D., Chisan, J., Vaidyanathsamy, L., & Pal, Y. (2003, September).
An industrial case study of the impact of requirements engineering on
62
downstream development. In Empirical Software Engineering, 2003. ISESE
2003. Proceedings. 2003 International Symposium on (pp. 40-49). IEEE.
[S10] Maiden, N., Manning, S., Robertson, S., & Greenwood, J. (2004, August).
Integrating creativity workshops into structured requirements processes. In
Proceedings of the 5th conference on Designing interactive systems:
processes, practices, methods, and techniques (pp. 113-122). ACM.
[S11] Paulson, J. W., Succi, G., & Eberlein, A. (2004). An empirical study of open-
source and closed-source software products. Software Engineering, IEEE
Transactions on, 30(4), 246–256.
[S12] Itkonen, J., & Rautiainen, K. (2005, November). Exploratory testing: a
multiple case study. In Empirical Software Engineering, 2005. 2005
International Symposium on (pp. 10-pp). IEEE.
[S13] Yang, X., Dong, J., & Ghafoor, M. A. (2005, May). Heterogeneous and
homogenous pairs in pair programming: an empirical analysis. In Electrical
and Computer Engineering, 2005. Canadian Conference on (pp. 1116-1119).
IEEE.
[S14] Maiden, N., Ncube, C., & Robertson, S. (2007, May). Can requirements be
creative? experiences with an enhanced air space management system. In
Software Engineering, 2007. ICSE 2007. 29th International Conference on
(pp. 632-641). IEEE.
[S15] Kidane, Y. H., & Gloor, P. A. (2007). Correlating temporal communication
patterns of the Eclipse open source community with performance and
creativity. Computational and mathematical organization theory, 13(1), 17–
27.
[S16] Itkonen, J., Mäntylä, M. V., & Lassenius, C. (2007, September). Defect
detection efficiency: Test case based vs. exploratory testing. In Empirical
Software Engineering and Measurement, 2007. ESEM 2007. First
International Symposium on (pp. 61-70). IEEE.
[S17] Ojala, P. (2008, March). Assessing Value of SW Requirements. In Software
Engineering, 2008. ASWEC 2008. 19th Australian Conference on (pp. 536-
542). IEEE.
[S18] Mangano, N., Baker, A., & Van Der Hoek, A. (2008, May). Calico: a
prototype sketching tool for modeling in early design. In Proceedings of the
2008 international workshop on Models in software engineering (pp. 63-68).
ACM.
63
[S19] Lewis, T. L., & Smith, W. J. (2008). Creating high performing software
engineering teams: the impact of problem solving style dominance on group
conflict and performance. Journal of Computing Sciences in Colleges, 24(2),
121–129.
[S20] Kasirun, Z. M., & Salim, S. S. (2008, December). Focus group discussion
model for requirements elicitation activity. In Computer and Electrical
Engineering, 2008. ICCEE 2008. International Conference on (pp. 101-105).
IEEE.
[S21] Goodman, B. D., & Goldman, S. N. (2007, July). Freeing creativity by
understanding the role of best practices. In Engineering Management
Conference, 2007 IEEE International (pp. 308–311). IEEE.
[S22] Yilmaz, L. (2008). Innovation Systems Are Self-organizing Complex
Adaptive Systems. In AAAI Spring Symposium: Creative Intelligent Systems
(pp. 142–148).
[S23] Petre, M. (2009, August). Insights from expert software design practice. In
Proceedings of the 7th joint meeting of the European software engineering
conference and the ACM SIGSOFT symposium on The foundations of
software engineering (pp. 233-242). ACM.
[S24] Sosa, M., & Danilovic, M. (2009). A Structured Approach to Re-Organize
for Creativity. In DS 58-3: Proceedings of ICED 09, the 17th International
Conference on Engineering Design, Vol. 3, Design Organization and
Management, Palo Alto, CA, USA, 24.-27.08. 2009.
[S25] Sharp, H., & Hall, T. (2009, May). An initial investigation of software
practitioners' motivation. In Proceedings of the 2009 ICSE Workshop on
Cooperative and Human Aspects on Software Engineering (pp. 84-91). IEEE
Computer Society.
[S26] Christiaans, H., & Almendra, R. A. (2010). Accessing decision-making in
software design. Design Studies, 31(6), 641–662.
[S27] Wang, J., & Carroll, J. M. (2011, November). Beyond fixing bugs: case
studies of creative collaboration in open source software bug fixing
processes. In Proceedings of the 8th ACM conference on Creativity and
cognition (pp. 397-398). ACM.
[S28] Shih, P. C., Venolia, G., & Olson, G. M. (2011, July). Brainstorming under
constraints: why software developers brainstorm in groups. In Proceedings
64
of the 25th BCS Conference on Human-Computer Interaction (pp. 74-83).
British Computer Society.
[S29] Nikitina, N., & Kajko-Mattsson, M. (2011, May). Developer-driven big-bang
process transition from Scrum to Kanban. In Proceedings of the 2011
International Conference on Software and Systems Process (pp. 159-168).
ACM.
[S30] Cukier, D., & Kon, F. (2011, October). Extending patterns for fearless
change. In Proceedings of the 18th Conference on Pattern Languages of
Programs (p. 18). ACM.
[S31] Zhu, L., Vaghi, I., & Barricelli, B. R. (2011, October). A meta-reflective wiki
for collaborative design. In Proceedings of the 7th International Symposium
on Wikis and Open Collaboration (pp. 53-62). ACM.
[S32] Walny, J., Haber, J., Dörk, M., Sillito, J., & Carpendale, S. (2011,
September). Follow that sketch: Lifecycles of diagrams and sketches in
software development. In Visualizing Software for Understanding and
Analysis (VISSOFT), 2011 6th IEEE International Workshop on (pp. 1-8).
IEEE.
[S33] Cao, J., Fleming, S. D., & Burnett, M. (2011, September). An exploration of
design opportunities for “gardening” end-user programmers' ideas. In Visual
Languages and Human-Centric Computing (VL/HCC), 2011 IEEE
Symposium on (pp. 35-42). IEEE.
[S34] Wanderley, F., da Silveira, D. S., Araujo, J., & Lencastre, M. (2012,
September). Generating feature model from creative requirements using
model driven design. In Proceedings of the 16th International Software
Product Line Conference-Volume 2 (pp. 18-25). ACM.
[S35] Crawford, B., de la Barra, C. L., Soto, R., & Monfroy, E. (2012, June). Agile
software engineering as creative work. In Proceedings of the 5th
International Workshop on Co-operative and Human Aspects of Software
Engineering (pp. 20-26). IEEE Press.
[S36] Crawford, B., De la Barra, C. L., Soto, R., & Monfroy, E. (2012, May). Agile
software teams must be creatives. In Engineering Applications (WEA), 2012
Workshop on (pp. 1-6). IEEE.
[S37] Sutcliffe, A. (2013, July). Bridging users' values and requirements to
architecture. In Twin Peaks of Requirements and Architecture (TwinPeaks),
2013 3rd International Workshop on the (pp. 15-21). IEEE.
65
[S38] Bartelt, C., Vogel, M., & Warnecke, T. (2013). Collaborative Creativity:
From Hand Drawn Sketches to Formal Domain Specific Models and Back
Again. InMoRoCo@ ECSCW (pp. 25–32).
[S39] Haines, J. K. (2013, June). Cultivating creativity in diverse teams. In
Proceedings of the 9th ACM Conference on Creativity & Cognition (pp. 32-
41). ACM.
[S40] Scharf, A., & Amma, T. (2013, May). Dynamic injection of sketching
features into GEF based diagram editors. In Software Engineering (ICSE),
2013 35th International Conference on (pp. 822-831). IEEE.
[S41] Jirachiefpattana, W. (2013, March). Exploring the effect of meta-individual
values on management of software development using Schwartz's model: A
case of Thai IT professionals. In Computer Science and Information
Technology (CSIT), 2013 5th International Conference on (pp. 162-168).
IEEE.
[S42] Dexter, S., & Kozbelt, A. (2013). Free and open source software (FOSS) as
a model domain for answering big questions about creativity. Mind &
Society, 12(1), 113–123.
[S43] Wanderley, F., & Araujo, J. (2013, July). Generating goal-oriented models
from creative requirements using model driven engineering. In Model-
Driven Requirements Engineering (MoDRE), 2013 International Workshop
on (pp. 1-9). IEEE.
[S44] Wenjun, W. U., Wei-Tek, T. S. A. I., & Wei, L. I. (2013). An evaluation
framework for software crowdsourcing. Frontiers of Computer Science, 7(5),
694–709.
[S45] Niknafs, A., & Berry, D. M. (2012, September). The impact of domain
knowledge on the effectiveness of requirements idea generation during
requirements elicitation. In Requirements Engineering Conference (RE),
2012 20th IEEE International (pp. 181-190). IEEE.
[S46] Kwiatkowska, J., Szóstek, A., & Lamas, D. (2014, October). (Un)structured
sources of inspiration: comparing the effects of game-like cards and design
cards on creativity in co-design process. In Proceedings of the 13th
Participatory Design Conference: Research Papers-Volume 1 (pp. 31-39).
ACM.
[S47] Bhowmik, T., Niu, N., Mahmoud, A., & Savolainen, J. (2014, August).
Automated support for combinational creativity in requirements engineering.
66
In Requirements Engineering Conference (RE), 2014 IEEE 22nd
International (pp. 243-252). IEEE.
[S48] Talluri, M., & Haddad, H. M. (2014, March). Best managerial practices in
agile development. In Proceedings of the 2014 ACM Southeast Regional
Conference (p. 3). ACM.
[S49] Díaz, P., Aedo, I., & Cubas, J. (2014, May). CoDICE: balancing software
engineering and creativity in the co-design of digital encounters with cultural
heritage. In Proceedings of the 2014 International Working Conference on
Advanced Visual Interfaces (pp. 253-256). ACM.
[S50] Yang, H., & Zhang, L. (2014, July). Controlling and Being Creative:
Software Cybernetics and Creative Computing. In 2014 IEEE 38th
International Computer Software and Applications Conference Workshops
(COMPSACW) (pp. 19-24). IEEE.
[S51] Mahaux, M., Nguyen, L., Mich, L., & Mavin, A. (2014, August). A
framework for understanding collaborative creativity in requirements
engineering: Empirical validation. In Empirical Requirements Engineering
(EmpiRE), 2014 IEEE Fourth International Workshop on (pp. 48-55). IEEE.
[S52] Graziotin, D., Wang, X., & Abrahamsson, P. (2014). Happy software
developers solve problems better: psychological measurements in empirical
software engineering. PeerJ, 2, e289.
[S53] Pfahl, D., Yin, H., Mäntylä, M. V., & Münch, J. (2014, September). How is
exploratory testing used?: a state-of-the-practice survey. In Proceedings of
the 8th ACM/IEEE International Symposium on Empirical Software
Engineering and Measurement (p. 5). ACM.
[S54] Sutling, K., Mansor, Z., Widyarto, S., Letchmunan, S., & Arshad, N. H.
(2014, September). Agile project manager behavior: The taxonomy. In
Software Engineering Conference (MySEC), 2014 8th Malaysian (pp. 234-
239). IEEE.
[S55] Murray-Rust, D., Scekic, O., Truong, H. L., Robertson, D., & Dustdar, S.
(2014, October). A collaboration model for community-based Software
Development with social machines. In Collaborative Computing:
Networking, Applications and Worksharing (CollaborateCom), 2014
International Conference on (pp. 84-93). IEEE.
[S56] LaToza, T. D., Chen, M., Jiang, L., Zhao, M., & van der Hoek, A. (2015).
Borrowing from the crowd: A study of recombination in software design
67
competitions. In Proceedings of the 37nd ACM/IEEE International
Conference on Software Engineering.
[S57] Lemma, R., Lanza, M., & Mocci, A. (2015, March). CEL: Touching software
modeling in essence. In Software Analysis, Evolution and Reengineering
(SANER), 2015 IEEE 22nd International Conference on (pp. 439-448).
IEEE.
[S58] Horkoff, J., Maiden, N., & Lockerbie, J. (2015, June). Creativity and Goal
Modeling for Software Requirements Engineering. In Proceedings of the
2015 ACM SIGCHI Conference on Creativity and Cognition (pp. 165-168).
ACM.
[S59] Wüest, D., Seyff, N., & Glinz, M. (2015, May). FlexiSketch Team:
collaborative sketching and notation creation on the fly. In Proceedings of
the 37th International Conference on Software Engineering-Volume 2 (pp.
685-688). IEEE Press.
[S60] Mangano, N., LaToza, T. D., Petre, M., & Van der Hoek, A. (2015). How
Software Designers Interact with Sketches at the Whiteboard. Software
Engineering, IEEE Transactions on, 41(2), 135-156.
[S61] Suscheck, C. A., & Ford, R. (2008). Jazz improvisation as a learning
metaphor for the scrum software development methodology. Software
Process: Improvement and Practice, 13(5), 439–450
[S62] Bhowmik, T., Niu, N., Savolainen, J., & Mahmoud, A. Leveraging topic
modeling and part-of-speech tagging to support combinational creativity in
requirements engineering. Requirements Engineering, 1-28.
[S63] Petre, M., & Damian, D. (2014, November). Methodology and culture:
drivers of mediocrity in software engineering?. In Proceedings of the 22nd
ACM SIGSOFT International Symposium on Foundations of Software
Engineering (pp. 829-832). ACM.
[S64] Fernández, D. M., & Wagner, S. (2015). Naming the pain in requirements
engineering: A design for a global family of surveys and first results from
Germany. Information and Software Technology, 57, 616–643.
[S65] Dekel, U., & Herbsleb, J. D. (2007, October). Notation and representation in
collaborative object-oriented design: an observational study. In ACM SIGPLAN
Notices (Vol. 42, No. 10, pp. 261–280). ACM.
[S66] Amin, A., Basri, S., Hassan, M. F., & Rehman, M. (2011, December).
Occupational stress, knowledge sharing and GSD communication barriers as
predictors of software engineer's creativity. In Industrial Engineering and
68
Engineering Management (IEEM), 2011 IEEE International Conference on (pp.
394-398). IEEE.
[S67] Wen, Y., Zhang, H., Liu, L., & Yang, H. (2010, September). One bridge, two
gaps-beyond an engineering approach: Creativity in requirements elicitation. In
Requirements Engineering Visualization (REV), 2010 Fifth International
Workshop on (pp. 40-42). IEEE.
[S68] Cox, A., & Fisher, M. (2009, February). Programming Style: Influences, Factors,
and Elements. In Proceedings of the 2009 Second International Conferences on
Advances in Computer-Human Interactions (pp. 82-89). IEEE Computer Society.
[S69] Belshee, A. (2005, July). Promiscuous pairing and beginner's mind: embrace
inexperience [agile programming]. In Agile Conference, 2005. Proceedings (pp.
125–131). IEEE.
[S70] Maiden, N., Gizikis, A., & Robertson, S. (2004). Provoking creativity:
Imagine what your requirements could be like. Software, IEEE, 21(5), 68-
75.
[S71] Sutcliffe, A., & Sawyer, P. (2013, July). Requirements elicitation: Towards
the unknown unknowns. In Requirements Engineering Conference (RE),
2013 21st IEEE International (pp. 92-104). IEEE.
[S72] Kauppinen, M., Savolainen, J., & Männistö, T. (2007, October). Requirements
engineering as a driver for innovations. In Requirements Engineering Conference,
2007. RE'07. 15th IEEE International (pp. 15-20). IEEE.
[S73] Maiden, N., Jones, S., Karlsen, K., Neill, R., Zachos, K., & Milne, A. (2010,
September). Requirements engineering as creative problem solving: A research
agenda for idea finding. In Requirements Engineering Conference (RE), 2010 18th
IEEE International (pp. 57-66). IEEE.
[S74] Grube, P. P., & Schmid, K. (2008, September). Selecting creativity techniques for
innovative requirements engineering. In Multimedia and Enjoyable Requirements
Engineering-Beyond Mere Descriptions and with More Fun and Games, 2008.
MERE'08. Third International Workshop on (pp. 32–36). IEEE.
[S75] Chung, E., Jensen, C., Yatani, K., Kuechler, V., & Truong, K. N. (2010,
September). Sketching and drawing in the design of open source software. In
Visual Languages and Human-Centric Computing (VL/HCC), 2010 IEEE
Symposium on (pp. 195-202). IEEE.
[S76] Craft, B., & Cairns, P. (2009, September). Sketching: outlines of a collaborative
design method. In Proceedings of the 23rd British HCI Group Annual Conference
on People and Computers: Celebrating People and Technology (pp. 65-72).
British Computer Society.
[S77] Mangano, N., Baker, A., Dempsey, M., Navarro, E., & van der Hoek, A. (2010,
September). Software design sketching with calico. In Proceedings of the
IEEE/ACM international conference on Automated software engineering (pp. 23-
32). ACM.
69
[S78] Glass, R. L., & Vessey, I. (1994, January). Software tasks: intellectual, clerical...
or creative?. In System Sciences, 1994. Proceedings of the Twenty-Seventh Hawaii
International Conference on (Vol. 4, pp. 377-382). IEEE.
[S79] Bull, C. N., Whittle, J., & Cruickshank, L. (2013, May). Studios in software
engineering education: towards an evaluable model. In Proceedings of the 2013
International Conference on Software Engineering (pp. 1063-1072). IEEE Press.
[S80] Pinto-Albuquerque, M., & Rashid, A. (2014, August). Tackling the requirements
jigsaw puzzle. In Requirements Engineering Conference (RE), 2014 IEEE 22nd
International (pp. 233-242). IEEE.
[S81] Hunter, C., Jemielniak, D., & Postula, A. (2010). Temporal and spatial shifts
within playful work. Journal of Organizational Change Management, 23(1), 87–
102.
[S82] Niknafs, A., & Berry, D. M. (2012, September). The impact of domain knowledge
on the effectiveness of requirements idea generation during requirements
elicitation. In Requirements Engineering Conference (RE), 2012 20th IEEE
International (pp. 181-190). IEEE.
[S83] Tolfo, C., & Wazlawick, R. S. (2008). The influence of organizational culture on
the adoption of extreme programming. Journal of systems and software, 81(11),
1955–1967.
[S84] Kamthan, P. (2014, May). Towards Understanding the Use of Wiki in Agile
Software Development. In Advanced Information Networking and Applications
Workshops (WAINA), 2014 28th International Conference on (pp. 566-571).
IEEE.
[S85] Wanderley, F., Silva da Silveira, D., Araujo, J., & Moreira, A. (2013, May).
Transforming creative requirements into conceptual models. In Research
Challenges in Information Science (RCIS), 2013 IEEE Seventh International
Conference on (pp. 1-10). IEEE.
[S86] Stol, K. J., & Fitzgerald, B. (2014, May). Two's company, three's a crowd: a case
study of crowdsourcing software development. In Proceedings of the 36th
International Conference on Software Engineering (pp. 187-198). ACM.
[S87] Simmons, M. M., Vercellone-Smith, P., & Laplante, P. (2006, April).
Understanding open source software through software archaeology: The case of
nethack. In Software Engineering Workshop, 2006. SEW'06. 30th Annual
IEEE/NASA (pp. 47-58). IEEE.
[S88] Jones, S., Lynch, P., Maiden, N., & Lindst, S. (2008, September). Use and
influence of creative ideas and requirements for a work-integrated learning
system. In International Requirements Engineering, 2008. RE'08. 16th IEEE (pp.
289-294). IEEE.
[S89] Ghanbari, H., Similä, J., & Markkula, J. (2015). Utilizing online serious games to
facilitate distributed requirements elicitation. Journal of Systems and Software,
109, 32-49.
70
